windows下搭建scrapy开发环境

python：python3.6

开发工具：intellij

1、安装scrapy 模块

pip3 install scrapy

2、创建项目目录和虚拟机

E:\work\proc>d:\Python\Python36\python.exe -m scrapy startproject compass

3、安装虚拟机

E:\work\proc>d:\Python\Python36\python.exe -m venv venv

开发scrapy

1、打开ide加载目录并指定ide的虚拟机目录为刚创建的venv目录即可。

2、创建requirements.txt里面写上scrapy按照提示安装scrapy模块和pypiwin32【windows下安装】。

 

3、编写代码

在spider包里新建一个toscrape-xpath.py：

# -*- coding: utf-8 -*-
import scrapy


class ToScrapeSpiderXPath(scrapy.Spider):
    name = 'toscrape-xpath'
    start_urls = [
        'http://quotes.toscrape.com/',
    ]

    def parse(self, response):
        for quote in response.xpath('//div[@class="quote"]'):
            yield {
                'text': quote.xpath('./span[@class="text"]/text()').extract_first(),
                'author': quote.xpath('.//small[@class="author"]/text()').extract_first(),
                'tags': quote.xpath('.//div[@class="tags"]/a[@class="tag"]/text()').extract()
            }
        next_page_url = response.xpath('//li[@class="next"]/a/@href').extract_first()
        if next_page_url is not None:
            yield scrapy.Request(response.urljoin(next_page_url))


if __name__ == '__main__':
    from scrapy import cmdline
    name = ToScrapeSpiderXPath().name
    cmd = 'scrapy crawl {0}'.format(name)
    cmdline.execute(cmd.split())

运行此文件即可：【默认日志为debug模式可以在settings.py的最后加上LOG_LEVEL = 'INFO'等模式】