Personal tools
You are here: Home Plone XP Open Source Software Review Using Scrapy to Crawl Websites

Using Scrapy to Crawl Websites

Scrapy Homepage:, It has a blog site with the most recent updates:

Scrapy split to 2 modules: w3lib and scrapely.  Both have pypi page.

After these changes, Scrapy codebase has been reduced by 4574 lines, including blank and comments (according to cloc).

Scrapy team also offer a sample project to explain how to use scrapy:

Document Actions