Scrapy Tutorial Series: Web Scraping Using Python

What is included in this Scrapy tutorial

High-quality articles help you compare Scrapy and other popular web scraping tools.
10+ articles teach you to build Scrapy project, develop spider and save the scraped data to Database.
10+ practical exercises help you improve web scraping skills by solving problems from the real world.

I spend much time on this Scrapy Tutorial (10+ articles and a Django project) to help people learn Web Scraping, try to make this world better.

For the better experience, I removed all annoying ads from all tutorial pages, so I can not get money from my blog.

You can support my work by buying my ebook on Scrapy, I wish I can help more people. Support My Work

Web Scraping Framework Review: Scrapy VS Selenium In this Scrapy tutorial, I will cover the features of Scrapy and Selenium, and help you decide which one is better for your projects.
Scrapy Tutorial #1: Scrapy VS Beautiful Soup In this Scrapy tutorial, I will talk about the features of Scrapy, BeautifulSoup, and help you decide which one is better for your projects.

Scrapy Tutorial #2: How To Install Scrapy On Mac In this Scrapy tutorial, I will talk about basic points of Python such as Python version and show you how to install Scrapy on your Mac.
Scrapy Tutorial #3: How To Install Scrapy On Linux In this Scrapy tutorial, I will show you how to install Scrapy on Linux distributions such as Ubuntu, CentOS, or other Linux distributions.
Scrapy Tutorial #4: How To Install Scrapy On Windows In this Scrapy tutorial, I will talk about how to install scrapy on Windows. You can use this guide helps you install Scrapy on Win 7, Win 8 or Win 10.

Scrapy Tutorial #5: How To Create Simple Scrapy Spider In this Scrapy tutorial, I will talk about how to create a Scrapy project and a simple Scrapy spider from scratch.
Scrapy Tutorial #6: Scrapy Shell Overview & Tips In this Scrapy tutorial, I will talk about how to use Scrapy shell to help us extract data, and I will share with you some tips about it.
Scrapy Tutorial #7: How to use XPath with Scrapy In this Scrapy tutorial, I will talk about how to use XPath in scrapy to extract info and how to help you quickly write XPath expressions.
Scrapy Tutorial #8: Scrapy Selector Guide In this Scrapy Tutorial, I would talk about Scrapy Selector and how to create it and use it with iteration.
Scrapy Tutorial #9: How To Use Scrapy Item In this Scrapy tutorial, I will talk about how to define Scrapy item, and how to create a custom Item Pipeline to save the data of Item into DB.
Scrapy Tutorial #10: How To Build A Real Spider In this Scrapy tutorial, I will show you how to write a real spider which can extract data and handle pagination.

Scrapy Tutorial #11: How To Extract Data From Native Javascript Statement In this Scrapy tutorial, I will show you how to extract data from native javascript statement using Regex and Json.
How to Crawl Infinite Scrolling Pages using Python I will talk about how to crawl infinite scrolling pages using Python. This tutorial also includes two code snippets based on Scrapy and Beautifulsoup.

Web Scraping Practice Exercises are to help Scrapy beginner quickly learn web scraping skills by solving problems from the real world.

Basic Info Scraping Web scraping using XPath or CSS expression
Analyze JSON Load JSON string and extract data
Recursively Scraping pages Not only crawl products but also handle pagination
Mimicking Ajax requests Inspect Ajax requests and mimic them
Inspect HTTP request Learn to inspect the fields of HTTP request
Scraping Infinite Scrolling Pages (Ajax) Learn to scrape infinite scrolling pages
Find gold in cookie Make your spider can work with the cookie
Login form Scrape data behind login form
Solve Captcha Learn to scrape data behind a captcha
Decode minified javascript Learn how to analyze minimized or compressed javascript