This article is the third part of my web scraping series. I focuse on creating a Scrapy Spider to scrape Apple's website for the latest iPhone models and their prices. I will explain how to create a new Scrapy project, and generate a Spider. The article also covers fetching the pages for different iPhone models from the US, Swedish, and Hungarian Apple sites.
The 2nd part of my web scraping tutorial covers Scrapy Shell, which can be used to interactively test and refine scraping logic. We will scrape Apple's website for the latest iPhone models and their prices.
Why is it a really bad idea to use Thread.sleep() in your tests? And what should you use instead?
