What Is Scrapy? – Definition and How It Works
Learn what Scrapy is, how spiders, items, and pipelines fit together, how its asynchronous engine achieves throughput, and where it does not fit.
Scrapy is a Python framework for building crawlers. It supplies the scheduling, concurrency, retry handling, and output plumbing that a crawl needs, so project code is mostly about which pages to visit and what to extract.
1. What Is Scrapy?
It is a crawling framework rather than a request library.
- Key idea: structure a crawl as spiders producing requests and items, processed by an asynchronous engine.
- Mechanism: an event-driven core schedules requests, applies middleware, and feeds responses to callbacks.
- Goal: high-throughput crawling with politeness, deduplication, and export handled by the framework.
Example Spider
class ProductSpider(scrapy.Spider):
name = "products"
start_urls = ["https://example.com/catalogue"]
def parse(self, response):
for card in response.css("article.product"):
yield {
"title": card.css("h3::text").get(),
"price": card.css(".price::text").get(),
}
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, self.parse)
2. The Core Concepts
| Concept | Role |
|---|---|
| Spider | Defines start URLs, how to parse, and what to follow |
| Selector | CSS and XPath querying built into the response object |
| Item | An optional typed container for extracted records |
| Item pipeline | Post-processing — validation, deduplication, storage |
| Middleware | Hooks around requests and responses |
| Feed export | Writes results to JSON, JSON Lines, CSV, or elsewhere |
3. Why It Is Fast
Scrapy is asynchronous. Many requests are in flight at once inside a single process, without threads, so throughput comes from concurrency rather than parallel processes.
Concurrency, per-domain limits, and delay are configured rather than coded, and an auto-throttling extension can adapt the rate to observed response times.
4. What Comes Built In
- Duplicate request filtering, so a crawl does not revisit the same URL.
- Automatic retries on failure conditions, with configurable policy.
robots.txtobedience, on by default.- Cookie and session handling across requests.
- Caching of responses during development, so re-runs do not re-fetch.
- A shell for interactively testing selectors against a live page.
5. Where It Does Not Fit
Scrapy fetches over HTTP. It does not run JavaScript, so client-rendered pages return the served shell rather than the content.
The usual answers are to find the underlying API the page calls and request that directly, or to pair Scrapy with a browser layer for the pages that genuinely need one. For a handful of pages, a simpler script is often the better tool — the framework earns its overhead at crawl scale.
6. Common Misconceptions
- "Scrapy renders pages." It does not. JavaScript-dependent content needs a browser or the underlying endpoint.
- "It is just requests plus BeautifulSoup." The scheduler, deduplication, concurrency control, and pipelines are the actual product.
- "More concurrency is always faster." Beyond a site's tolerance it produces errors and bans; per-domain limits exist for a reason.
- "Pipelines are optional detail." Validation and deduplication in a pipeline is what keeps a large crawl's output usable.
7. Real-World Examples
- Crawling a catalogue across thousands of paginated pages into JSON Lines.
- Following category links to discover product URLs, then extracting records from each.
- Running a scheduled crawl that writes into a database through a pipeline.
- Monitoring a set of pages for change, with caching to avoid refetching unchanged responses.
8. Summary
Scrapy structures a crawl into spiders, items, and pipelines on top of an asynchronous engine that handles scheduling, deduplication, politeness, and export.
It is the right tool for crawls measured in thousands of pages over HTTP. It is the wrong tool for a handful of URLs, and it needs help for pages that only exist after JavaScript runs.