Scrapy vs. Playwright – Definition and Differences
Compare a crawling framework with a browser automation library: what each is built for, cost per page, and how they are commonly combined.
These tools answer different questions. Scrapy asks how to crawl many pages efficiently. Playwright asks how to obtain a page that only exists after a browser runs it.
1. What Each Is Built For
Scrapy is a crawling framework: an asynchronous engine with scheduling, deduplication, retry policy, pipelines, and export. It fetches over HTTP and does not execute JavaScript.
Playwright is a browser automation library: it drives Chromium, Firefox, and WebKit, waits for elements, interacts, and returns rendered pages. It has no crawl scheduler.
2. Comparison
| Dimension | Scrapy | Playwright |
|---|---|---|
| Executes JavaScript | No | Yes |
| Crawl scheduling | Built in | None |
| Deduplication | Built in | None |
| Concurrency model | Async, very high | Bounded by browser memory |
| Cost per page | Milliseconds, kilobytes | Seconds, tens of megabytes |
| Interaction | None | Click, type, scroll, upload |
| Output pipelines | Built in | Yours to write |
The cost difference is the decisive factor at scale — roughly two orders of magnitude per page.
3. The Real Question
Not which tool, but whether the content is in the served HTML:
curl -s "https://example.com/page" | grep -c "a value visible in the browser"
Non-zero means Scrapy alone suffices. Zero, with the value visible in a browser, means scripts build it — and then the next question is whether those scripts call an endpoint you could call directly.
4. Combining Them
The common production shape is Scrapy for the crawl with a browser layer used selectively for the pages that require rendering.
That keeps scheduling, deduplication, and pipelines in the framework while paying browser cost only where it is unavoidable. Rendering every page in a large crawl is the pattern that makes crawls expensive and slow for no benefit.
5. When Scrapy Alone
- Server-rendered sites and templated catalogues.
- Crawls in the thousands to millions of pages.
- Feeds, sitemaps, and any structured source.
- Anywhere an underlying JSON endpoint can be called directly.
6. When Playwright Alone
- A small number of pages requiring interaction.
- Flows behind forms or logins.
- Screenshot and PDF generation.
- Exploratory work to understand how a site assembles content.
For a handful of pages, Scrapy's scheduling adds ceremony without benefit.
7. Language and Ecosystem
Both have Python bindings, so a Python team can use either or both. Playwright additionally targets JavaScript, Java, and .NET, while Scrapy is Python-only — which occasionally settles the decision on its own.
8. Summary
Scrapy crawls at scale over HTTP; Playwright renders and interacts with pages a plain fetch cannot reach. They are complements, not competitors.
Determine whether the content is in the served HTML, look for the underlying endpoint next, and reserve the browser for pages that genuinely need it — combining the two rather than choosing one is usually the right answer.