What Is a Headless Crawl? – Definition and When It Is Needed
Learn what a headless crawl is, why it costs orders of magnitude more than fetching, and how to use browsers selectively within a crawl.
A headless crawl fetches pages with a real browser running without a visible window, so client-rendered content is available. It is the expensive answer to a problem that often has a cheaper one.
1. Why It Exists
A plain HTTP request returns the served document. Where a page assembles its content in the browser, that document is a shell and the crawl collects nothing useful — while reporting success.
A browser executes the scripts, so the content exists by the time the page is read.
2. The Cost Difference
| Approach | Per page |
|---|---|
| HTTP fetch | Milliseconds, kilobytes of memory |
| Headless browser | Seconds, tens to hundreds of megabytes |
Two to three orders of magnitude, which at crawl scale is the difference between a job that finishes overnight and one that does not finish.
3. Check Before Committing
curl -s "https://example.com/page" | grep -c "a value visible in the browser"
Non-zero means a fetch suffices for that page. Zero means scripts build it — and then the next question is whether those scripts call an endpoint you can call directly.
Whole crawls are built on browsers because nobody ran that check.
4. Selective Rendering Is the Answer
The production pattern is a crawl that fetches by default and renders only the pages that require it.
Detection can be automatic: fetch, run a content assertion, and escalate to a browser only when the assertion fails. Most sites need rendering for a subset of their pages rather than all of them.
5. Making Rendering Cheaper
- Block images, fonts, media, and analytics. Frequently halves load time and cuts memory.
- Wait on a selector, not on network idle or a fixed pause.
- Reuse one browser with a context per task rather than launching per page.
- Cap concurrency by memory, which is the real constraint.
- Close contexts promptly.
6. Bandwidth Matters if Proxied
Rendering fetches every subresource, so through a bandwidth-billed proxy a rendered page can cost megabytes where a fetch costs kilobytes.
Blocking resource types is therefore a direct cost control as well as a speed one, and calling the underlying endpoint avoids the cost entirely.
7. It Does Not Change How You Are Seen
A headless browser is a real browser, and automation remains observable through its own signals. Rendering solves a content problem, not a classification one.
Volume, pacing, and client consistency remain the factors that determine whether collection proceeds smoothly.
8. Summary
A headless crawl uses a real browser without a window so client-rendered content is available, at two to three orders of magnitude more cost per page than fetching.
Check whether rendering is needed, look for the underlying endpoint, render selectively within a crawl, block unnecessary resources, and bound concurrency by memory.