What Is Crawl Budget? – Definition and How It Works
Learn what crawl budget means, what determines it, which pages waste it, and how to direct crawler attention to what matters.
Crawl budget is the number of requests a crawler will make to a site in a period. It matters on large sites, where the number of possible URLs exceeds what any crawler will fetch.
1. What Determines It
Two factors combine:
- Capacity — how much the site can serve without degrading. Fast, reliable responses raise it; slow responses and server errors lower it.
- Demand — how much the crawler considers the site worth fetching, based on popularity, freshness, and perceived value.
A site can therefore increase its budget by being faster and more reliable, and by being worth crawling.
2. Who Needs to Care
Sites with tens of thousands of URLs or more, sites generating URLs combinatorially, and sites with frequently changing inventories.
A site with a few hundred pages will be crawled completely regardless. Optimising crawl budget there is effort spent on a non-problem.
3. Where It Is Wasted
| Waste | Cause |
|---|---|
| Parameter variants | Tracking, sorting, and session parameters producing duplicate URLs |
| Infinite filter combinations | Faceted navigation generating unbounded URLs |
| Internal search results | Effectively unlimited, and rarely valuable |
| Calendar pages | Endless date navigation |
| Soft 404s | Requests spent on pages that say nothing |
| Redirect chains | Several requests to reach one page |
| Slow pages | Fewer total fetches within the same time |
Faceted navigation is the classic case: a handful of filters can generate millions of URLs, each a request not spent on a product page.
4. Directing Attention
- Canonical tags, consolidating parameter variants.
robots.txt, blocking genuinely worthless paths such as internal search.- Parameter handling, where the crawler offers configuration.
- Internal linking that reflects importance.
- Sitemaps with honest
lastmodvalues, enabling incremental crawling. - Removing redirect chains so one request reaches the destination.
5. Speed Is a Budget Lever
Because capacity depends on response time and error rate, making the site faster directly increases how much of it gets crawled.
That makes performance work a crawl-budget intervention as well as a user-experience one, which is a more compelling framing for prioritising it.
6. Signals That Budget Is a Problem
- Newly published pages taking a long time to appear.
- Crawl statistics showing most requests hitting parameterised or low-value URLs.
- Server logs dominated by paths nobody intended to be crawled.
- Large sections never fetched at all.
7. What It Means From the Other Side
The same economics apply to your own collection. A finite request budget spent on parameter variants, redirect chains, and soft 404s is a budget not spent on the pages holding the data.
Reading the sitemap, normalising to canonical URLs, and skipping known-worthless paths are exactly the same optimisations, applied to your crawler rather than to someone else's.
8. Summary
Crawl budget is the share of a crawler's attention a site receives, set by the site's serving capacity and the crawler's assessment of value.
It matters at scale, is wasted by parameter variants, infinite facets, soft 404s, and slow responses, and is directed by canonicalisation, selective blocking, honest sitemaps, and performance work.