How Does Googlebot Crawl a Site? – Discovery, Budget, and Rendering
Learn how a search crawler discovers URLs, decides how much to fetch, renders JavaScript, and what makes it crawl a site more or less.
Crawling is a scheduling problem. The crawler maintains a list of known URLs and decides continuously which to fetch, how often, and how much of a site's capacity to consume.
1. Discovery
URLs enter the list from links on already-crawled pages, sitemap submissions, and redirects encountered along the way.
Internal linking therefore does real work: a page with no inbound links and no sitemap entry may never be discovered, however good it is.
2. How Much Gets Fetched
Two factors combine into what is commonly called crawl budget:
- Capacity — how much the site can serve without degrading. Fast responses raise it; slow responses, timeouts, and server errors lower it.
- Demand — how much the crawler judges the site worth fetching, based on popularity, freshness, and value.
Performance work therefore increases crawling directly, which is a more concrete argument for it than most.
3. Rendering Happens Separately
Because so much content is assembled client-side, the crawler renders pages in a browser-like environment. That rendering is queued separately from the initial fetch.
The consequence is a delay: a client-rendered page may be fetched now and understood later. Server-rendered content is available immediately, which is why hybrid rendering became standard practice.
4. What Wastes the Budget
| Waste | Cause |
|---|---|
| Parameter variants | Tracking, sorting, and session parameters |
| Faceted navigation | Unbounded filter combinations |
| Internal search results | Effectively infinite |
| Soft 404s | Requests spent on pages that say nothing |
| Redirect chains | Several fetches to reach one page |
| Slow pages | Fewer fetches in the same time |
Faceted navigation is the classic trap — a few filters can generate millions of URLs, each a fetch not spent on a real page.
5. Directing Attention
- Canonical tags, consolidating variants.
robots.txt, blocking genuinely worthless paths.- Internal linking that reflects importance.
- Sitemaps with honest
lastmod, enabling incremental crawling. - Removing redirect chains.
- Improving response times.
6. Recrawl Frequency Varies by Page
Pages that change often and matter are revisited more; static, low-value pages are revisited rarely. Honest lastmod values in a sitemap help the crawler make that judgement well.
Fabricated lastmod values — every URL updated today — undermine it and are eventually discounted.
7. Verifying What Visited
User-agent strings are self-declared, so traffic claiming to be a search crawler may not be. Verification uses published address ranges, or a reverse-then-forward DNS check.
Server logs are the only reliable picture of what a site's automated traffic actually consists of.
8. Summary
A search crawler discovers URLs from links and sitemaps, fetches according to the site's serving capacity and its own assessment of value, and renders client-side content on a separate schedule.
Speed raises how much is crawled; parameter variants, infinite facets, soft 404s, and redirect chains waste it; and canonicals, selective blocking, honest sitemaps, and internal linking direct it.