Scrapeless Wiki

How Does a Web Crawler Work?

Informational P1 how does a web crawler work
Overlaps a live page. A published page already targets this keyword: https://www.scrapeless.com/en/blog/what-is-a-web-crawler. Publishing both risks cannibalisation — decide before this goes live.

Follow a crawl step by step: seeds, the frontier, fetching, link extraction, normalisation, deduplication, scheduling, and how a crawl terminates.

A crawler is a loop around two data structures: a queue of URLs to visit and a record of URLs already seen. Everything else — politeness, priority, persistence — is a refinement of that loop.

1. Seeds

A crawl starts from known URLs.

The best seed is almost always a sitemap. robots.txt usually points to one, and it hands you the site's own list of canonical URLs — no discovery traffic, no missed pages behind JavaScript navigation, no trap risk.

Failing that: the homepage, category indexes, or a search results page.

2. The Frontier

The frontier is the queue of URLs waiting to be visited. Its ordering determines crawl behaviour:

  • FIFO gives breadth-first — level by level, and a sensible default.
  • LIFO gives depth-first, which follows one branch deep and is easy to trap.
  • Priority ordering visits by estimated importance — shallow paths, sitemap priority, or observed value.

At scale the frontier is not one queue but one per host, because rate limiting is per host. A single global queue either starves you (waiting on one slow host) or overloads someone (draining their URLs consecutively).

3. Fetching

Pop a URL, check politeness, and request it.

Before fetching, the crawler checks whether robots.txt permits the path, whether the per-host delay has elapsed, and whether budgets — depth, host page count, total pages — still allow it.

Responses split three ways:

  • 2xx — parse it and extract links.
  • 3xx — follow, and record the target as canonical. Cap redirect chains; loops exist.
  • 4xx / 5xx — record the failure. Retry 5xx and 429 with backoff; do not retry 404.

Every URL in the response is a candidate:

for a in tree.css("a[href]"):
    yield urljoin(base, a.attributes["href"])

Sources beyond anchors matter too — <link rel="canonical">, pagination rel="next", and sitemap entries.

A limitation worth naming: links created by JavaScript are invisible to an HTTP-only crawler. If a site's navigation is client-rendered, discovery silently under-collects. The usual fix is not a browser but the sitemap, or the internal API that serves the navigation.

5. Normalisation

Before deduplication, each URL is canonicalised:

  • Resolve relative paths against the base.
  • Drop the fragment — #reviews is the same document.
  • Sort query parameters so ordering does not create duplicates.
  • Strip tracking parameters — utm_*, ref, fbclid.
  • Normalise the scheme and host case; leave the path case alone unless you know the server is case-insensitive.
  • Decide on trailing slashes and apply it consistently.

Skipping this is the single most common reason a crawl never finishes.

6. Deduplication

Check the normalised URL against the seen-set. If present, discard. If not, record it and enqueue.

In memory a Python set is fine into the low millions. Beyond that, the options are a Bloom filter — constant memory, a small false-positive rate that silently skips a few pages — or a disk-backed key-value store, which is exact and slower.

Content-level deduplication is a separate concern: different URLs frequently serve identical content. Hashing the normalised body catches those, which matters when the same product is reachable through several category paths.

7. Termination

A crawl ends when the frontier empties — or, more often in practice, when a limit is reached. Every crawl needs explicit bounds:

  • Max depth from the seed.
  • Max pages, total and per host.
  • URL pattern caps, so one parameterised path cannot dominate.
  • Wall-clock timeout.

Without them, a calendar widget with a "next month" link generates unique, valid URLs forever, and the crawl never terminates. Bounds are cheap; discovering the need for them at 3 a.m. is not.

Persist the frontier and seen-set to disk. A crawl that cannot resume loses days of work to a single restart.

8. Summary

A crawler pops a URL from the frontier, checks politeness and budgets, fetches it, extracts links, normalises and deduplicates them, and enqueues what is new — repeating until the frontier empties or a limit stops it.

Normalisation is what makes termination possible, per-host queues are what make politeness workable, and explicit bounds are what protect the crawl from traps that generate infinite valid URLs.

Start from a sitemap when one exists: it gives the site's own canonical URL list without any discovery traffic, and it sidesteps the blind spot around JavaScript-generated navigation entirely.