Scrapeless Wiki

What Is a Web Crawler? – Definition and How It Works

Informational P1 what is a web crawler
Overlaps a live page. A published page already targets this keyword: https://www.scrapeless.com/en/blog/what-is-a-web-crawler. Publishing both risks cannibalisation — decide before this goes live.

Learn what a web crawler is, how the frontier and deduplication work, how crawlers differ from scrapers, and what politeness means in practice.

A web crawler discovers pages by following links. Its output is not data about a page but a set of pages — which is a different problem from extraction, and one with its own characteristic failures.

1. What Is a Web Crawler?

A crawler traverses the link graph, visiting pages and queuing the links it finds.

  • Key idea: discovery, not extraction. The question is which pages exist.
  • Mechanism: start from seeds, fetch, extract links, enqueue unseen ones, repeat.
  • Goal: reach a defined portion of a site or the web.

Googlebot is the best-known example. So is the job that enumerates every product URL on a retailer before anything is scraped from them.

2. The Loop

  1. Seeds. Start URLs — a homepage, a sitemap, a category index.
  2. Frontier. The queue of URLs to visit, ordered by some priority.
  3. Fetch. Retrieve the next URL, respecting per-host rate limits.
  4. Extract links. Find every URL in the response.
  5. Normalise. Resolve relative paths, strip fragments, canonicalise.
  6. Deduplicate. Drop anything already seen or queued.
  7. Enqueue what remains, and repeat.

The frontier and the seen-set are the crawler. Everything else is plumbing.

3. URL Normalisation

The step that decides whether a crawl terminates.

These are usually one page:

https://example.com/products?id=1&ref=email
https://example.com/products?ref=email&id=1
https://example.com/products?id=1#reviews
https://example.com/Products?id=1

Without normalisation — sorting parameters, dropping tracking parameters and fragments, unifying trailing slashes and case where appropriate — a crawler treats each as new and revisits the same content indefinitely.

The pathological case is a crawler trap: a calendar with a "next month" link generates infinite URLs, each genuinely distinct, each leading to another. A crawler without depth limits, URL-pattern caps and a budget will follow it forever.

4. Crawler vs. Scraper

Crawler Scraper
Answers Which pages exist? What is on this page?
Output URLs Structured records
Core state Frontier and seen-set Selectors
Scales with Site size Pages of interest
Fails by Missing pages, looping Extracting nothing

They are usually two phases of one job. Keeping them separate is worth doing deliberately: a crawl that produced 4,000 URLs where yesterday it produced 40,000 is a diagnosable event, and it is invisible if discovery and extraction are entangled.

5. Politeness

A crawler can generate more load than a site's entire human audience. The conventions that keep it welcome:

  • Read robots.txt. Honour Disallow and Crawl-delay. It is a convention rather than a law, and ignoring it is a decision you should make consciously.
  • Rate limit per host, not globally. One request per second to each of fifty hosts is reasonable; fifty per second to one is not.
  • Identify yourself. A user agent naming your crawler with a URL explaining it converts an anonymous flood into something an administrator can evaluate.
  • Crawl off-peak where you know the pattern.
  • Honour 429 and Retry-After. Back off exponentially with jitter.
  • Prefer sitemaps. They give you the URL set directly and spare the site the discovery traffic entirely.

6. Scaling Considerations

  • The seen-set grows unboundedly. At tens of millions of URLs, an in-memory set stops fitting — Bloom filters and disk-backed stores are the usual answers.
  • The frontier needs priority. Breadth-first is a reasonable default; importance-ordered crawling reaches valuable pages sooner.
  • Per-host queues. Politeness is per host, so a global queue starves you or overloads someone.
  • Persist state. A crawl that cannot resume loses everything to one restart.
  • Bound everything. Depth, total pages, per-host pages, and wall-clock time. Traps are common and cheap to defend against in advance.

Scrapy provides all of this and is usually a better choice than reimplementing it.

7. Real-World Examples

  • Googlebot discovering and refreshing the index.
  • A retail crawl enumerating category pages to build a product URL list before extraction.
  • A crawler stuck in a calendar generating dated URLs indefinitely until a depth cap was added.
  • A crawl that silently shrank by 90% after a navigation change, unnoticed because only extraction was monitored.
  • A sitemap-driven job that replaced link discovery entirely, cutting requests by an order of magnitude.

8. Summary

A web crawler discovers URLs by following links, driven by a frontier of pending URLs and a seen-set that prevents revisiting. Its output is a set of pages; extraction is a separate concern, and separating them makes discovery failures visible.

URL normalisation is what makes a crawl terminate, and bounded depth and page budgets are what protect it from traps that generate infinite valid URLs.

Politeness is not optional courtesy — per-host rate limiting, honouring robots.txt and Retry-After, and identifying your crawler are what keep it running. And checking for a sitemap first often removes the need to crawl for discovery at all.