Scrapeless Wiki

What Is Politeness in Crawling? – Definition and How It Works

Informational P3 what is politeness in crawling

Learn what crawl politeness means, how per-host delay and concurrency limits work, and why it is self-interest as well as courtesy.

Politeness is the practice of crawling in a way that does not degrade the sites you crawl: limited concurrency per host, spacing between requests, honouring access rules, and identifying yourself.

1. It Is Not Only Courtesy

An impolite crawler gets blocked, and a blocked crawler collects nothing. So politeness is the strategy that keeps access available.

Impolite behaviour Consequence
High concurrency per host Blocked, sometimes permanently
No delay between requests Rate limited, then challenged
Ignoring access rules Blocked, and indefensible
No identifying user agent Treated as hostile by default
Retrying failures aggressively Escalates a soft limit into a hard one

The last row is the one that turns a recoverable situation into a lasting one.

2. Per-Host Concurrency Is the Main Control

The single most important limit: how many simultaneous requests you make to one host.

One or two concurrent requests per host is conservative and widely tolerated. Twenty is a load test nobody agreed to.

Total throughput comes from crawling many hosts concurrently, not from hammering one. A crawler at two concurrent requests across 200 hosts is doing 400 concurrent requests while remaining polite to each.

3. Delay Between Requests

Beyond concurrency, spacing matters:

  • A fixed delay — simple, and perfectly periodic, which is itself a machine signature.
  • A randomised delay around a mean — better on both counts.
  • Adaptive delay, scaled from the host's observed response time. A site responding in two seconds is under load; a crawler should slow, not persist.
  • Crawl-delay directives where a site publishes one, which is an explicit request to honour.

Adaptive delay is the most respectful because it responds to evidence rather than to a guess.

4. Honouring Access Rules

A site's robots.txt states what it does not want crawled. Honouring it is the baseline expectation, and disregarding it is the fact that makes any subsequent dispute indefensible.

Fetch it before crawling a host, cache it with an expiry, apply the most specific matching rule, and re-check periodically since it changes.

Where a site publishes a crawl delay, use it even when it is slower than you would prefer — it is the site telling you its capacity.

5. Identify Yourself

A user agent naming your crawler and providing a contact URL lets an operator reach you instead of blocking you. That is the practical value: a site that can ask you to slow down often will, rather than escalating.

Conversely, impersonating a browser or a search engine's crawler is both checkable and worse than an honest identifier — search-engine claims fail forward-confirmed reverse DNS immediately.

6. Timing and Load

Crawl during the target's off-peak hours where you can determine them, and respond to signals that you are causing load:

  • Rising response times.
  • Increasing 5xx rates.
  • 429 responses.
  • Connection resets.

All four say the same thing: reduce concurrency and increase delay. Continuing at the same rate converts a warning into a block.

7. A Concrete Default

Reasonable starting settings for an unfamiliar site:

  1. 1–2 concurrent requests per host.
  2. 1–3 seconds between requests, randomised.
  3. Honour robots.txt including any crawl delay.
  4. An identifying user agent with a contact URL.
  5. Exponential backoff on failures, with a small attempt cap.
  6. Stop on repeated 429 or 403 rather than pushing through.
  7. Cache aggressively, so nothing is fetched twice.
  8. Conditional requests with If-Modified-Since or If-None-Match on refetches.

Points 7 and 8 are the ones most often neglected, and they reduce load without reducing coverage at all — which makes them free politeness.

8. Summary

Politeness means limited per-host concurrency, spaced and preferably adaptive delays, honouring published access rules, and identifying yourself with a contact route.

It is self-interest as much as courtesy: throughput comes from crawling many hosts concurrently rather than one host hard, rising latency and 429s are instructions to slow down, and caching plus conditional requests cut load without costing any coverage.