What Is a Crawl Delay? – Definition and How It Works
Learn what crawl-delay means in robots.txt, which crawlers honour it, how to pace politely without it, and what to do when none is stated.
Crawl-delay is a directive in robots.txt requesting a minimum interval between requests from one client. It expresses a site's tolerance for automated traffic in seconds.
1. The Directive
User-agent: *
Crawl-delay: 10
Disallow: /admin/
That asks any automated client to wait ten seconds between requests. It is a request rather than an enforced limit.
2. It Is Not Part of the Original Standard
Crawl-delay was an extension rather than part of the original robots specification, and support varies. Some major crawlers ignore it entirely and manage their own crawl rate instead, sometimes offering a control in a webmaster interface.
That inconsistency does not make it meaningless. For anyone writing a collector, a stated delay is the clearest possible statement of what the site considers acceptable.
3. Honour It When Present
If a site states a delay, respect it. It is the site telling you its tolerance without your having to guess, and following it removes the most common reason a site escalates to blocking.
A delay of ten seconds means a maximum of six requests per minute — plan collection volume around that rather than treating it as a suggestion.
4. When No Delay Is Stated
Absence is not permission for unlimited speed. Reasonable defaults:
- One request at a time per host, not several in parallel.
- A short pause between requests, in the region of a second or two for ordinary pages.
- Slower for expensive pages — search results and generated reports cost the site more than static ones.
- Slower still if latency rises, which usually precedes refusal.
5. Watch the Signals Instead
A site communicates its tolerance in several ways beyond robots.txt:
| Signal | Meaning |
|---|---|
Retry-After on a 429 |
An explicit wait instruction |
| Rate-limit headers | Your remaining budget |
| Rising response times | Approaching capacity |
| Challenges appearing | Confidence dropping, often due to pace |
Reacting to these is more useful than any fixed delay, because they reflect current conditions.
6. Concurrency Matters as Much as Delay
A one-second delay across sixteen parallel workers is sixteen requests per second to one host. The delay applies per client, and a fleet is one client from the destination's perspective.
Cap concurrency per host and coordinate pacing across workers, or the delay is decorative.
7. Why Politeness Is Practical
A site that does not notice your traffic has no reason to block it. Most escalation from rate limiting to outright refusal follows sustained load, which is precisely what pacing avoids.
Collecting at the frequency a decision requires — rather than the maximum achievable — is usually both sufficient and unremarkable.
8. Summary
Crawl-delay in robots.txt requests a minimum interval between requests, is honoured inconsistently by large crawlers, and should be respected by anyone writing a collector.
Where none is stated, default to serial requests with a short pause, cap concurrency per host, coordinate across workers, and let the site's own signals — Retry-After, rate headers, latency — set the pace.