Scrapeless Wiki

What Is Rate Limiting? – Definition and How It Works

Informational P1 what is rate limiting
Overlaps a live page. A published page already targets this keyword: https://www.scrapeless.com/en/blog/rate-limiting. Publishing both risks cannibalisation — decide before this goes live.

Learn what rate limiting is, the common algorithms, which headers communicate limits, and how clients should respond to a 429 correctly.

Rate limiting caps how many requests a client may make in a period. Servers use it to protect capacity, ensure fair access, and control cost — and clients that respect it get better throughput than clients that fight it.

1. What Is Rate Limiting?

It is a policy that rejects requests once a client exceeds an allowed rate.

  • Key idea: bound consumption per identity, per unit of time.
  • Mechanism: count requests against a key — API token, IP, account, endpoint — and reject beyond the threshold.
  • Signal: HTTP 429 Too Many Requests, usually with a Retry-After header.

2. The Algorithms

Algorithm Behaviour Trade-off
Fixed window N requests per calendar minute Simple; allows a 2N burst across a boundary
Sliding window N requests in the trailing 60s Smooth, more state to track
Token bucket Tokens refill at a rate, each request spends one Permits controlled bursts — most common
Leaky bucket Requests drain at a fixed rate Perfectly smooth output, queues inputs
Concurrency limit N requests in flight at once Bounds load rather than rate

Token bucket dominates public APIs because it tolerates natural bursts while holding the long-run average.

The fixed-window boundary problem is worth knowing: with a 100-per-minute limit, a client can send 100 at 11:59:59 and 100 more at 12:00:00 — 200 requests in one second, entirely within policy.

3. What Gets Limited

Limits apply to a key, and which key matters:

  • API token — the norm for authenticated APIs.
  • IP address — the norm for unauthenticated traffic. Shared NAT means unrelated users share a budget.
  • Account — across all of an organisation's tokens.
  • Endpoint — expensive routes get tighter limits than cheap ones.
  • Device fingerprint — survives IP rotation, used where abuse is the concern.

Many services apply several simultaneously, which is why staying under a documented per-token limit can still produce a 429 from a per-endpoint or per-IP rule.

4. The Headers

HTTP/1.1 429 Too Many Requests
Retry-After: 30
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1735689600

Retry-After is the important one: it is the server telling you exactly how long to wait, in seconds or as an HTTP date. Honouring it is both correct and faster than guessing.

Older APIs use X-RateLimit-*; the unprefixed RateLimit-* names are the standardised form. Read both.

5. Responding Correctly

Honour Retry-After first. If the server states a wait, use it. Any backoff you invent is a guess about information you have been given.

Otherwise use exponential backoff with jitter. Double the delay each attempt, and add randomness — without jitter, every blocked client retries in lockstep and re-creates the spike that caused the limit.

Throttle proactively. Watch RateLimit-Remaining and slow down before hitting zero. Reacting to 429s means you are always operating at the edge.

Cap retries. Infinite retry against a sustained limit is a denial-of-service attempt with extra steps.

Distinguish 429 from 403. A 429 is "slow down" and will clear. A 403 is "you are not allowed" and will not — retrying it accomplishes nothing.

6. Rate Limiting vs. Blocking

They are frequently confused and call for opposite responses.

Rate limiting is capacity management: a documented, temporary, mechanical policy that resets. The correct response is to wait and continue at a lower rate.

Blocking is an access decision about who you are — informed by fingerprint, IP reputation or behaviour. It does not reset on a timer, and waiting does not help.

Retrying a block wastes hours. Rotating identity to defeat a rate limit escalates a normal interaction into an adversarial one, and typically converts a temporary throttle into a durable block.

7. Real-World Examples

  • A scraper hammering a 429 in a tight loop, treating a documented backoff as a network error.
  • Ten workers retrying at exactly 60 seconds, colliding on every cycle because nobody added jitter.
  • A client under its per-token limit still refused, because a stricter per-endpoint rule applied.
  • Office users sharing a NAT address exhausting an IP-keyed limit collectively.
  • A pipeline that halved its runtime by slowing down, because steady sub-limit traffic beat bursts and penalties.

8. Summary

Rate limiting bounds how many requests an identity may make in a window, implemented most often as a token bucket and signalled by HTTP 429 with Retry-After. Limits are applied to a key — token, IP, account, endpoint — and several usually apply at once.

The correct client behaviour is to honour Retry-After, otherwise back off exponentially with jitter, throttle proactively against the remaining-quota headers, and cap retries. And keep 429 separate from 403: one is a timer, the other is a decision, and treating them the same wastes effort on the first and provokes escalation on the second.