What Is a Rate Limiter? – Definition and How It Works
Learn what a rate limiter does, the algorithms behind it, where it is enforced, and how distributed limiting is coordinated.
A rate limiter caps how frequently an operation may occur. It protects capacity, prevents one caller degrading service for others, and turns an implicit resource into an explicit budget.
1. The Algorithms
| Algorithm | Mechanism | Trade-off |
|---|---|---|
| Fixed window | Count per clock interval | Bursts cluster at boundaries, allowing double the rate briefly |
| Sliding window log | Timestamps of recent requests | Exact, and memory grows with volume |
| Sliding window counter | Weighted blend of two windows | Nearly exact, small memory |
| Token bucket | Tokens refill steadily, requests spend one | Allows controlled bursts; the usual default |
| Leaky bucket | Requests drain at a constant rate | Strictly smooth, queues instead of bursting |
Token bucket dominates because it tolerates short bursts while bounding the sustained rate, which matches how real clients behave.
2. The Boundary Problem
A fixed window permits a client to send its full allowance at the end of one window and again at the start of the next — twice the intended rate across the boundary.
Sliding windows and token buckets both avoid this, which is why fixed windows survive mainly for their simplicity.
3. What the Limit Is Keyed On
- API key or account — the usual unit, tied to a plan.
- Address — protects unauthenticated surfaces, and penalises shared networks.
- Endpoint — expensive operations get tighter budgets.
- User plus endpoint — finer control, more state.
Keying on address is the reason a well-behaved user behind a shared network can be limited by someone else's traffic.
4. Rate and Concurrency Are Different
A rate limit bounds requests per period. A concurrency limit bounds simultaneous in-flight requests. A client can be well under one and exceed the other.
Hitting a concurrency cap while comfortably under the rate limit is common and confusing if only the rate is being tracked.
5. Distributed Limiting
With several servers, each holding its own counter, the effective limit is multiplied by the number of servers.
Shared state — typically a fast in-memory store holding counters or buckets — is the usual answer. The trade is a network round trip per decision, which is why some systems accept approximate limiting with local counters and periodic synchronisation.
6. Communicating the Limit
Well-behaved services publish the budget in response headers and return 429 with Retry-After when it is exhausted.
That turns limiting from a surprise into a contract: a client that reads the headers can pace itself and never be refused, which is better for both sides.
7. Implementing One as a Client
Clients need limiters too, to stay within a destination's published limits. A local token bucket per host, with a shared counter across workers, prevents the common failure where each worker is individually compliant and the fleet is not.
8. Summary
A rate limiter caps operation frequency using fixed windows, sliding windows, or token buckets, keyed on an identity such as an account, address, or endpoint.
Token buckets suit real traffic best, concurrency is a separate limit, distributed enforcement needs shared state, and publishing the budget in headers is what lets clients comply rather than discover the limit by hitting it.