Scrapeless Wiki

HTTP 429 Too Many Requests – What It Means and How to Fix It

Troubleshooting P1 http 429 meaning

Learn what HTTP 429 Too Many Requests means, how rate limits work, how to read Retry-After, and how to back off correctly instead of making things worse.

A 429 is the most cooperative rejection on the web. The server is not refusing you, questioning your identity, or claiming anything broke β€” it is telling you that you are going too fast and asking you to wait. Handled properly it costs a short delay. Handled badly it escalates into a durable block.

1. What Does HTTP 429 Mean?

429 Too Many Requests indicates the client has sent more requests than the server's rate limit permits within a given window.

  • Key idea: the request was valid; the frequency was not.
  • Mechanism: the server counts requests per identifier β€” API key, IP address, user account, or token β€” and rejects those over the threshold.
  • Goal: protect capacity and enforce fair use.

429 is explicitly temporary. The same request will usually succeed later, unchanged.

2. Common Causes

Genuine limit exceeded. The documented ceiling is 100 requests per minute and you sent 150. The simplest case, and the easiest to fix.

Concurrency, not volume. Ten parallel workers each staying under the limit collectively breach it. Per-worker limiting does not compose into a global limit.

Retries amplifying load. A failure triggers an immediate retry, which fails, which retries. Retry storms convert a small problem into sustained overload, and the server responds accordingly.

Shared identity. Rate limits keyed to IP address are shared by everyone behind that address β€” a NAT gateway, an office network, or a proxy pool where several of your own jobs exit through the same node.

Burst versus sustained limits. Many APIs enforce both. Staying under 1,000 per hour still fails if 900 arrive in the first minute.

Unauthenticated requests. Anonymous limits are commonly far tighter than authenticated ones. Forgetting to attach a key can drop your allowance by an order of magnitude.

3. How to Diagnose It

Read Retry-After. It is the server telling you exactly how long to wait, either in seconds or as an HTTP date. Ignoring it and guessing is the most common handling mistake.

Read the rate-limit headers. Most APIs expose the state:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1735689600

Newer services use the standardised RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset. Either way, logging these turns rate limiting from a surprise into something you can see approaching.

Check what the limit is keyed to. Per key, per IP, or per endpoint changes the fix entirely. If it is per IP, adding API keys achieves nothing.

Count your actual concurrency. Not your intended concurrency β€” measure requests in flight. This is where the answer usually is.

4. How to Fix It

Honour Retry-After first. If the server states a wait, wait that long. Nothing else you do matters more.

Use exponential backoff with jitter. Retry after 1s, 2s, 4s, 8s, with a random component so parallel clients do not resynchronise and hit simultaneously β€” a thundering herd turns one 429 into a repeating cycle.

Rate-limit on your side. A token bucket or leaky bucket enforced across all workers, not per worker. Deliberately staying at 80% of the documented limit leaves headroom for clock skew and bursts.

Cap concurrency globally. A shared semaphore across the whole job, not one per thread.

Cache and deduplicate. The cheapest request is the one you do not send. Repeated fetches of unchanged resources are pure waste against your quota.

Use bulk endpoints where offered. One request for 100 records beats 100 requests, and many APIs price them the same.

Request a higher limit. For a legitimate integration, asking is often faster and cheaper than engineering around the ceiling.

Do not rotate IPs to evade it. For an API you have credentials with, that is circumvention rather than a fix, and it typically converts a temporary 429 into a permanent 403.

Code Meaning Retry?
429 Too Many Requests Rate limit exceeded Yes, after Retry-After
503 Service Unavailable Server overloaded or down Yes, with backoff
403 Forbidden Refused outright No β€” retrying may worsen it
408 Request Timeout Client too slow to send Yes
509 Bandwidth Limit Exceeded Non-standard, bandwidth cap Usually not soon

The important distinction is 429 versus 403. A 429 is an invitation to come back; a 403 is not. Retrying a 403 aggressively is how clients earn IP bans.

6. Real-World Examples

  • A job that works at 5 workers and 429s at 20. The per-request logic was never the issue; total concurrency was.
  • 429 exactly on the hour. A fixed window reset, with everything queued firing simultaneously the moment it opens. Jitter fixes it.
  • 429 only in production. Multiple instances sharing one limit. Local testing with one process could never reproduce it.
  • Intermittent 429 from a proxy pool. Several exit nodes share an address, so per-IP limits are hit despite modest per-worker rates.
  • 429 escalating to 403. The server stopped asking and started refusing. Backoff was missing or too aggressive.

7. Summary

429 means slow down, and it is the only rejection that tells you precisely how. Read Retry-After, respect it, and back off exponentially with jitter when it is absent.

Most persistent 429s are concurrency problems rather than request-rate problems β€” the fix is a shared limiter across all workers rather than a delay inside each one. Log the rate-limit headers so you can see the ceiling approaching, cache to avoid spending quota on data you already have, and treat repeated 429s as a signal to slow down rather than to try harder, because the next status code after ignoring one is usually 403.