Scrapeless Wiki

What Is Bandwidth Throttling? – Definition and How It Works

Informational P3 what is bandwidth throttling

Learn what bandwidth throttling is, who applies it, how it differs from rate limiting, and how to detect and handle it.

Bandwidth throttling deliberately limits the data-transfer rate of a connection. It slows transfer rather than rejecting requests, which makes it much harder to notice than an outright block.

1. Throttling Versus Rate Limiting

Bandwidth throttling Rate limiting
Limits Bytes per second Requests per interval
Response Slow but successful Rejected, usually 429
Visibility Silent Explicit status code
Effect Long transfers Request frequency
Detection Requires measurement Immediate

Rate limiting says no. Throttling says yes, slowly β€” and that difference is why throttling produces timeouts and truncated data rather than clean errors.

2. Who Applies It

Origin servers, limiting per-connection transfer to protect capacity.

CDNs and edge networks, often as a graduated response to traffic they consider excessive β€” slowing before blocking.

Proxy providers, enforcing plan bandwidth limits, which is the most common cause in collection work.

ISPs, shaping traffic by protocol or after a data cap.

Your own infrastructure, where container network limits or a saturated uplink throttle you without anyone intending it.

3. Why It Is Worse Than a Block

A block is honest. Throttling degrades in ways that corrupt data:

  • Timeouts on large responses, while small ones succeed β€” so the failure looks content-dependent.
  • Truncated bodies where a client accepts a partial read, producing valid-looking HTML with content missing.
  • Browser waits expiring before a page finishes loading, so extraction runs against a half-rendered DOM.
  • Retry storms, since each attempt is slow rather than failed, multiplying the load.

The truncation case is the serious one: it is a silent data-quality failure wearing a 200 status.

4. Detecting It

Measure transfer rate, not just total time:

curl -o /dev/null -s -w 'size=%{size_download} time=%{time_total} rate=%{speed_download}\n' https://example.com/large-file

The signals that indicate throttling rather than an ordinary slow server:

  • A transfer rate that is suspiciously constant β€” a natural bottleneck varies; an enforced cap does not.
  • Rate falling after a threshold of transferred bytes, which is a burst allowance being exhausted.
  • Small responses fast, large ones slow, at the same rate ceiling.
  • The same URL fast from a different network.

A steady, round-numbered rate is close to proof.

5. Handling It

Set read timeouts generously enough for the throttled rate, or every large response fails. A read timeout, not a total timeout, is the correct control β€” it detects a stall while permitting slow progress.

Verify completeness rather than trusting the status. A byte count against Content-Length, a JSON parse, or a closing-tag check turns silent truncation into a caught error.

Reduce concurrency. Where a cap is per-connection, more connections help; where it is per-account or per-address, they simply divide the same budget and make every request slower.

Request less. Skip images and media in a browser context, prefer a JSON endpoint over a rendered page, and accept compression β€” throttling counts bytes, so smaller responses cost proportionally less.

That last point is the most effective single mitigation: blocking image and font requests in a headless browser often removes the large majority of transferred bytes.

6. When It Is Your Proxy Plan

Bandwidth-metered proxy plans throttle or bill once a quota is consumed, and browser traffic consumes bandwidth far faster than plain HTTP because every asset counts.

Practical measures: block unnecessary resource types, cache static assets locally across runs, use plain HTTP wherever rendering is not required, and monitor bytes per collected record rather than bytes in total β€” that ratio is what reveals a wasteful pipeline.

7. Distinguishing It From a Slow Server

Before concluding anything, check the alternatives: a genuinely slow origin, a slow database behind it, network congestion on your side, a distant proxy adding round trips, or DNS delay.

The discriminator is the shape of the rate curve. A constant ceiling regardless of conditions is throttling; variability that tracks load is congestion.

8. Summary

Bandwidth throttling caps transfer rate instead of rejecting requests, so it appears as timeouts on large responses and, worse, as truncated bodies behind successful status codes.

Detect it by measuring transfer rate and looking for a suspiciously constant ceiling; handle it with generous read timeouts, explicit completeness checks, and by transferring fewer bytes β€” blocking images and fonts in a browser context is usually the largest available saving.