What Is Chunked Transfer Encoding? – Definition and How It Works
Learn how chunked transfer encoding streams a response of unknown length, and why a truncated stream is a silent failure.
Chunked transfer encoding sends an HTTP/1.1 response body as a series of length-prefixed pieces, so a server can start sending before it knows the total size. It replaces Content-Length with a terminating zero-length chunk.
1. The Wire Format
HTTP/1.1 200 OK
Content-Type: text/html
Transfer-Encoding: chunked
1a
<html><body><h1>Hello
10
world</h1></body>
0
Each chunk is a hexadecimal size, CRLF, that many bytes, CRLF. A chunk of size 0 ends the body, optionally followed by trailer headers.
2. Why Servers Use It
A dynamically generated page has no known length until it is finished. The alternatives are buffering the whole response to compute Content-Length — which delays the first byte and costs memory — or streaming with chunked encoding.
Streaming lets the browser begin parsing and requesting sub-resources while the server is still generating, which is why it is the default for dynamic responses.
3. The Failure That Matters
The terminating zero-length chunk is the only signal that a body is complete. If a connection drops mid-response, a client that ignores this receives a partial body with a 200 status.
That is the classic silent truncation: a successful status code, valid-looking HTML, and missing content. Extraction returns fewer rows, or none, and nothing in the response says anything went wrong.
A well-behaved HTTP library raises on an incomplete chunked read. Verify yours does, because the ones that do not turn a network error into wrong data.
4. Detecting Truncation
Where correctness matters, add a content-level check rather than trusting transport alone:
- HTML should end with a closing tag, not mid-element.
- JSON must parse; a truncated document will not.
- A results page should contain a plausible number of items compared with its stated count.
- A page that declares a total should have the rows to match.
The check need not be sophisticated. Any assertion that a truncated body fails is worth more than none.
5. Interaction With Compression
Content-Encoding: gzip and Transfer-Encoding: chunked combine routinely: the body is compressed, then the compressed stream is chunked.
The ordering means a truncated chunked stream produces a truncated compressed stream, and decompression usually errors — which is fortunate, because it converts a silent truncation into a loud one. Do not suppress that error.
6. HTTP/2 and HTTP/3
Chunked encoding is an HTTP/1.1 mechanism. HTTP/2 and HTTP/3 have framing built into the protocol, so Transfer-Encoding: chunked is not used — indeed it is forbidden.
Streaming still happens; it is expressed by the protocol's own DATA frames rather than by an encoding header. A client that special-cases chunked encoding should not expect to see it on those connections.
7. Streaming Consumption
Chunked responses invite processing as they arrive, which is what makes very large downloads feasible without holding them in memory:
import requests
with requests.get(url, stream=True, timeout=(10, 60)) as r:
r.raise_for_status()
with open("out.bin", "wb") as f:
for chunk in r.iter_content(chunk_size=65536):
f.write(chunk)
The two-value timeout matters here: a read timeout applies between chunks, so a stalled stream fails rather than hanging indefinitely.
8. Summary
Chunked transfer encoding sends an HTTP/1.1 body as length-prefixed chunks terminated by a zero-length chunk, letting a server stream a response of unknown size.
Its important consequence is that only the terminator proves completeness — so use a client that raises on an incomplete stream, add content-level completeness checks, never suppress decompression errors, and set a read timeout so a stalled stream fails instead of hanging.