What Is Gzip Compression? – Definition and How It Works
Learn how gzip works in HTTP and on files, the difference between Content-Encoding and Transfer-Encoding, and its limits.
Gzip is the most widely supported compression format on the web. It uses DEFLATE — LZ77 matching plus Huffman coding — wrapped in a container with a header and a checksum.
1. In HTTP
Accept-Encoding: gzip, deflate, br
Content-Encoding: gzip
HTML typically compresses to a small fraction of its original size, because markup is highly repetitive. That is why gzip is enabled almost universally for text responses.
2. Content-Encoding Versus Transfer-Encoding
Two headers, frequently confused:
| Header | Meaning |
|---|---|
Content-Encoding: gzip |
The resource is compressed; end to end |
Transfer-Encoding: gzip |
The transfer is compressed; hop by hop |
Content-Encoding is what is used in practice. It matters because Content-Length, ETags, and range requests apply to the compressed representation, so a byte range is a range of compressed bytes.
3. Client Handling
Most HTTP clients decompress transparently when they advertised the encoding, which makes gzip invisible in normal use:
import requests
r = requests.get(url) # gzip offered and decoded automatically
print(r.headers.get("content-encoding"), len(r.content))
Two consequences worth knowing. len(r.content) is the decompressed size while Content-Length was the compressed one, so they will not match. And a client that advertises an encoding it cannot decode receives binary garbage rather than an error — the failure mode to avoid by only offering what you can handle.
4. As a File Format
gzip -9 large.ndjson # compress, replacing the file
gzip -dc large.ndjson.gz | head # stream-decompress without writing
Gzip's important limitation for data work is that a single gzip stream is not splittable: it cannot be decompressed from an arbitrary offset, so one large file cannot be processed in parallel.
The standard fix is many moderately sized files rather than one huge one — which also makes partial reprocessing possible.
5. Compression Levels
Levels 1–9 trade time for size, with the useful range narrower than it looks. Level 1 is dramatically faster than 9 while giving up a modest amount of ratio; level 9 rarely justifies its cost on data compressed once and read once.
Decompression speed is essentially independent of level, so for archives written once and read many times, the higher level is worth it.
6. Where Gzip Loses
Brotli compresses web text smaller, thanks to its built-in dictionary of common web strings.
Zstandard offers a much better speed-to-ratio curve and supports framing for splittable parallel reads.
Columnar formats with per-column compression beat gzipped row-oriented text substantially for analytical data, because similar values sit together.
Gzip's advantage is not ratio or speed — it is that everything, everywhere, supports it.
7. Storing Collected Data
Practical defaults:
- Raw HTML archives — gzip or Brotli at a high level; text compresses very well and storage dominates the cost.
- NDJSON records — gzip in bounded chunks so files stay processable in parallel.
- Analytical copies — a columnar format instead.
- Already-compressed media — leave it alone; recompressing images or video wastes CPU for nothing.
8. Summary
Gzip is DEFLATE plus a container, universally supported, and the default compression for HTTP text where Content-Encoding: gzip marks the resource itself as compressed.
Only advertise it if your client decodes it, expect Content-Length to describe compressed bytes while the body you read is decompressed, and remember a single gzip stream is not splittable — so write many bounded files rather than one large one.