What Is Parquet? – Definition and How It Works
Learn what Parquet is, how columnar layout and row groups work, why compression and predicate pushdown help, and when to choose it.
Parquet is a columnar file format for analytical data. It stores values column by column rather than row by row, which changes both how much space data occupies and how quickly it can be queried.
1. Columnar Layout
A row-oriented file stores each record's fields together. A columnar file stores all values of one column together.
That matters because analytical queries read few columns of many rows. Reading three columns from a fifty-column table touches three columns' worth of bytes rather than the whole file.
2. Why It Compresses Well
Values in one column are the same type and often similar — repeated categories, sequential timestamps, prices in a narrow range.
Encoding schemes exploit that directly: dictionary encoding for repeated values, run-length encoding for repetition, and delta encoding for sequences. General compression then applies on top, and the combined effect is frequently several times smaller than the equivalent CSV.
3. Row Groups and Statistics
A Parquet file is divided into row groups, each holding a chunk of rows column-wise, with metadata recording per-column minimum, maximum, and null count.
A query filtering on a column can read that metadata and skip entire row groups whose range cannot match — predicate pushdown. On partitioned datasets the combination of directory pruning and row-group skipping means a filtered query may read a small fraction of the data.
4. Types Are Preserved
Unlike CSV, Parquet stores types in the file: integers stay integers, timestamps stay timestamps, and nulls are explicit and distinct from empty strings.
That removes the class of bug where identifiers lose leading zeros or become floats on a round trip.
5. Nested Data
Parquet supports nested structures — lists and maps — so a record with an array of tags does not require flattening. Support quality varies by tool, so it is worth verifying with the specific engine in use before relying on deep nesting.
6. When to Use It
| Situation | Format |
|---|---|
| Analytical queries over many rows | Parquet |
| Handing a file to a colleague | CSV |
| Streaming appends of records | JSON Lines |
| API payloads | JSON |
| Long-term storage of collected data | Parquet |
A common and effective pattern for collection: land raw payloads compressed, write cleaned records as Parquet partitioned by date and source, and export CSV only when a person needs one.
7. Practical Notes
- File size matters. Many tiny files are slow; consolidate into reasonably sized ones.
- Partition by the columns you filter on, usually date and source.
- Compression is configurable; the default is usually fine, and options trade size against decode speed.
- It is not human-readable, so keep a sample export for inspection.
8. Summary
Parquet stores data column-wise in row groups with per-column statistics, preserving types and compressing far better than text formats.
Those properties make it the default for stored analytical data: smaller on disk, faster to filter, and free of the type-loss problems that make CSV round trips unreliable.