What Is a Columnar Format? – Definition and How It Works
Learn how column-oriented storage differs from row-oriented, why it compresses and filters better, and when row storage is still right.
A columnar format stores all values of one column together rather than storing each record's fields together. That single layout change alters compression, query cost, and what kinds of work the format suits.
1. The Layout Difference
Row-oriented: [id1,name1,price1][id2,name2,price2][id3,name3,price3]
Column-oriented: [id1,id2,id3][name1,name2,name3][price1,price2,price3]
Reading three columns of a fifty-column dataset touches three columns' worth of bytes rather than the whole file.
2. Why It Compresses Better
Values in one column share a type and are frequently similar — repeated categories, sequential timestamps, prices in a narrow band.
That enables encodings row layouts cannot use: dictionary encoding for repeated values, run-length encoding for repetition, and delta encoding for sequences. General compression then applies on top.
3. Why It Filters Faster
Columnar files record per-column statistics — minimum, maximum, null count — for each block of rows. A query filtering on a column can skip whole blocks whose range cannot match.
Combined with partitioned directories, a filtered query may read a small fraction of the dataset. Row formats must read everything to find anything.
4. Where Row Storage Wins
| Workload | Better layout |
|---|---|
| Read or write one whole record | Row |
| Frequent single-record updates | Row |
| Transactions | Row |
| Scan many rows, few columns | Column |
| Aggregations | Column |
| Compress a large archive | Column |
Transactional systems are row-oriented for exactly this reason: fetching one customer's full record touches one contiguous region.
5. The Common Formats
Parquet dominates for analytical storage; ORC is common in some ecosystems; Arrow is the in-memory columnar representation that libraries exchange data through with little copying.
Arrow is worth knowing separately: it is why modern data frame libraries can hand data to each other cheaply.
6. Practical Consequences
- Write in batches. Columnar formats are built for bulk writes, not row-at-a-time appends.
- Avoid tiny files. Many small files are far slower to query than a few large ones.
- Partition on filter columns, usually date and source.
- Types are stored, so identifiers keep their leading zeros and dates stay dates.
7. Where It Fits in a Collection Pipeline
Land raw payloads compressed, write cleaned records as columnar files partitioned by date and source, and export a row-oriented format only when a person or a system requires one.
That gives cheap storage, fast filtered queries, and preserved types, while keeping the text formats where they belong — at the collection and export boundaries.
8. Summary
Columnar formats store values column by column, enabling type-aware compression and block-level filtering that make analytical queries read a fraction of the data.
Row storage remains correct for whole-record access and transactions. For stored collected data, columnar is the default — written in batches, partitioned on filter columns, and consolidated into reasonably sized files.