What Is Apache Iceberg? – Definition and How It Works
Learn how Iceberg adds transactions, time travel, and schema evolution to files in object storage.
Apache Iceberg is an open table format. It layers a metadata structure over data files in object storage so they behave like a real table — with transactions, schema evolution, and snapshots.
1. The Problem It Solves
A directory of Parquet files in object storage is cheap and awkward. It has no transactions, so a reader can see a half-written update. It has no schema management. Listing millions of files to plan a query is slow. And there is no way to see what the table looked like yesterday.
Iceberg adds a metadata layer that fixes all four.
2. The Structure
catalog → table metadata (schema, partitioning, snapshot list)
└── manifest list (per snapshot)
└── manifest files (groups of data files + statistics)
└── data files (Parquet, ORC, or Avro)
The key property: a snapshot is an immutable pointer to an exact set of data files. Committing a change writes new metadata pointing at a new set — it never mutates the old one.
3. What That Buys
Atomic commits. A writer publishes a new snapshot in one operation, so readers see either the old state or the new one, never a partial one. That is the property a plain directory cannot offer.
Time travel. Because old snapshots remain, you can read the table as of a timestamp or a snapshot id — which for collected data means reproducing exactly what a report saw last month.
Schema evolution. Add, drop, rename, and reorder columns safely, because columns are tracked by id rather than by name or position. Renaming a column does not rewrite data.
Hidden partitioning. Partitioning is defined in metadata rather than encoded in file paths, so queries need not know the partition scheme and the scheme can change without rewriting the table.
Fast planning. Manifest statistics let the engine skip files without listing them.
4. Why Hidden Partitioning Matters
In a path-based layout, a query must filter on the partition column exactly as the paths encode it, and changing the scheme means rewriting everything.
Iceberg records the partition transform in metadata, so a query filtering on a timestamp gets day-level pruning automatically, and the partitioning can be changed later for new data while old data stays queryable.
That removes one of the most persistent operational annoyances of file-based lakes.
5. Where It Fits Collected Data
The natural arrangement:
raw payloads → object storage, immutable, never a table
cleaned records → Iceberg table, partitioned by fetch date
modelled/published → Iceberg table, or a warehouse
The properties that earn their place:
- Atomic commits so a failed load leaves no partial state.
- Time travel so a corrected extraction can be compared against what was previously published.
- Schema evolution for the source drift that collected data always has.
- Snapshot expiry to bound storage as history accumulates.
Time travel is the one that pairs best with raw retention: reprocess raw payloads, publish a new snapshot, and the difference between snapshots is the correction, auditable rather than asserted.
6. Iceberg Versus Delta Lake
Both are open table formats solving the same problem with similar mechanisms — a metadata log over files, atomic commits, time travel, schema evolution.
The practical differences are ecosystem and engine support rather than capability, and both have converged substantially. The decision usually follows whichever your query engines and catalogue support best rather than a feature comparison.
7. Operational Notes
- Expire old snapshots on a schedule, or metadata and storage grow without bound.
- Compact small files. Frequent small commits produce many small files and slow queries.
- Choose a catalogue deliberately — it is the component that makes commits atomic.
- Do not put raw payloads in a table. Keep them as immutable objects; tables are for modelled data.
- Record the extraction version as a column, so a reprocessing is attributable.
Small-file accumulation is the most common operational complaint, and it follows directly from frequent commits — which a collection pipeline naturally produces.
8. Summary
Apache Iceberg is an open table format that adds atomic commits, time travel, id-based schema evolution, and hidden partitioning to data files in object storage via an immutable snapshot-based metadata layer.
For collected data it gives loads that cannot leave partial state and snapshots that make a corrected extraction auditable against what was previously published — provided snapshots are expired and small files compacted, since frequent commits produce both.