What Is Delta Lake? – Definition and How It Works

Learn how Delta Lake's transaction log provides ACID guarantees over object storage, and how it compares with Iceberg.

Delta Lake is an open table format that adds ACID transactions, time travel, and schema enforcement to Parquet files in object storage, using an ordered transaction log.

1. The Transaction Log Is the Whole Design

my_table/
├── _delta_log/
│   ├── 00000000000000000000.json     ← commit 0
│   ├── 00000000000000000001.json     ← commit 1
│   ├── 00000000000000000002.json
│   └── 00000000000000000010.checkpoint.parquet
├── part-00000-….snappy.parquet
└── part-00001-….snappy.parquet

Each commit is a JSON file recording actions: files added, files removed, metadata changes. The table's current state is the log replayed in order. Checkpoints periodically compact the log so replay stays fast.

Because a commit is a single atomic file write, and readers replay the log up to the highest committed version, readers never see a partial state.

2. What It Provides

ACID transactions. Concurrent writers are serialised through the log with optimistic concurrency — a conflicting commit fails and retries rather than corrupting.

Time travel. Read any prior version:

SELECT * FROM my_table VERSION AS OF 42;
SELECT * FROM my_table TIMESTAMP AS OF '2026-08-17T00:00:00Z';

Schema enforcement. Writes that do not match the schema are rejected rather than silently absorbed — the opposite of a raw file dump, and usually what you want for a modelled table.

Schema evolution, explicitly permitted when requested.

Upserts and deletes via MERGE, which raw Parquet cannot do.

3. MERGE Is the Practical Feature

For collected data, upsert semantics are what a plain file layout lacks:

MERGE INTO listings t
USING staging s
ON t.source_url = s.source_url
WHEN MATCHED AND s.fetched_at > t.fetched_at THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

That is idempotent ingestion expressed in one statement, with the freshness guard preventing an out-of-order replay from overwriting newer data.

4. Delta Lake Versus Iceberg

Both solve the same problem with comparable mechanisms and have converged closely.

The honest summary: capability differences are narrow and shrinking. Choose by engine and catalogue support in your stack, not by feature list.

5. Where It Fits Collected Data

raw payloads      → object storage, immutable, not a table
cleaned records   → Delta table, MERGE-upserted, partitioned by date
published views   → Delta table, or a warehouse

The properties that matter:

• MERGE for idempotent loads keyed on a natural identifier.
• Time travel to compare a corrected extraction against what was published.
• Schema enforcement so source drift fails loudly at the modelled layer.
• Atomic commits so a failed load leaves nothing behind.

Note the division of labour: schema enforcement is right for the modelled table and wrong for raw payloads, which must accept anything. That is why raw stays as plain objects.

6. Operational Notes

1. VACUUM old files on a schedule, respecting the retention you need for time travel.
2. OPTIMIZE to compact small files, which frequent commits produce.
3. Understand the retention trade — vacuuming aggressively shortens how far back you can time travel.
4. Expect commit conflicts with concurrent writers, and retry.
5. Keep raw data outside the table.
6. Record the extraction version as a column so reprocessing is attributable.

Point 3 is the one people discover awkwardly: VACUUM with a short retention silently removes the ability to read older versions.

7. Why Table Formats Matter Here At All

Collected data has two properties that plain files handle badly: it arrives incrementally and needs upserts, and it gets reprocessed when extraction improves. A table format provides idempotent upserts and an auditable before-and-after — both of which otherwise get hand-rolled, badly.

8. Summary

Delta Lake adds ACID transactions, time travel, schema enforcement, and MERGE upserts to Parquet in object storage via an ordered transaction log with checkpoints.

MERGE with a freshness guard gives idempotent ingestion in one statement, and time travel makes a corrected extraction auditable against what was published — while VACUUM retention directly bounds how far back time travel reaches, and raw payloads belong outside the table where schema enforcement would reject them.

Potential Related Glossary

• Data Structure
• Data Wrangling
• Data Fusion
• Unstructured Data