What Is Change Data Capture? – Definition and How It Works
Learn what CDC does, how log-based capture differs from polling, and how the idea translates to web sources that expose no change feed.
Change data capture identifies and propagates changes in a data source rather than re-reading everything. It exists because full reloads are wasteful and slow once a dataset is large.
1. The Approaches
| Approach | Mechanism | Trade-off |
|---|---|---|
| Log-based | Read the database's transaction log | Lowest impact, captures deletes, needs access |
| Trigger-based | Database triggers write change records | Reliable, adds write overhead |
| Timestamp polling | Query rows changed since a marker | Simple, misses deletes |
| Full diff | Compare snapshots | Works anywhere, most expensive |
Log-based capture is the preferred method where the source is a database you can reach, because it observes changes without adding load to queries.
2. Why Deletes Are the Hard Part
Timestamp polling finds rows that changed and cannot find rows that vanished. A record deleted at the source simply stops appearing, and a consumer polling by timestamp never learns of it.
Log-based capture sees deletes explicitly. Where only polling is available, detecting deletions requires periodically comparing full key sets — which is expensive and often skipped, leaving stale records forever.
3. Ordering and Idempotency
Changes must be applied in order, or an older value can overwrite a newer one. Log-based capture provides ordering; polling generally does not.
Consumers should also be idempotent, since delivery is usually at-least-once. Upserts keyed on a stable identifier handle both concerns.
4. Translating the Idea to Web Sources
Web sources expose no transaction log, so the available options are narrower:
lastmodin a sitemap, where it is honest — the closest thing to a change feed.- Conditional requests with
ETagorIf-Modified-Since, which tell you cheaply that nothing changed. - Content hashing — fetch, hash the extracted content, and compare against the last hash.
- A changes endpoint, where an API provides one.
Sitemap lastmod plus conditional requests is the cheapest combination and worth verifying before relying on it.
5. Content Hashing in Practice
Hash the extracted content rather than the raw response. Raw HTML changes constantly — timestamps, session tokens, advertisement identifiers — while the content does not.
Hashing extracted fields detects real changes and ignores noise, which is the difference between a useful change signal and a permanently dirty one.
6. Detecting Deletions on the Web
A page returning 404 or 410 is an explicit deletion. A page that quietly stops being linked, or starts soft-404ing, is not.
Periodically re-checking known URLs and marking those that fail is the web equivalent of a full key comparison — expensive, necessary, and frequently omitted.
7. Why It Matters for Collected Data
Without change detection, every run re-fetches everything: more load on the source, more bandwidth, and more cost. With it, a run touches only what moved.
It also improves the data: knowing when something changed is often more valuable than knowing its current value.
8. Summary
Change data capture propagates changes rather than reloading, with log-based capture preferred where a database is accessible because it captures ordering and deletions.
Web sources have no log, so the equivalents are honest lastmod values, conditional requests, and hashing extracted content — with periodic full re-checks the only reliable way to detect that something has gone.