What Is Data Lineage? – Definition and How It Works

Learn what lineage records, the difference between table-level and field-level tracking, and why collected data needs it more than most.

Data lineage is the record of where data came from and what happened to it. It answers two questions: for any output, which inputs produced it, and for any input, what depends on it.

1. The Two Directions

Backwards — given a value in a report, which source produced it and through which transformations. This is what you need when a number is disputed.

Forwards — given a source that changed or broke, which downstream outputs are affected. This is what you need during an incident.

Both matter, and systems that record only one are half-useful.

2. Levels of Granularity

Field-level is the usual target. Record-level is expensive and occasionally necessary — regulated environments, or any dataset where an individual value may need to be traced.

3. Why Collected Data Needs It Especially

Web sources change without notice, extraction improves over time, and observations vary by location and device.

Without lineage, a value in a report cannot be traced to the page, the extraction version, or the collection context that produced it. When someone asks whether a figure is right, the honest answer becomes "we cannot tell".

4. What to Record Per Observation

• The source URL, and the canonical URL where they differ.
• The fetch timestamp.
• The egress region, where content varies by location.
• The extraction version or schema version.
• The pipeline run identifier.
• A reference to the retained raw payload.

Those six fields make any stored value traceable, and they cost almost nothing to store.

5. Pipeline-Level Lineage

Beyond per-record fields, the pipeline itself should record which run produced which partition, which code version it ran, and which upstream inputs it consumed.

That is what allows a bad transformation to be identified and its output reprocessed selectively rather than rebuilding everything.

6. It Enables Reprocessing

Lineage plus retained raw payloads is what makes extraction improvable. When a parsing error is found, lineage identifies exactly which records are affected and the raw payloads allow them to be redone.

Without either, the only options are recollecting — impossible for web sources — or leaving known-bad data in place.

7. Automating It

Lineage recorded manually drifts immediately. It should be emitted by the pipeline as a side effect of running: identifiers propagated through stages, and metadata written alongside outputs.

Anything requiring a human to update a diagram is documentation rather than lineage.

8. Summary

Data lineage records where data came from and what transformed it, traceable backwards for auditing and forwards for impact analysis.

For collected data it is what makes any value defensible, and combined with retained raw payloads it is what makes extraction errors fixable rather than permanent — so it must be emitted automatically rather than maintained by hand.

Potential Related Glossary

• Data Fusion
• Data Wrangling
• Web Crawling
• Data Structure