What Is a Data Pipeline? – Definition and How It Works
Learn what a data pipeline is, its stages, batch versus streaming, idempotency and backfills, and what makes one operable rather than merely working.
A data pipeline moves data from where it is produced to where it is used, transforming it along the way. Building one that runs is straightforward; building one that can be re-run, debugged, and trusted is the actual work.
1. The Stages
| Stage | Responsibility |
|---|---|
| Ingest | Fetch from the source, with retrieval metadata |
| Land | Store the raw payload unchanged |
| Transform | Parse, clean, normalise, and type |
| Validate | Check structure, values, and set-level expectations |
| Store | Write to the destination in a queryable form |
| Publish | Make it available, with freshness recorded |
Landing raw data before transformation is the stage most often skipped and most often regretted — without it, improving extraction means refetching everything.
2. Batch Versus Streaming
Batch processes bounded sets on a schedule. Simple to reason about, easy to re-run, and latency is bounded by the interval.
Streaming processes records as they arrive. Lower latency, and substantially more complexity — ordering, late arrivals, and windowing all become explicit concerns.
Most collection work is legitimately batch. Choosing streaming for a daily report buys complexity with no benefit.
3. Idempotency Is the Central Property
Running the same stage twice must produce the same result, not duplicates. Pipelines are re-run constantly — after failures, after fixes, after upstream corrections.
The usual mechanisms are deterministic identifiers per record, upserts rather than appends, and partitioning by the period being processed so a re-run replaces exactly that partition.
4. Backfills Are Normal
Sooner or later a transformation is wrong and history must be reprocessed. That is only possible if the raw payloads were retained and the transformation is a pure function of them.
A pipeline that transforms during ingestion and discards the original cannot be backfilled — only recollected, which for web sources means the past is unrecoverable.
5. What Makes One Operable
- Observability. Record counts, durations, and failure rates per stage and per source.
- Alerting on absence. A stage that did not run is as serious as one that failed.
- Lineage. For any output record, which input produced it.
- Partitioning. Process and replace by period rather than mutating in place.
- Schema versioning, so changes are deliberate rather than discovered.
6. Failure Handling
Distinguish transient failures from permanent ones. A network timeout is worth another attempt; a parsing error on changed markup is not, and re-attempting it wastes time while hiding the real problem.
Fail the run rather than writing partial results, and keep failed payloads for inspection.
7. Where Collection Pipelines Differ
Web sources add problems internal pipelines do not have: sources change without notice, availability is not guaranteed, and content can vary by location and device.
That makes validation, raw retention, and per-source expectations more important here than in a pipeline drawing from a controlled database.
8. Summary
A data pipeline ingests, lands, transforms, validates, stores, and publishes. Idempotency, retained raw payloads, and partitioning are what make it re-runnable, and observability is what makes it trustworthy.
Choose batch unless latency genuinely demands otherwise, keep transformations pure so history can be reprocessed, and alert on a stage that did not run at all.