What Is Data Validation? – Definition and How It Works

Learn what data validation checks, where it belongs in a pipeline, the levels from schema to business rules, and how to fail usefully.

Data validation is confirming that data meets expectations before it is used. In a collection pipeline it is the difference between a failure that is noticed and one that is discovered months later in a report.

1. The Levels

Most pipelines implement the first and stop. The last two catch the failures that matter most, because they detect collection problems rather than parsing problems.

2. Validate at the Boundary

Check data as it enters the system, not deep inside it. Invalid data that reaches storage has already contaminated whatever reads from it.

The rule of thumb: parse and validate in the same step, so nothing untyped ever propagates.

3. Structural Validation

A schema declaration handles this mechanically — types, required fields, allowed values. Libraries exist in every language, and using one is preferable to hand-written checks that drift from the actual expectations.

Failures here should be loud, because a structural mismatch usually means the source changed.

4. The Checks That Catch Silent Failures

• Minimum record count per source, tuned per source rather than globally.
• Required-field fill rate — not just presence, but non-empty across a share of records.
• Value plausibility — a price of zero or a rating of 47 is a parsing failure.
• Duplicate identifiers, which usually indicate a pagination bug.
• Run-over-run comparison, which catches a source that quietly started returning less.

A run returning zero records should always fail, never write an empty file.

5. Failing Usefully

An error that says "validation failed" is nearly useless. A good failure names the check, the expected value, the actual value, and an example record.

Keep the raw response for any failed batch. Without it the failure cannot be diagnosed after the fact, and the temptation is to relax the check rather than investigate.

6. Quarantine Rather Than Discard

Records failing validation are evidence. Route them to a quarantine location with the reason attached, rather than dropping them.

Reviewing quarantine periodically is how systematic problems — one source with a different date format, one category with missing prices — become visible.

7. Validation Is Not Cleaning

Validation decides whether data is acceptable; cleaning transforms it. Keeping them separate matters: silently coercing bad values hides problems, while rejecting them surfaces the source that needs fixing.

Trim whitespace and normalise formats by all means, but do not quietly invent a value that was not there.

8. Summary

Data validation checks structure, values, cross-field consistency, set-level properties, and run-over-run comparisons before data enters a system.

The last two catch silent collection failures that structural checks miss. Fail with specifics, quarantine rather than discard, keep raw responses, and never let a zero-record run pass as success.

Potential Related Glossary

• Data Wrangling
• Data Structure
• Parsing
• Unstructured Data