What Is Schema Validation for Scraped Data? – Definition and Practice
Learn how to apply schema validation to collected data, which constraints catch extraction failures, and how to handle records that fail.
Schema validation checks extracted records against a declared structure before they are stored. For collected data it does something more specific than type checking: it turns a silent extraction failure into a loud one.
1. Why Collected Data Needs It Especially
Sources change without notice. A redesign moves an element, a selector matches something else, and extraction continues without error — producing records that are well-formed and wrong.
A schema is the assertion that catches this, because the wrong element usually produces the wrong type, an implausible value, or an empty field.
2. What to Declare
class Product(BaseModel):
url: HttpUrl
title: str = Field(min_length=1, max_length=300)
price: Decimal = Field(gt=0, lt=1_000_000)
currency: Literal["USD", "EUR", "GBP"]
in_stock: bool
rating: float | None = Field(default=None, ge=0, le=5)
collected_at: datetime
Every constraint here corresponds to a real failure: an empty title from a missing element, a zero price from a currency symbol parsed as a number, a rating of 47 from the review count landing in the wrong field.
3. Constraints That Catch Extraction Failures
| Constraint | Catches |
|---|---|
| Minimum string length | Element found but empty |
| Numeric bounds | Wrong field extracted |
| Enumerated values | Currency or status parsed from the wrong place |
| Pattern matching | Identifiers picking up surrounding text |
| Required fields | A selector matching nothing |
| Cross-field rules | Discount exceeding price |
4. Validate at Extraction, Not at Load
The value of validation is proportional to how early it runs. Validating at extraction attributes the failure to a specific page, which is what makes it fixable.
Validating at load says only that something upstream is wrong, and by then the batch has been assembled from many sources.
5. Set-Level Checks Too
Per-record validation cannot see that a page returned three products instead of the usual sixty. Add checks over the batch:
- Minimum record count per source.
- Fill rate for optional-but-usually-present fields.
- Duplicate identifier rate.
- Comparison against the previous run.
These catch the failures that individually valid records hide.
6. Handling Failures
Quarantine rather than discard. Write failing records with their validation error and the raw payload to a separate location, and review periodically.
A rising failure rate for one source is the earliest reliable signal that its markup changed — usually well before anyone notices in a report.
7. Keep the Schema Versioned
Store the schema with the pipeline and record which version produced each batch. When a source legitimately changes — a new field, a widened range — the schema change should be deliberate and dated, not a quiet relaxation to make errors stop.
8. Summary
Schema validation for collected data declares what a correct record looks like and rejects what does not match, at extraction time where the failure can be attributed.
Its purpose is not tidiness but detection: constraints on length, range, and enumerated values are what convert a silently wrong extraction into an error somebody sees.