What Is a Golden Dataset? – Definition and How It Works

Informational P2 what is a golden dataset

Learn what a golden dataset is, how to build one for extraction quality, and why hand-labelled truth is worth the cost.

A golden dataset is a hand-verified set of inputs with known-correct outputs, used as ground truth for measuring quality. Without one, "extraction accuracy" is an opinion.

1. Why You Cannot Measure Without One

Every quality signal available from a pipeline alone is indirect. Fill rates say a field was populated, not that the value was right. Record counts say rows arrived, not that they were correct.

A golden dataset is the only thing that answers: is the extracted value the value on the page?

2. What It Contains

For an extraction pipeline, each entry pairs a saved input with verified expected output:

golden/
├── example.com/product-standard.html       + .expected.json
├── example.com/product-on-sale.html        + .expected.json
├── example.com/product-out-of-stock.html   + .expected.json
├── example.com/listing-empty.html          + .expected.json
└── example.com/listing-last-page.html      + .expected.json

{
  "title": "Wireless Keyboard",
  "price_cents": 1999,
  "currency": "USD",
  "in_stock": true,
  "rating": 4.5,
  "review_count": 218
}

The expected values are read off the page by a person. That is the whole point — the cost is the reason it is trustworthy.

3. Choose Cases That Discriminate

A golden set of twenty normal pages measures almost nothing. The value is in the awkward cases:

Case	Catches
Sale price beside original	Picking the wrong price
Missing optional field	Fabricating a value
Zero results	Crashing, or reporting phantom rows
Single result	Plural-only parsing assumptions
Last page of pagination	Non-terminating pagination
Unicode and accents	Encoding faults
Very long text	Truncation
Multiple currencies	Currency assumptions
Thousands separators	Number parsing

Ten well-chosen hard cases beat two hundred easy ones.

4. Measuring Against It

def evaluate(golden_dir, extract):
    results = []
    for html_path, expected in load_golden(golden_dir):
        actual = extract(read(html_path))
        for field, want in expected.items():
            got = actual.get(field)
            results.append({"file": html_path.name, "field": field,
                            "want": want, "got": got, "ok": got == want})
    per_field = {}
    for r in results:
        s = per_field.setdefault(r["field"], {"ok": 0, "n": 0})
        s["n"] += 1; s["ok"] += r["ok"]
    return {f: v["ok"] / v["n"] for f, v in per_field.items()}, \
           [r for r in results if not r["ok"]]

Reporting per field is what makes it actionable. An overall 94% hides a price field at 60%, and price is the field that matters.

5. It Is Not the Same as a Test Suite

Related, and distinct in purpose:

Tests ask: did behaviour change? They are pass/fail and run on every commit.
A golden dataset asks: how good is it? It produces a score and a list of specific failures.

You want both. Tests catch regressions; the golden set tells you whether the thing was ever good, and where the remaining errors are.

6. Maintaining It

Keep it current. Recapture inputs periodically as sites change, retaining old entries as regression cases.

Record who verified what, and when. A golden value with no provenance is just an assertion.

Correct it when it is wrong. Golden data contains human errors; a disputed failure sometimes means the expectation was wrong, and fixing it is legitimate — provided the correction is deliberate and recorded, not a quiet adjustment to make a number look better.

Sanitise before committing. Captured pages contain tokens and personal data, exactly as ordinary fixtures do.

7. Where Else It Applies

The same discipline covers more than HTML extraction:

Entity resolution — labelled pairs, including hard negatives that look similar but are different.
Classification — labelled examples per class.
LLM extraction — inputs with verified structured outputs, which is the only way to compare prompts or models honestly.
Deduplication — known duplicate and non-duplicate sets.

For anything involving a model, a golden set is the difference between choosing a prompt on evidence and choosing it on impression.

8. Summary

A golden dataset pairs saved inputs with hand-verified expected outputs, and it is the only thing that measures whether extracted values are actually correct rather than merely present.

Populate it with discriminating cases — sale prices, missing fields, empty results, last pages, unicode, multiple currencies — score per field rather than overall, keep it distinct from the regression test suite, and sanitise the captured inputs before committing them.