What Is a Fixture in Scraper Testing? – Definition and How It Works

Learn what a scraper fixture is, how to capture and organise them, and why they must be sanitised before being committed.

A fixture is a saved real response — HTML, JSON, or a full HTTP exchange — used as deterministic test input. It lets extraction be tested without the network and without the site's cooperation.

1. Why a Saved Response Beats a Handwritten One

A handwritten test document contains the structure you think the site produces. A captured one contains what it actually produces: the unclosed tags, the inconsistent whitespace, the empty attributes, the tracking markup, the two different price formats on one page.

Those details are precisely where extraction breaks, and they are exactly what a hand-written fixture omits.

2. Capturing Them

import pathlib, requests

def capture(url, name):
    r = requests.get(url, timeout=30)
    r.raise_for_status()
    path = pathlib.Path("tests/fixtures") / name
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(r.text, encoding="utf-8")
    (path.with_suffix(path.suffix + ".meta.json")).write_text(
        json_dumps({"url": url, "status": r.status_code,
                    "captured_at": iso_now(),
                    "content_type": r.headers.get("Content-Type")}))

The sidecar metadata matters as much as the body. A fixture without its URL and capture date is an anonymous blob nobody can reason about a year later.

3. What to Capture

Beyond the happy path, the fixtures that earn their keep:

4. Sanitise Before Committing

This is the step that gets skipped, and it matters. A captured page can contain a session cookie in an inline script, a CSRF token, an API key in a configuration blob, a signed URL, personal data in a review or comment, or an authenticated user's name and email.

Committing that to a repository publishes it.

import re

PATTERNS = [
    (re.compile(r'("csrfToken"\s*:\s*")[^"]+'), r'\1REDACTED'),
    (re.compile(r'(Bearer\s+)[A-Za-z0-9._-]+'), r'\1REDACTED'),
    (re.compile(r'([?&](?:token|key|sig|signature)=)[^&"\'\s]+'), r'\1REDACTED'),
    (re.compile(r'[\w.+-]+@[\w-]+\.[\w.]+'), '[email protected]'),
]

def sanitize(html):
    for pat, repl in PATTERNS:
        html = pat.sub(repl, html)
    return html

Review a new fixture by eye as well. A pattern list catches the known shapes, not the unknown ones, and a fixture is committed once and lives for years.

5. Organising Them

tests/fixtures/
├── example.com/
│   ├── listing-full-2026-08-18.html
│   ├── listing-full-2026-08-18.html.meta.json
│   ├── listing-empty-2026-08-18.html
│   └── product-no-price-2026-08-18.html
└── other.org/
    └── search-page-1-2026-07-02.html

Directory by host, descriptive name, capture date in the filename. The date is what tells you a fixture predates a redesign, which is the question you will actually ask.

6. Keeping Them Current

A fixture captured two years ago tests against a page that no longer exists. That is not useless — it proves the parser still handles the old shape — but it is not evidence about today.

The workable practice: recapture periodically, keep the old fixture as a regression case, and add a new one for the current shape. Passing tests against both means the parser handles the redesign without having lost the old path.

7. Size and Repository Health

Full HTML pages are large, and hundreds of them bloat a repository. Practical measures: trim irrelevant sections while keeping the structure under test, compress fixtures and decompress in the test helper, and keep one representative fixture per case rather than a dozen near-identical captures.

Very large binary fixtures belong in object storage referenced by the tests, not in the repository.

8. Summary

A fixture is a captured real response used as deterministic test input, and it beats a handwritten document because it contains the malformed, inconsistent markup where extraction actually breaks.

Capture the awkward cases — empty results, missing fields, block pages, soft 404s, truncated bodies — store URL and date in a sidecar, and sanitise every fixture for tokens, keys, and personal data before committing, because a fixture is published once and lives for years.

Potential Related Glossary

• Parsing
• Web Crawling
• Unstructured Data
• Data Structure