How to Debug a Broken Scraper – A Systematic Method
Learn a stage-by-stage method for diagnosing collection failures: fetch, render, parse, validate — and how to tell a silent failure from a real one.
Collection failures divide into loud ones, which raise errors, and silent ones, which return nothing while reporting success. The second kind is more common and more damaging, and both are diagnosed by isolating the stage that broke.
1. Establish Which Stage Failed
| Stage | Question | Test |
|---|---|---|
| Fetch | Did a response arrive? | Status, size, and elapsed time |
| Content | Does the response contain the value? | Grep the raw body |
| Render | Does it appear only after scripts run? | Compare fetched HTML against the browser |
| Parse | Does the selector match? | Query the saved HTML directly |
| Validate | Are the values plausible? | Check counts, types, and ranges |
Work in that order. Debugging a selector against a challenge page wastes considerable time.
2. Save the Raw Response First
curl -s -D headers.txt "https://example.com/page" -o page.html
wc -c page.html && grep -c "expected value" page.html
An unexpectedly small file usually means a challenge or an empty shell. A normal size with zero matches means either client rendering or a genuinely changed page.
Debugging against a saved file also makes the process repeatable, which debugging against a live site never is.
3. Distinguish the Common Causes
- Challenge or interstitial — small body, unusual title, often a 200 status.
- Empty shell — a container element and a script bundle, no content.
- Soft block — full page structure, zero records.
- Changed markup — content present, selectors no longer match.
- Session lost — a logged-out view where a logged-in one was expected.
- Wrong region — different currency or language than intended.
Each has a distinct signature in the saved response, which is why saving it comes first.
4. Then Check the Selector Against the Saved File
If the value is present in the file and your selector returns nothing, the problem is the selector. Common causes: generated class names changed, an extra wrapper appeared, the element moved into an iframe or shadow root, or the page served a variant you had not seen.
5. Check the Silent Failures Explicitly
A run that completed without error is not a run that worked. Assert on:
- Record count against expectation, per source.
- Required fields non-empty across a share of records.
- Value plausibility — zero prices, ratings out of range.
- Comparison against the previous run.
Adding these turns the next silent failure into a loud one, which is the actual fix.
6. Reproduce Before Changing Anything
Fix the inputs: the saved response, a fixed viewport, a pinned browser version, one URL. A change that appears to work against a live site may simply have hit a different variant.
Once a failure reproduces reliably, the fix can be verified. Without that, verification is guesswork.
7. Keep the Evidence
Store the failing response, the headers, the resolved selector, and the exact request that produced it. The next occurrence is then a comparison rather than a fresh investigation, and recurring failures reveal patterns that individual ones hide.
8. Summary
Debug by isolating the stage: did a response arrive, does it contain the value, was rendering required, does the selector match, are the values plausible.
Save the raw response before anything else, work against that saved file for repeatability, and add validation assertions so the next failure announces itself instead of writing an empty file.