Why Is My Scraper Returning Empty Results?
Diagnose a scraper that runs without errors but extracts nothing. Covers JavaScript rendering, changed selectors, silent blocks, and soft 404s.
This is the worst failure mode in scraping precisely because nothing fails. Exit code zero, HTTP 200, no exception — and empty records. A crash is a gift by comparison, because it tells you where to look. Silent emptiness requires you to go and ask.
1. Narrow It Down First
Four questions, in order, resolve almost every case:
- Did you get a 200? Check the status code and the response size. A 200 of 8 KB where you expect 800 KB is a challenge page, not your content.
- Is your data in the raw HTML? Save the response to a file and search it for a value you expect. If the price is not in the bytes, no selector will ever find it.
- Does the selector match in a browser? Open DevTools on the live page and run the same selector. If it matches there but not in your saved HTML, the page is rendered by JavaScript.
- Are you parsing the page you think you are? Print the final URL after redirects. Consent walls, region redirects, and login pages all return 200.
Skipping straight to rewriting selectors is the standard mistake, and it wastes the most time.
2. Common Causes
JavaScript rendering. The initial HTML is a shell; content arrives via XHR after load. Your HTTP client sees the shell. This is the single most common cause.
Changed markup. Selectors keyed to generated class names (.css-1x2y3z) break on every deploy. The site is fine; your selector is archaeology.
Silent blocking. Anti-bot systems increasingly return 200 with content stripped rather than a visible 403, because a soft failure is harder to detect and cheaper to serve than a challenge.
Soft 404. The URL is wrong and the site returns 200 with a "not found" page. Status-code checks pass; the content is an error message.
Consent or region interstitial. A cookie banner page served in place of content until consent is recorded.
Login wall. Content requires a session. You get a 200 with a sign-in form.
Wrong parser mode. Parsing XML as HTML or vice versa. lxml in the wrong mode silently returns nothing rather than complaining.
Content in a JSON blob. Modern frameworks embed data in __NEXT_DATA__ or a similar script tag. It is in the HTML, but not as markup — no CSS selector will reach it.
iframes. The content is in a nested document your parser never entered.
3. How to Diagnose It
Save the raw response and read it. Not print(len(html)) — actually open it. Thirty seconds of reading beats an hour of theorising, and it immediately distinguishes challenge page, login form, consent wall, and genuine content.
Grep the saved HTML for a known value. Search for a price or title you can see in your browser. Present means a selector problem; absent means a rendering or blocking problem. This one test splits the entire problem space.
Compare response sizes between a browser fetch and your client's.
Check the final URL after redirects.
Disable JavaScript in your browser and reload. If the content vanishes, your HTTP client will never see it either — a faster test than reasoning about it.
Look for __NEXT_DATA__, window.__NUXT__, or a JSON-LD block. If your data is there, parsing that blob is usually easier and far more stable than scraping rendered markup.
4. How to Fix It
If it is JavaScript rendering: use a browser (Playwright, Puppeteer) — or better, find the XHR endpoint the page calls in the Network tab and request that JSON directly. The API is usually cleaner, faster, and more stable than the rendered page.
If it is selectors: prefer stable anchors — data-testid, semantic elements, text content, or structural relationships — over generated class names.
If it is silent blocking: treat it as a blocking problem, not a parsing one. Coherent headers, a matching TLS fingerprint, slower pacing, and a reconsidered IP.
If it is a soft 404 or login wall: fix the URL, or authenticate and persist the session.
If the data is in a JSON blob: extract and parse that. It is the same data the page renders, before the rendering.
Add assertions. The durable fix is refusing to let this be silent again:
if not records:
raise RuntimeError(f"0 records from {url} ({len(html)} bytes) — check fixture")
A scraper that returns zero rows without complaining will do it again next month, and nobody will notice until the dataset is already wrong.
5. Empty vs Partial
Partial results deserve their own suspicion. Getting 20 records when the page shows 100 usually means pagination is not being followed, lazy-loaded content below the fold never rendered, or a selector matching only one variant of a card layout. The diagnosis is the same — read the saved HTML and count what is actually in it.
6. Real-World Examples
- Works for weeks, then returns nothing. A deploy changed generated class names.
- Works locally, empty in production. Different egress IP, silently blocked in one environment.
- Every field empty except the title. The title is in the server-rendered
<head>; everything else is client-rendered. - "Page Not Found" stored as a product name. A soft 404 ingested as valid data.
- Empty only for some products. Out-of-stock items use a different template with different markup.
7. Summary
Empty results with no error means something between the request and the parse quietly did not do what you assumed. Do not start by rewriting selectors. Save the response, read it, and search it for a value you expect — that single step tells you whether you have a selector problem, a rendering problem, or a blocking problem, and those three have entirely different fixes.
Then make it loud. A scraper that yields zero records should fail, not succeed quietly, because the cost of this bug is not downtime — it is a dataset that looks fine and is wrong.