Why Does My Scraper Work Locally But Not in Production?
Diagnose a scraper that passes on your machine and fails on a server. Covers IP origin, headless differences, missing environment, timezone and concurrency.
Nothing about the code changed, so the difference is environmental. There are only a handful of ways a server differs from a laptop in ways a target site can observe, and one of them dominates all the others.
1. The Most Likely Cause, First
Your egress IP changed, and that is usually the whole answer.
Your laptop leaves through a residential ISP connection. Your server leaves through a hosting provider's address, and whether an address belongs to a hosting ASN is a single public lookup. Many sites refuse or challenge datacenter ranges categorically, before examining anything else about the request.
Confirm it in one step — run the identical code on the server through a residential proxy. If it works, the diagnosis is finished and the rest of this page is irrelevant.
2. Other Real Differences
Headless versus headful. Locally you may run a visible browser; in production, headless. Headless Chrome differs measurably — navigator.webdriver, missing plugin lists, different WebGL renderer strings, no window chrome. Sites fingerprint exactly these.
Missing browser dependencies. Playwright and Puppeteer need system libraries that a slim container image lacks. The failure is usually a launch error, but sometimes a browser starts in a degraded state and renders incompletely.
No display server. Headful mode on a server needs Xvfb. Without it, launching fails or silently falls back.
Environment variables absent. API keys, proxy credentials, and feature flags present in your shell and missing from the container. Frequently the code degrades rather than crashing — a missing proxy variable means requests go out unproxied and get blocked.
Timezone and locale. Servers run UTC; laptops run local time. Sites serve different content by locale, and a fingerprint claiming a US browser from a UTC machine with en-GB headers is incoherent.
Concurrency. Locally you test one request; production runs twenty in parallel. The per-request logic is identical — the rate is not, and rate is what triggers limits.
Different network path. Corporate DNS, split-horizon resolution, or an egress firewall that permits your laptop and not the server.
Filesystem and cache. A local run reuses a cached session or cookie jar that never existed in production, so production is always making a cold first request.
Clock skew. A badly wrong server clock breaks TLS validation and signed requests.
3. How to Diagnose It
Compare what the target sees, from both places:
curl -s https://httpbin.org/ip # egress address
curl -s https://httpbin.org/headers # exactly what you send
Run these on the laptop and on the server. The diff is your answer, and it is usually one line.
Run the identical command in both environments. Not "the same script" — the same command, same arguments, same environment loading. Differences in how the job is invoked hide plenty of bugs.
Save the production response to a file and read it. A challenge page, a consent wall, and a login form all return 200 and all look like "it didn't work".
Drop production concurrency to one. If it starts working, the problem is rate, not environment.
Run the production container locally. This isolates "container versus host" from "server versus laptop" — two different hypotheses that are easy to conflate.
Log the resolved configuration at startup, including whether a proxy is in use and which browser executable was launched. Not the secret values — just whether they are present.
4. How to Fix It
- Route production through residential or ISP addresses when the target refuses datacenter ranges. This is the fix for the dominant cause.
- Match browser mode across environments so local testing is representative. Testing headful and shipping headless means your tests never exercised what you deployed.
- Use the official Playwright or Puppeteer base image, or install the documented dependency set, rather than a slim image plus guesswork.
- Fail fast on missing configuration. Assert required variables at startup rather than degrading silently — an unproxied request that gets blocked is far harder to diagnose than a clear boot-time error.
- Set timezone and locale explicitly so the environment is coherent with the identity you present.
- Cap global concurrency rather than per-worker, and add jitter.
- Verify the egress IP in a health check so a proxy misconfiguration surfaces immediately.
5. The Test That Settles It
Run this on the server before doing anything else:
curl -s https://httpbin.org/ip
If that address belongs to a hosting provider and the target blocks hosting providers, nothing else on this page matters. Fix that first, then re-test. Debugging headless fingerprints while the address is the actual problem is the most common way this loses a day.
6. Real-World Examples
- Instant 403 in production, clean locally. Datacenter egress, and the ASN alone was enough.
- Browser fails to launch in the container. Missing shared libraries in a slim base image.
- Requests silently unproxied.
HTTPS_PROXYwas in the developer's shell profile and not in the deployment environment. - Works at 1 worker, blocked at 20. Rate, not environment — and the code was identical.
- Different content returned. Server in UTC with a different
Accept-Language, served a different regional variant.
7. Summary
The code is the same, so the environment is the difference — and the environment differs in a small, enumerable number of ways. Check the egress address first, because hosting-provider IPs are the single most common cause and the cheapest to confirm.
After that, compare exactly what each environment sends using httpbin.org/headers, match browser mode between local and production so your testing is representative, and make missing configuration fail loudly at startup. Most of the remaining time in this class of bug is spent debugging the wrong layer, and one curl at the beginning prevents it.