Why Am I Getting Blocked While Scraping?

Troubleshooting P1 why am i getting blocked scraping

Learn why scrapers get blocked, which signals anti-bot systems actually check, how to diagnose the specific cause, and what genuinely fixes it.

Blocking is rarely one thing, and it is almost never the thing people assume. The instinct is to blame the IP address and reach for proxies. Sometimes that is right. Often the request was identifiable long before the address mattered, and rotating IPs simply burns money while changing nothing.

1. What Does "Blocked" Actually Mean?

Blocking is a spectrum, and the shape of the response tells you which system stopped you.

403 Forbidden — refused outright, often by a bot-management layer.
429 Too Many Requests — rate limited, the most cooperative rejection.
A challenge page — a JavaScript or CAPTCHA interstitial returned with 200 or 403.
Empty or partial content — 200 OK with the data silently removed.
A CAPTCHA — the system is unsure and is asking.
Timeouts or resets — connections dropped at the network layer before HTTP.

The last two categories matter most, because both can look like success. A 200 containing an error page or stripped content is the failure mode that quietly poisons a dataset.

2. What Anti-Bot Systems Actually Check

IP reputation and origin. Whether the address belongs to a hosting provider is one public ASN lookup. Datacenter ranges are treated with suspicion by default, before anything else is examined.

TLS fingerprint. The handshake itself — cipher suites, extensions, their order — produces a JA3 or JA4 fingerprint. Python's requests has a signature nothing like Chrome's. Claiming to be Chrome in your User-Agent while handshaking like a Python library is a contradiction visible in the first packet.

HTTP/2 fingerprint. Frame settings, header table size, and pseudo-header order differ between real browsers and libraries.

Header composition and order. Real browsers send a specific set of headers in a consistent order. Missing Accept-Language, absent Sec-Fetch-* headers, or alphabetically sorted headers are all tells.

Browser environment. If JavaScript runs, the page can inspect navigator.webdriver, plugin lists, screen dimensions, timezone, WebGL renderer strings, and canvas rendering. Headless browsers differ from headful ones in ways that are measurable.

Behaviour. Request rate, timing regularity, whether assets are loaded, navigation patterns, mouse movement. Perfectly even 500ms intervals are not human.

Cookie and session continuity. Arriving at a deep product page with no session, no referrer, and no prior navigation is not how browsers behave.

3. How to Diagnose Which One Caught You

Work from cheapest test to most expensive, and stop as soon as one is conclusive.

Does a browser on the same network load it? If yes, your network is fine and the request is the problem. If no, the address is implicated.

Does curl fail where the browser succeeds? Strongly indicates TLS or header fingerprinting, since both sent the same request from the same IP.

Does it work for the first N requests and then stop? That is rate or volume based. Note the number — it is usually stable and tells you the threshold.

Does it fail immediately, on request one? Fingerprint or IP reputation. Nothing about your behaviour has been observed yet, so it was your identity.

Does the same code work from a home connection but not from a server? IP origin, definitively.

Read the response body. cf-, _px, datadome, or captcha-delivery markers name the vendor, and each behaves differently.

Compare response sizes. A page that is 40 KB instead of 800 KB is a challenge or a stripped response, not the content you asked for.

4. How to Fix It

Match the fix to the diagnosis, in this order.

Slow down first. It is free, it fixes a large share of real cases, and no amount of fingerprint work compensates for an obviously inhuman request rate. Add jitter — evenly spaced requests are themselves a signal.

Send a coherent request. A complete browser header set in browser order, a real User-Agent, Accept-Language, and a plausible Referer when arriving at a deep page. Coherence matters more than any individual value.

Fix the TLS fingerprint. A library such as curl_cffi impersonates real browser handshakes without running a browser — far cheaper than launching Chrome, and it resolves a large class of instant 403s.

Reconsider the IP only when evidence points there. If a home connection works and a datacenter one does not, that is the signal. Residential proxies are expensive, so escalate deliberately rather than reflexively.

Use a real browser when the page genuinely requires one. If content is rendered by JavaScript, or the site fingerprints the browser environment, no header combination substitutes for actually being a browser.

Maintain sessions. Persist cookies across requests and navigate the way a user would rather than jumping straight to deep URLs.

Check for an official route first. An API, a data export, or a sitemap makes the entire problem disappear. This is not a workaround — it is the correct answer whenever it exists, and it is more reliable than anything else on this list.

Respect what the site asks. robots.txt, terms of service, and rate limits are part of scraping responsibly, and ignoring them is how access is lost permanently.

5. What Usually Does Not Work

Common attempt	Why it disappoints
Rotating `User-Agent` strings only	The TLS handshake still says Python
Buying more datacenter IPs	The ASN is the problem, not the count
Random delays alone	Fixes rate limits, not fingerprinting
Headless Chrome with defaults	`navigator.webdriver` and headless tells remain
Retrying a 403 harder	Escalates toward a permanent ban
Copying one header from DevTools	Coherence across all headers is what is checked

6. Real-World Examples

Instant 403 from a cloud server, fine from a laptop. IP origin, and no header work will change it.
Works 200 times, then blocks every time. A volume threshold. The fix is pacing, not proxies.
200 OK with prices missing. Content stripped for a suspected bot — the most dangerous outcome, because nothing errors.
Headless browser blocked, headful browser fine. The environment is being fingerprinted, not the network.
Blocked only on product pages. Protection is applied per route; the blog was never defended.

7. Summary

Blocking is a set of independent checks, and effort spent on the wrong one is wasted. Diagnose before you spend: a browser test on the same network separates address problems from request problems, and immediate versus delayed blocking separates identity from behaviour.

Slow down first because it is free. Fix coherence — headers and TLS fingerprint agreeing with each other — before buying proxies. Escalate to residential addresses and real browsers only when evidence demands it. And check whether an API or export exists before any of this, because the most reliable way not to be blocked is not to need to be.