How Does Web Scraping Work?
Follow a scraper end to end: the request, what the server decides, parsing, selection, and where the process actually breaks in production.
Mechanically, scraping is simple: request a page, find the values in it, write them down. Everything difficult happens in the gap between "request a page" and "receive the page you expected".
1. The Request
Your client opens a TCP connection, performs a TLS handshake, and sends an HTTP request.
Three things are already being observed before any content is exchanged:
- The TLS handshake. The ClientHello's cipher and extension ordering identifies your HTTP library. This happens before headers are sent, which is why headers cannot conceal it.
- The connection metadata. Your IP, and what its address range implies — consumer ISP or hosting provider.
- HTTP/2 behaviour. Frame ordering and settings differ between implementations much as TLS does.
By the time your User-Agent arrives, the server already has a well-formed opinion about what you are.
2. What the Server Decides
The response is not a single decision but a score assembled from signals:
| Signal | What it contributes |
|---|---|
| TLS fingerprint | Which software is connecting |
| IP classification | Datacenter, residential, mobile, reputation |
| Header set and order | Real browsers send a consistent, ordered set |
| HTTP/2 fingerprint | Corroborates or contradicts the TLS one |
| Request pattern | Rate, timing regularity, navigation shape |
| Cookies | Whether prior state exists |
Four outcomes are common: the real page, the page with content withheld, a challenge, or a 403.
The most troublesome outcome is the middle one: HTTP 200 with a stripped body. No error is raised, and a scraper that only checks status codes records the success and stores nothing.
3. Getting the Content
If the HTML contains the data, you are done fetching. If it does not, there are three routes, in ascending order of cost:
The internal API. DevTools → Network → Fetch/XHR shows what the page requests. That JSON endpoint is smaller, cleaner and typically more stable than the markup.
Embedded state. __NEXT_DATA__, __NUXT__, window.__INITIAL_STATE__, or a JSON-LD block — the data already serialised inside the page.
A browser. Playwright, Puppeteer or Selenium executes the JavaScript and produces the rendered DOM. Correct, and one to two orders of magnitude more expensive than a request.
4. Parsing
The parser turns a string of markup into a tree. Real HTML is frequently malformed — unclosed tags, misnested elements — and parsers differ in how they recover, which is why the same broken document can yield different trees in different libraries.
In Python: BeautifulSoup for tolerance and readability, lxml for XPath and speed, selectolax for volume. In Node: Cheerio. The choice affects performance far more than correctness for well-formed pages.
5. Selection
You then locate values by pattern:
price = soup.select_one("[data-testid='price']").get_text(strip=True)
Selector durability is the whole game here. Generated class names like .css-1x9k2p change on every deploy. data-* attributes, IDs, semantic elements and visible text survive redesigns because they carry meaning rather than styling.
Then comes normalisation, which is where most quiet data corruption enters: "$1,299.00" must become 1299.00, dates must be parsed into a single timezone, and "From $19.99" must not silently become 19.99 in a field labelled price.
6. Where It Actually Breaks
- Silent extraction failure. A redesign, and every selector matches nothing. Success is reported, nulls are stored.
- Partial success. Nine fields extract, one returns empty. The record looks valid.
- The 200-with-no-content case. Blocked, but not by a status code.
- Encoding. Mojibake from a guessed charset, discovered downstream.
- Pagination that lies. A "next" link that loops, or a cursor that repeats, silently duplicating or truncating.
- Rate limits mistaken for errors. Retrying a 429 immediately turns a pause into a ban.
Every one of these produces wrong data rather than a crash — which is why validation, not exception handling, is what makes a scraper trustworthy.
7. Making It Reliable
- Assert on counts. Zero rows from a listing page must raise.
- Validate types after normalisation. A price that will not parse is an error, not a null.
- Check content, not just status. A 200 containing a challenge page is a failure.
- Save fixtures and test offline, so you can tell your bug from their redesign.
- Cache during development. Faster iteration, less load on the target.
- Back off properly. Honour
Retry-After; otherwise exponential with jitter.
8. Summary
A scraper connects, and the server judges it on the TLS handshake, the IP, the header set and the request pattern before any content is served. The response may be the page, a reduced page, a challenge, or a refusal — and the reduced page is the dangerous case, because it arrives as a 200.
Getting content means the HTML itself, an internal JSON endpoint, an embedded state blob, or a browser, in ascending order of cost. Parsing and selection are well-supported; durability comes from anchoring on selectors that carry meaning rather than styling.
The failures that matter are silent ones — empty extractions, partial records, blocked responses with successful status codes. Reliability comes from asserting on what was extracted rather than on whether the request completed.