Web Scraping With Python

Learn how to scrape with Python: which HTTP client and parser to choose, when a browser is needed, and the practices that keep a scraper working.

Python dominates web scraping because the whole pipeline — fetch, parse, transform, store — has mature libraries, and because pandas is waiting at the end of it. The tooling is not the hard part; choosing correctly between the layers is.

1. The Three Layers

Every scraper is the same three decisions:

1. Fetch — get the bytes. requests, httpx, curl_cffi, or a browser.
2. Parse — turn markup into data. BeautifulSoup, lxml, selectolax, or json.loads.
3. Store — CSV, JSONL, a database.

Most difficulty lives in the first layer. Parsing is largely solved.

2. Choosing an HTTP Client

curl_cffi deserves attention. Ordinary Python clients have a TLS fingerprint that identifies them before any header is sent, so a request with perfect browser headers is still recognisably requests. curl_cffi reproduces a real browser's handshake:

from curl_cffi import requests
r = requests.get(url, impersonate="chrome")

For targets that block on TLS, this single change often does what hours of header tuning cannot.

3. Choosing a Parser

BeautifulSoup with the lxml parser is a good default. At millions of pages, selectolax is meaningfully faster:

from selectolax.parser import HTMLParser
tree = HTMLParser(html)
title = tree.css_first("h1").text()

4. When You Need a Browser

You need Playwright or Selenium only when content genuinely does not exist in the fetched HTML.

Before assuming that, check: open DevTools → Network → Fetch/XHR and reload. Most "JavaScript-heavy" pages fetch their data from a JSON endpoint you can call directly — cleaner, smaller and more stable than the rendered DOM, and typically an order of magnitude faster than driving a browser.

Also look for __NEXT_DATA__ or a similar embedded script blob. The data is often already JSON, sitting in the HTML you have.

Reach for a browser when interaction is required, when rendering depends on state you cannot reconstruct, or when the page is the thing you are measuring.

5. Scrapy

For crawls rather than single pages, Scrapy provides scheduling, deduplication, concurrency, retries, throttling, pipelines and export in one framework.

Use it when you are crawling a site — following links, managing a frontier, handling thousands of URLs. Use plain requests plus a parser when you have a known list of pages. Adopting Scrapy for twenty URLs is overhead; hand-rolling a crawler for two hundred thousand is a slow reimplementation of Scrapy.

6. Practices That Keep It Working

Fail loudly. The dangerous scraper is the one that returns empty strings after a redesign and writes nulls for a month. Assert on record counts and on required fields, and raise when a run returns nothing.

if not products:
    raise ValueError(f"no products parsed from {url}")

Prefer stable selectors. Generated class names (.css-1x9k2p) change on every deploy. Anchor on data-* attributes, IDs, semantic structure or visible text.

Save fixtures. Keep real HTML samples and test parsers against them offline. That is how you tell a site change from a code change.

Respect the server. Delays between requests, concurrency limits, Retry-After on a 429, and exponential backoff with jitter.

Cache during development. Store responses on disk so iterating on a parser does not re-fetch. This alone makes development several times faster and is far kinder to the target.

Handle encoding explicitly. r.encoding is a guess. Where the site declares a charset, use it.

7. Real-World Examples

• A requests scraper failing where curl_cffi with identical headers succeeds — the block was at the TLS layer.
• A Selenium scraper replaced by two HTTP calls, after the internal API turned up in the Network tab, cutting a six-hour run to twelve minutes.
• A pipeline that silently stored nulls for weeks because nobody asserted on record counts.
• A parser rewritten from BeautifulSoup to selectolax, halving CPU on a multi-million-page crawl.
• A scraper broken by a class-name change and fixed permanently by anchoring on a data-testid instead.

8. Summary

Python scraping is three choices: how to fetch, how to parse, and where to store. Fetching is where the difficulty concentrates — requests for simple targets, httpx for concurrency, curl_cffi when TLS fingerprinting is the obstacle, a browser only when the page truly requires one.

For parsing, BeautifulSoup is the best place to start and selectolax the right destination at scale. For crawling rather than fetching, Scrapy supplies the infrastructure you would otherwise rebuild.

What separates a scraper that survives from one that quietly rots is not the library choice: it is loud failure, stable selectors, saved fixtures and a cache during development.

Potential Related Glossary

• Parsing
• Web Crawling
• Unstructured Data
• Data Wrangling