What Is Web Scraping? – Definition and How It Works

Learn what web scraping is, the four stages every scraper shares, how it differs from crawling and APIs, and what makes one survive in production.

Web scraping is the automated extraction of data from websites. It exists because an enormous amount of valuable information is published only as pages designed for people to read, with no API behind it.

1. What Is Web Scraping?

It is turning content built for human readers back into structured data.

• Key idea: derive structure that the publisher never promised.
• Mechanism: fetch a page, locate the values within its markup, store them in a defined shape.
• Goal: get at data that exists publicly but is not offered in machine-readable form.

2. The Four Stages

Every scraper, from a ten-line script to a distributed platform, does the same four things:

1. Fetch. Retrieve the page — an HTTP request, or a browser when JavaScript is required.
2. Parse. Turn markup into a navigable tree.
3. Extract. Select the values with CSS selectors, XPath, or by reading embedded JSON.
4. Store. Write to CSV, JSONL, or a database.

Fetching is where nearly all the difficulty lives. Parsing and extraction are largely solved problems with mature libraries.

3. Scraping vs. Crawling vs. APIs

Crawling and scraping are usually two phases of one job: crawl to discover product URLs, scrape each to extract the product. They are separable, and treating them separately makes both easier to debug.

Against an API, scraping always loses when an API exists — a versioned contract beats an inference from markup every time. Scraping is the technique for when no door exists.

4. Check for an API First

This step gets skipped constantly and takes minutes:

• Open DevTools → Network → Fetch/XHR and reload. Most modern pages fetch their own data from internal JSON endpoints. That endpoint is cleaner, smaller and more stable than the markup, even when undocumented.
• Search the HTML for __NEXT_DATA__ or a similar embedded blob — often the whole dataset, already JSON.
• Look for application/ld+json structured data, placed there deliberately for machines.
• Check the developer documentation. Plenty of APIs are simply not advertised.

Finding the endpoint behind a page is routinely the highest-value thing you can do before writing any extraction code.

5. What Makes It Hard

Detection. Sites distinguish automated traffic by TLS fingerprint, HTTP/2 behaviour, header order, browser attributes and request patterns. Note the order: IP address is rarely the first signal, which is why proxy rotation alone so often changes nothing.

Rendering. When content is built client-side, the fetched HTML is an empty shell.

Change. You are coupled to a presentation layer, and presentation is what changes most freely.

Silent failure. The dominant production problem. A redesign does not raise an error — selectors simply match nothing and the pipeline stores empty rows. Unless something asserts on record counts, the first signal is a colleague asking why a number looks wrong, weeks and one contaminated dataset later.

6. What Makes One Survive

• Fail loudly. Assert that a product page produced products. Raise on zero records.
• Anchor on stable selectors. data-* attributes and semantic structure outlive generated class names.
• Save fixtures. Real HTML on disk distinguishes a site change from a code change.
• Cache during development. Iterate on parsing without re-fetching — faster for you, kinder to the target.
• Respect the server. Delays, concurrency caps, Retry-After on 429s, backoff with jitter.
• Monitor the shape, not just the run. A job that completes successfully with 3 records instead of 3,000 has failed.

7. Legality and Ethics

Scraping publicly accessible data is broadly lawful in many jurisdictions, and that is not the whole answer. What matters in practice:

• Personal data brings GDPR, CCPA and similar regimes into scope regardless of public availability.
• Copyright applies to content; facts are generally not protected, expression is.
• Terms of service may bind you contractually, particularly after creating an account.
• Load matters. Degrading a service is a separate wrong from reading it.
• robots.txt is a convention, not a law — and disregarding it is a decision you should make deliberately.

8. Summary

Web scraping extracts structured data from pages built for people, in four stages: fetch, parse, extract, store. Fetching is where the difficulty concentrates; parsing is well served by existing libraries.

Always check for an API first, including the internal JSON endpoints visible in the Network tab — they frequently deliver the same data far more cleanly and survive redesigns that break markup parsing.

The characteristic failure is not a crash but silence: selectors that stop matching and a pipeline that keeps reporting success while storing nothing. Scrapers that last are the ones built to fail loudly, anchored on selectors that survive redesigns, tested against saved fixtures, and operated within the limits the target actually publishes.

Potential Related Glossary

• Web Crawling
• Parsing
• Unstructured Data
• Data Wrangling