What Is Common Crawl? – Definition and How It Works
Learn what Common Crawl is, what its WARC, WAT, and WET files contain, how the corpus is used, and where its coverage and freshness fall short.
Common Crawl is a public archive of web pages, published as periodic snapshots and freely available. It is one of the most widely used sources of web-scale text, particularly for training and research.
1. What It Is
A non-profit crawl of the open web, released as datasets.
- Cadence: snapshots released on a regular schedule.
- Scale: billions of pages per snapshot.
- Access: hosted publicly, downloadable in bulk or queried by index.
- Cost: free to use, subject to the licences of the underlying content.
2. The File Formats
| Format | Contains | Typical use |
|---|---|---|
| WARC | Raw HTTP request and response records, including headers and full HTML | Faithful reprocessing, custom extraction |
| WAT | Metadata extracted from the WARC — links, headers, and structure | Link graphs, site-structure analysis |
| WET | Plain text extracted from each page | Language corpora, quick text mining |
WET is the fastest route to text and the lossiest; WARC is the honest source when extraction quality matters.
3. Finding Pages Without Downloading Everything
A snapshot is enormous, so bulk download is rarely the right approach. The published index allows lookup by URL or domain, returning the file, offset, and length of the matching record — so a single page can be fetched with a ranged request.
That makes targeted historical retrieval practical: a specific domain's pages across several snapshots, without touching the rest of the corpus.
4. What It Is Used For
- Training corpora for language models, after heavy filtering and deduplication.
- Historical analysis — how a page or a domain changed across snapshots.
- Link graph research at web scale.
- Language and domain sampling for datasets that need breadth.
- Prototyping, where a representative sample beats building a crawler first.
5. Where It Falls Short
- Freshness. Snapshots are periodic; nothing in them is current.
- Coverage is a sample. Not every page, and not every site, is included.
- No rendering. Pages are captured as served, so client-rendered content is largely absent.
- Extraction quality varies. WET text includes boilerplate and misses structure.
- Blocked content is absent. Sites that disallow crawling are not represented.
That last point matters when the corpus is used to reason about the whole web: it represents the crawlable web.
6. Licensing and Ethics
The archive is free to access, but the pages within it carry their own copyright and terms. Availability in a public dataset is not a licence to republish content, and derived uses should respect the original terms.
For personal data, presence in a crawl does not remove obligations under applicable privacy law.
7. Practical Notes
- Process from cloud storage in the same region to avoid large egress costs.
- Deduplicate aggressively; near-duplicates are abundant at this scale.
- Prefer WARC plus your own extractor when text quality matters.
- Compare several snapshots to distinguish stable content from transient pages.
8. Summary
Common Crawl publishes periodic, public snapshots of the crawlable web in WARC, WAT, and WET formats, with an index that makes targeted retrieval practical.
It is excellent for scale, history, and breadth, and unsuitable when currency, complete coverage, or client-rendered content is required — a sample of the crawlable web at a point in time, not a copy of the web.