What Is Training Data? – Definition and How It Works
Learn what training data is, where web-scale corpora come from, how filtering and deduplication work, and why provenance and licensing matter.
Training data is the corpus a model learns from. For language models it is assembled at web scale, then filtered heavily — and what survives filtering shapes everything the model can do.
1. Where It Comes From
| Source | Character |
|---|---|
| Web crawls | Vast, varied, and noisy |
| Curated collections | Books, papers, reference works — higher quality, smaller |
| Code repositories | Structured, licence-sensitive |
| Licensed corpora | Negotiated access to proprietary material |
| Synthetic data | Model-generated, used to fill gaps |
Web crawl material dominates by volume; curated sources dominate by contribution per token.
2. Filtering Is Most of the Work
Raw crawl data is largely unusable. Typical stages:
- Language identification, keeping the target languages.
- Quality heuristics — length, punctuation, symbol ratios, boilerplate detection.
- Deduplication, exact and near-duplicate, which is critical at scale.
- Safety filtering, removing categories excluded by policy.
- Personal-data reduction, removing identifiers where detectable.
- Benchmark decontamination, removing evaluation sets so scores mean something.
Deduplication matters more than it sounds: duplicated text is memorised disproportionately and wastes capacity.
3. Composition Shapes Capability
A corpus heavy in code produces stronger coding ability. One thin in a language produces weak performance there. Domains absent from the corpus are absent from the model.
This is why "the model does not know X" is usually a data question rather than a capability question — and why retrieval, not retraining, is the practical fix.
4. Provenance and Licensing
Public availability is not a licence. Web text carries copyright, code carries licences with obligations, and personal data carries legal duties independent of where it was found.
Serious training pipelines track provenance per document so licensing questions can be answered afterwards. Retrofitting that record is close to impossible.
5. The Publisher's Side
Sites influence inclusion through crawler directives, and increasingly distinguish training collection from live retrieval, allowing one and restricting the other.
Blocking future collection does not remove material already gathered, which is why the decision is usually framed forward rather than retroactively.
6. Synthetic Data
Model-generated data fills gaps where real examples are scarce, and is widely used for instruction tuning. Its risk is compounding: training on outputs of the same model family can amplify existing biases and errors.
It works best as a supplement anchored to verified sources rather than as a substitute for real data.
7. Evaluation Depends on Clean Separation
If evaluation material appears in training data, scores measure memorisation. Decontamination is therefore part of dataset construction rather than an afterthought, and reported benchmark results depend on it having been done properly.
8. Summary
Training data is a heavily filtered assembly of web crawls, curated collections, code, licensed corpora, and synthetic text, with deduplication and decontamination doing much of the quality work.
Composition determines capability, provenance determines what can be answered about licensing later, and nothing after the cutoff is in there at all — which is why current information has to arrive through retrieval.