What Is Chunking? – Definition and How It Works
Learn what chunking is in retrieval pipelines, how strategies differ, why boundaries decide retrieval quality, and how to choose size and overlap.
Chunking splits documents into passages for indexing and retrieval. Because retrieval returns whole chunks, the boundaries decide what can ever be retrieved — which makes it the highest-leverage step in a RAG pipeline.
1. Why Chunk at All
- Precision. A whole document averages into a vague point in embedding space; a passage is specific.
- Context budget. Only a few passages fit in a prompt, so they must be small enough to be worth their space.
- Citation. A cited passage is checkable; a cited document is not.
2. The Strategies
| Strategy | How it splits | Trade-off |
|---|---|---|
| Fixed size | Every N characters or tokens | Simple; cuts through sentences and tables |
| Sentence-aware | On sentence boundaries | Better; still ignores document structure |
| Structural | On headings and sections | Best for documents with real structure |
| Recursive | Tries structural, falls back to smaller units | Good general default |
| Semantic | Splits where topic shifts, measured by embeddings | Highest quality, highest cost |
Structural splitting outperforms fixed-size on almost any real corpus, because it keeps a definition with its example and a table with its header.
3. Size and Overlap
Smaller chunks retrieve precisely and may lack the context to answer alone. Larger chunks carry context and dilute similarity, so they match more queries less specifically.
Overlap — repeating a little text between adjacent chunks — prevents an answer that straddles a boundary from being lost. It costs storage and introduces near-duplicate results, so it should be modest rather than generous.
4. Carry the Context Into the Chunk
A passage retrieved alone often lacks the subject, because the subject was stated in a heading above it.
Prepending the document title and section heading to each chunk's stored text is a small change with an outsized effect: it makes passages self-contained for both the retriever and the model reading them.
5. Content That Needs Special Handling
- Tables — keep whole with their headers; splitting a table produces unusable fragments.
- Code — split on function or class boundaries, never mid-block.
- Lists — keep the introducing sentence with the list.
- Long prose without headings — semantic or sentence-aware splitting is the only sensible option.
6. Evaluating a Strategy
Chunking quality is measurable. Build a small set of questions with known answers, then check whether retrieval surfaces the chunk containing each answer.
That test finds boundary problems immediately, and it is far more informative than inspecting chunks by eye.
7. Common Mistakes
- Fixed-size splitting on structured documents, destroying the structure that made them retrievable.
- No overlap on prose, losing answers that straddle a boundary.
- Excessive overlap, filling results with near-duplicates.
- Chunking before cleaning, so navigation and boilerplate become retrievable passages.
- Changing strategy without re-indexing, leaving a corpus with mixed boundaries.
8. Summary
Chunking determines the units retrieval can return, so it decides the ceiling on answer quality. Structural splitting, modest overlap, and heading context prepended to each chunk cover most needs.
Clean the text before splitting it, and evaluate with a question set rather than by inspection — boundary errors are invisible until a specific answer cannot be found.