What Is Chunking for RAG? – Definition and How It Works

Informational P2 what is chunking for rag

Learn why chunking decides RAG quality, how the main strategies differ, and what metadata every chunk must carry.

Chunking splits documents into passages small enough to embed and retrieve. It is the stage that most determines retrieval quality, and the one most often treated as a detail.

1. Why It Matters More Than the Model

Retrieval returns chunks, so a chunk is the unit of everything downstream. Get it wrong and no embedding model recovers:

Too large — one chunk covers several topics, its embedding averages them, and it matches everything weakly and nothing strongly.
Too small — the chunk lacks the context needed to be interpretable, and a retrieved fragment answers nothing.
Split mid-thought — the sentence that contained the answer is cut across two chunks, and neither ranks.

Changing the chunking strategy routinely moves retrieval quality more than changing the embedding model.

2. The Strategies

Strategy	How	Trade-off
Fixed size	N characters or tokens	Simple; ignores meaning
Fixed size with overlap	N with M overlap	Cheap fix for boundary loss
Sentence-based	Group whole sentences	Respects language boundaries
Paragraph-based	Split on blank lines	Respects author intent
Structural	Split on headings	Best for documentation
Semantic	Split where meaning shifts	Expensive, often worth it
Recursive	Try structural, fall back smaller	Sensible default

3. Overlap Is Insurance, Not a Strategy

Overlapping chunks by 10–20% means a sentence cut at a boundary still appears whole in one of the two chunks.

It costs storage and creates near-duplicate retrievals that must be deduplicated after the fact. It is a mitigation for bad boundaries, not a substitute for good ones — a structural split needs far less of it.

4. Structure Beats Length for Collected Content

Scraped pages already carry structure: headings, sections, list items, table rows. Splitting on that produces chunks that correspond to something a human would recognise as a unit.

def chunk_by_heading(html_sections, max_chars=1200):
    for section in html_sections:                 # (heading, text) pairs
        heading, text = section
        if len(text) <= max_chars:
            yield {"heading": heading, "text": text}
            continue
        # oversized section: split on paragraphs, keeping the heading on each piece
        buf = []
        for para in text.split("\n\n"):
            if sum(len(p) for p in buf) + len(para) > max_chars and buf:
                yield {"heading": heading, "text": "\n\n".join(buf)}
                buf = []
            buf.append(para)
        if buf:
            yield {"heading": heading, "text": "\n\n".join(buf)}

Carrying the heading onto every chunk from that section is the highest-value trick here: it gives an orphaned paragraph the context it needs to be retrievable.

5. What Never to Split Across Chunks

A table and its header row. A table body without headers is uninterpretable.
A code block.
A list and the sentence introducing it.
A definition and its term.

These are the cases where a mechanically correct split produces a chunk that is worse than useless, because it looks like content and answers nothing.

6. Every Chunk Needs Metadata

A chunk stripped of provenance cannot be cited, verified, or filtered:

{
  "text": "...",
  "source_url": "https://example.com/guide",
  "title": "Guide to X",
  "heading_path": ["Guide to X", "Configuration", "Timeouts"],
  "chunk_index": 7,
  "fetched_at": "2026-08-18T14:00:00Z",
  "content_hash": "…",
}

heading_path is the one worth insisting on. It lets an answer say where a fact came from within a document, and lets retrieval filter by section.

7. Boilerplate Must Go First

Chunking raw page text produces chunks of navigation menus and cookie notices, which then get retrieved. Run content extraction before chunking, not after.

This is the single most common defect in a RAG pipeline built over scraped pages: the index fills with site furniture, and retrieval returns footers.

8. Summary

Chunking splits documents into retrievable passages, and it affects RAG quality more than the embedding model does — too large blurs the embedding, too small loses context, and a bad boundary hides the answer.

Split on the structure collected pages already have, carry the heading path onto every chunk, never split a table from its header or a list from its introduction, attach provenance metadata, and strip boilerplate before chunking rather than indexing the navigation.