How Does RAG Work? – The Retrieval and Generation Steps

A step-by-step walkthrough of retrieval-augmented generation: ingestion, chunking, embedding, retrieval, re-ranking, and grounded generation.

RAG runs in two halves that execute at different times. The indexing half runs ahead of the question; the retrieval half runs while the user waits. Nearly every quality problem traces to the first half, and nearly every latency problem to the second.

1. Step One: Ingestion

Source documents are collected and converted to plain text. For web sources this means fetching the page and stripping navigation, scripts, styling, cookie notices, and footers.

• Why it matters: boilerplate that survives ingestion becomes retrievable content and competes with real answers.
• What to keep: headings, paragraphs, list structure, tables, and the source URL.
• What to record: a stable document identifier and a timestamp, so the index can be refreshed selectively.

2. Step Two: Chunking

Documents are split into passages. The split determines what can be retrieved at all, since retrieval returns whole chunks.

document → [chunk 1] [chunk 2] [chunk 3] …
           each self-contained enough to answer alone

Splitting on structure — headings and section boundaries — beats splitting on a fixed character count, because it keeps a definition with its example and a table with its header.

3. Step Three: Embedding and Storage

Each chunk is passed through an embedding model and stored as a vector alongside its text and metadata.

The same embedding model must be used for chunks and for queries; vectors from different models are not comparable.

4. Step Four: Retrieval

At question time the query is embedded and the nearest chunks are fetched. Two refinements are near-universal in production:

• Hybrid search. Run keyword search alongside vector search and fuse the rankings. Vectors handle paraphrase; the lexical index anchors identifiers, error codes, and product names.
• Metadata filters. Constrain by date, source, or tenant before or during the search, so the answer draws only from permitted, current documents.

5. Step Five: Re-Ranking

Similarity search is fast and approximate. A re-ranker takes the top few dozen candidates and scores each one against the query with a more expensive model that reads the query and passage together.

This is usually the cheapest large quality win available: retrieve broadly, re-rank precisely, then keep only the handful of passages that fit the context budget.

6. Step Six: Grounded Generation

The surviving passages are inserted into the prompt with an instruction to answer only from them and to say so when they are insufficient. The model composes the answer and the application returns it with its citations.

The refusal instruction is what separates grounding from decoration. Without it, a model that retrieved nothing useful will still answer from its weights, and the answer will look identical to a grounded one.

7. Where the Steps Fail

• Ingestion — boilerplate retained, tables flattened into unreadable runs of text.
• Chunking — passages split mid-definition, or so large that similarity is diluted.
• Embedding — model swapped without re-indexing the corpus.
• Retrieval — no lexical component, so exact identifiers never match.
• Re-ranking — skipped, so marginal passages consume the context budget.
• Generation — no refusal instruction, so gaps are filled from memory.

8. Summary

RAG works by turning documents into retrievable passages ahead of time, then selecting the few that answer the current question and letting the model compose from those. Indexing decides what is findable; retrieval and re-ranking decide what is found; the prompt decides whether the model stays inside the evidence.

Debugging follows the same order. Before changing model or prompt, inspect the retrieved chunks — most disappointing answers are retrieval failures wearing a generation costume.

Potential Related Glossary

• Unstructured Data
• Data Wrangling
• Parsing
• Data Fusion