What Is RAG? – Definition and How It Works
Learn what retrieval-augmented generation is, how the retrieve-then-generate loop works, why it beats fine-tuning for fresh facts, and where it fails.
Retrieval-augmented generation puts source documents into a model's context window at question time, so the answer is grounded in text the model can actually read rather than in whatever its weights happen to encode.
1. What Is Retrieval-Augmented Generation?
RAG is a pattern, not a model. It wraps a language model in a retrieval step that selects relevant passages before generation begins.
- Key idea: separate knowledge from the model. Knowledge lives in a searchable store; the model supplies reasoning and phrasing.
- Mechanism: embed the question, find the closest passages, paste them into the prompt, generate an answer that cites them.
- Goal: current, verifiable answers from a model whose training data is frozen.
Example of the Prompt Shape
System: Answer only from the context. If the context does not contain the
answer, say so.
Context:
[1] <passage retrieved from the store>
[2] <passage retrieved from the store>
Question: <the user's question>
The instruction to refuse when the context is insufficient is what converts retrieval into grounding.
2. How the Pipeline Works
A RAG system has an offline half and an online half, and most failures come from the offline half.
Indexing, done ahead of time:
- Collect source documents — pages, PDFs, tickets, product data.
- Clean them into plain text, stripping navigation, boilerplate, and markup.
- Split into chunks small enough to be specific and large enough to be self-contained.
- Convert each chunk into an embedding vector and store it with its text and metadata.
Retrieval and generation, done per question:
- Embed the question with the same model used for the chunks.
- Find the nearest chunks by vector similarity, often combined with keyword search.
- Re-rank the candidates and keep the few that fit the context budget.
- Generate the answer from those chunks and return it with its sources.
3. Why Chunking Decides Quality
Chunk boundaries determine what can be retrieved at all. A chunk that splits a table from its header, or a definition from its example, retrieves as noise.
| Chunk size | Retrieval precision | Risk |
|---|---|---|
| Very small | High — the match is tightly on topic | Context is missing, answers become fragmentary |
| Moderate | Balanced, the common default | Requires clean structural boundaries |
| Very large | Low — many topics per chunk | Wasted context budget, diluted similarity scores |
Splitting on document structure — headings, sections, list boundaries — outperforms splitting on a fixed character count, because it keeps semantically whole units together.
4. RAG Compared With Fine-Tuning
The two techniques solve different problems and are often confused.
| Question | RAG | Fine-tuning |
|---|---|---|
| Adds new facts? | Yes, immediately | Poorly and expensively |
| Adds new behaviour or format? | Limited | Yes, this is its strength |
| Updating cost | Re-index the changed documents | Re-train and re-deploy |
| Traceable to a source? | Yes, chunks can be cited | No |
Fresh or proprietary facts belong in retrieval. Consistent tone, structure, or a domain-specific output format belongs in fine-tuning.
5. Where RAG Fails
- Retrieval misses. If the right chunk never surfaces, the model answers from its weights, and the answer looks just as confident.
- Stale index. RAG is only as current as its last crawl or sync. A daily index cannot answer about this morning.
- Aggregation questions. "How many of our documents mention X" is a database question; top-k similarity search does not count.
- Dirty source text. Navigation menus, cookie banners, and boilerplate retrieved as content crowd out the real passage.
6. Real-World Examples
- Documentation assistants index a product's docs so answers track the current release rather than the training cutoff.
- Support automation retrieves resolved tickets that match a new one, giving the model a precedent to work from.
- Market and competitor monitoring indexes freshly collected web pages so a question about current positioning is answered from this week's text.
- Internal search replaces keyword-only lookup, matching on meaning so a question phrased differently from the document still finds it.
7. Summary
RAG grounds a language model in documents fetched at question time, which fixes the two structural limits of a fixed model: an outdated training cutoff and no access to private data. The generation step is rarely the hard part.
Quality is decided upstream — by how clean the source text is, how sensibly it is chunked, and how reliably retrieval surfaces the right passage. Teams that treat RAG as a prompt-engineering problem usually discover it was a data-pipeline problem.