RAG (Retrieval-Augmented Generation)

RAG is the most common alternative to fine-tuning for the “missing domain knowledge” problem we saw in Chapter 1. Instead of stuffing the information into the model’s weights, we give the model an “open book” so it can look the answer up before it replies.

The scenario: the same customer-support company from earlier chapters wants its assistant to answer employee questions about internal policies — “how many vacation days do I get if I have been at the company for 3 years?”. Unlike the tone problem we solved with fine-tuning in Why (usually) not to fine-tune (Chapter 4), the problem here is concrete, verifiable information, and it can change when the company updates its policies — fine-tuning would be the wrong tool, because “baking” factual information into the weights is brittle, and every update would mean retraining.

The full flow, with the real scenario

This phase happens once, when you load or update the knowledge base.

This phase happens every time a user asks something.

Imagine this chunk in the company’s knowledge base:

Vacation policy — Excerpt. Employees with fewer than 2 years of seniority are entitled to 15 business days of vacation per year. Employees with 2 to 5 years of seniority are entitled to 20 business days. Employees with more than 5 years are entitled to 25 business days.

Embeddings: the paragraph becomes a numeric vector, using the same mechanism from the Embeddings chapter in Part I.
Vector database: that vector is stored together with the original text and metadata (“document: vacation policy, section: seniority”) — Vector databases (Chapter 5b) goes deeper on this.
The query: the employee asks, and their question also becomes a vector, with the same embeddings model.
Retrieval: you compute how close the question vector is to the vector of each stored chunk — literally the same cosine similarity from Part I, except now you are comparing whole paragraphs instead of individual words.

How many chunks should you retrieve? The top_k parameter: in this chapter’s code we use top_k=3, but that number is not arbitrary — it is a design decision with a real trade-off. A top_k that is too low (1, for example) risks missing relevant information if it is split across more than one chunk in the knowledge base. A top_k that is too high (20, for example) floods the model’s context with chunks of declining relevance, and brings back the “needle in a haystack” problem from Base-model limits (Chapter 1) — the more irrelevant context surrounds the right answer, the harder it is for the model to find it. A common starting point in production is between 3 and 10, but the right way to tune it is not intuition: it is to run RAG evals (RAGAS, LLM judges and evals, Chapter 3) trying different top_k values on the same set of test questions, and keep the one that maximizes context precision and recall without unnecessarily inflating the cost per query.
Grounded generation: the system injects the retrieved chunk into the prompt, with an explicit instruction: “Answer using ONLY the information provided. If it is not here, say you don’t know.” The model answers correctly not because it memorized this company’s policy — it never saw it during pretraining — but because it has it right in front of it, in its context window, at generation time. RAG does not change the fundamental mechanism from Part I (predict the most likely next token given the context) — it changes what is in the context so the correct answer becomes the most likely one.

Chunking

Splitting long documents into smaller chunks before you vectorize them and store them. As a starting point, it is common to try chunks of 300-500 tokens and then adjust the size with evals, based on the structure of the documents and the real questions.

Overlap between chunks: it is good practice for consecutive chunks to overlap partially — typically 10-20% of the chunk size, or about 50 tokens in a 300-500 chunk. That way, if an important idea lands right on the boundary between two chunks, at least one of them contains it whole, instead of both having it cut in half. A small cost in redundant storage, in exchange for a lower risk of losing context right at the chunk boundaries.

There are three strategies, from simplest to most expensive:

Fixed-size split — fast, but it can cut an idea in half.
Recursive / structural split — split first by paragraph; if a chunk is still too large, fall back to line, then to sentence. It is a backoff pattern, like the one for n-grams in Part I: use the most specific unit available, and drop to a more generic one only when you have to.
Semantic split — uses embeddings or an LLM to group sentences that talk about the same topic. The most precise, and also the most expensive.

Why it is necessary — three different reasons:

Reason	Where it happens	What happens if you ignore it
Technical limit of the embeddings model	At vectorization time	You simply cannot generate the vector for a text that is too long
Vector dilution	Also at vectorization time, but it is a quality problem, not a hard limit	A giant chunk produces a “blurry average” vector that does not represent any topic well — search loses precision
Generator context window	After retrieval, when you assemble the final prompt	The “needle in a haystack” from Chapter 1 — the model cannot find the answer well among too much context

It is important to keep location metadata (section, page) so you can retrieve the neighboring chunk if you need more context, and so you can cite the source afterward.

HyDE: Hypothetical Document Embeddings

A subtle problem: if the employee asks in colloquial language —“hey, if I take three weeks of vacation in a row, am I going to get in trouble?”— that text uses a vocabulary (“take”, “get in trouble”) very different from the formal policy (“seniority”, “business days”, “entitled to”). Both texts are conceptually related, but they can end up with a lower vector similarity than you would want, simply because of a difference in register.

HyDE solves this by inserting an intermediate step, before you touch the vector database: you ask the LLM to generate a complete hypothetical answer — based on its general knowledge of the topic, not on profiling who is asking — written naturally in the typical register of the domain. You vectorize that hypothetical answer, and you use that vector to search.

An important nuance: HyDE does not reduce how many results you get — a vector search always returns a ranked list, no matter how vague the query is. What HyDE does is improve the ranking: the correct chunk, which with the original question might have landed in a poor position because of vocabulary differences, moves toward first place because you are now comparing “formal register against formal register”.

And something worth making explicit: the hypothetical answer is never shown to the user and is not used to generate the final answer — it is disposable “bait” that only serves to search. Once it has found the real chunks, it is discarded. The final grounded-generation prompt uses the retrieved real chunks together with the user’s original question, not the hypothetical answer.

Do you always use HyDE? No — it is optional, and it has a real cost: an extra LLM call before you even touch the vector database, with its own time to first token and its own token cost (remembering the inference trilemma from Part III). It is worth it when there is a known, consistent register gap between how people ask and how the documents are written (colloquial questions vs. formal legal-style regulations); it adds little when the questions already look like the vocabulary of the documents (for example, a system used only by specialists). The right way to decide is not intuition — it is to run RAG evals (RAGAS, Chapter 3) comparing the same set of test questions with and without HyDE, and see whether the gain in context precision and recall justifies the extra cost and latency.

Contextual Retrieval: solving the chunking problem with even more precision

We already saw the problem of poorly tuned chunking — a chunk that is too aggressive can lose the context that gives the fragment its meaning (remember the example of “employees with 2 to 5 years of seniority are entitled to 20 business days” without the previous sentence that makes clear it is talking about vacation). On September 19, 2024, Anthropic published the Contextual Retrieval technique on its official blog, Contextual Retrieval. It attacks this problem directly, with measured and verified results, not just theoretical ones.

The problem, precisely: when you split a long document (for example, a financial report covering several quarters), an isolated chunk like “the company’s revenue grew 3% versus the previous quarter” loses all the information that gives it meaning — which company? which quarter? That loss of context can make a RAG system fail to find the correct answer, even when the relevant chunk is technically in the database.

The solution, in two techniques combined:

Contextual embeddings: before generating each chunk’s embedding, you use an LLM to generate a short context (50-100 tokens) that situates that fragment inside the full document — something like “This chunk belongs to ACME Corp’s second-quarter 2023 financial report, in the revenue results section” — and you prepend that context to the chunk’s original text before vectorizing it. Now the embedding captures both the content and its location inside the larger document.
Contextual BM25: the same context-enriched text (not just the original chunk) is also used to build the keyword search index (BM25, which we already saw in the Embeddings chapter in Part I) — combining semantic search and keyword search over the same enriched context, not just over the original raw text.

The results, measured by Anthropic across multiple knowledge domains (code, fiction, ArXiv papers, scientific papers), using as the metric the failure rate of retrieving the correct chunk among the top 20 results:

Configuration	Retrieval failure rate	Reduction vs. traditional RAG
Traditional RAG (baseline)	5.7%	—
+ Contextual embeddings	3.7%	35% fewer failures
+ Contextual BM25 (full hybrid search)	2.9%	49% fewer failures
+ Reranking (additional step, see below)	1.9%	67% fewer failures

What is reranking? An additional step, optional but effective: after the initial retrieval (fast, using cosine similarity over the embeddings), a specialized and more computationally expensive model — a reranker — re-evaluates the best candidates, comparing the question directly against each candidate more deeply than a simple comparison of precomputed vectors. It is slower than the initial vector search, which is why it is only applied to an already-reduced set of candidates, not to the entire database — a fast filter first, followed by a more careful judgment only on the finalists. In the cited experiment, Anthropic initially retrieved 150 chunks, reordered them, and kept the top 20.

Real cost of implementing this: generating the context for each chunk requires an extra LLM call during indexing (not on every query, only when you build the database) — a cost worth managing with prompt caching (the same concept from Part III) when you generate contexts for multiple chunks of the same source document, and by using a cheap, fast model for this specific task, not the most expensive model available.

Why this matters for our own HR example: if we applied this technique to our vacation-policy base, each policy chunk would have, prepended, something like “This chunk belongs to the Vacation Policy in effect since January 2026, seniority calculation section” — solving at the root the problem we identified earlier, where a poorly cut chunk could lose the reference to “vacation” or “seniority” that gave the day-count its meaning.

Under the hood: the full pipeline

# Setup (once, when you load the knowledge base)
chunks = chunk_documents(documents, chunk_size=300)
for chunk in chunks:
    vector = embedding_model.encode(chunk.text)
    vector_db.store(vector, chunk.text, chunk.metadata)

# On every user query
question = "how many vacation days do I get if I have been at the company for 3 years?"

# (Optional: HyDE) generate a hypothetical answer before searching — discarded afterward
hypothetical_answer = llm.generate(f"Answer in general terms: {question}")
search_vector = embedding_model.encode(hypothetical_answer)  # or encode(question) without HyDE

relevant_chunks = vector_db.search(search_vector, top_k=3)

# The final prompt uses the ORIGINAL question, not the hypothetical
final_prompt = f"""
Answer using ONLY these documents. If it is not here, say so.

{relevant_chunks}

Question: {question}
"""
answer = llm.generate(final_prompt)

RAG solves “missing domain knowledge” and “missing up-to-date information” from Chapter 1 — updating the vacation policy next year is as simple as replacing that document in the vector database, with no retraining. But for the model to also be able to act (file the request directly, not just inform), you need the piece from MCP (Chapter 5c): the Model Context Protocol.