Michael Limberger

Need me? Email mike@limberger.ca

Retrieval-augmented generation is a long name for a simple idea. Find the useful pages, add them to the prompt, then let the model write an answer from that material.

The whole night in one line

The AI does not answer from memory. It pulls the matching passage from your documents first, then writes the reply from that text. Read that twice. Everything that follows just unpacks it.

An open-book test

This is the best teaching tool in the whole deck. If you only keep one mental model for RAG, make it this one.

A closed-book exam tests what you memorized. That is a stock language model, or a fine-tuned one, answering from training. A student under pressure who half-remembers will guess, and a confident guess looks the same as a real answer to anyone in the room. That is the source of AI hallucination: the model is genuinely trying its best from what it remembers, and what it remembers is fuzzy.

An open-book test is different. You still need to be smart. You need to find the right page and understand what it says. You are not relying on memory.

That is RAG. The model's job shifts from "remember everything" to "read well." Modern language models are very good at reading. They are not as reliable at remembering. RAG plays to the strength.

Retrieve

Your question is compared against every chunk of your documents. The comparison is not keyword matching. It is a math comparison of meaning. We go deep on how this works in part 11. The system picks the top five passages closest in meaning to your question.

Why not keyword matching? Because language varies. Someone might ask "how do I file a sharps incident report" while the policy says "needle-stick injury reporting procedure." A keyword search misses entirely. A meaning-based search nails it.

Plain English: it finds the five most relevant pieces of your documents, even if they do not share any exact words with your question. Five is a default, tunable. Enough context to answer well, not so much that you bury the model or blow past its context window (how much text it can see at once). Part 8 gets into what happens when that number is set carelessly.

Augment

The AI receives your original question plus those five passages, bundled together into one enriched prompt. "Augment" just means the question got padded with relevant context before the model ever saw it.

A simplified version of what the model actually receives:

Here are some passages from our policy documents:

[Passage 1, from Policy H.03 Section 4.2]
[Passage 2, from Policy A.11 Section 1.1]
[Passage 3, from Policy H.14 Section 3.5]
...

Now answer this question using only the passages above:
"How do I report a needle-stick injury?"

The model does not know any of this glue was added by software. From its point of view, it just received a prompt that happens to have the answer material sitting right there in it. It reads. It answers from what it read.

Generate

The AI reads the passages and writes an answer from them, not from its training or memory. Because the passages came from real documents, every answer is citable.

The model can name the exact source policy and section ("According to Policy H.03, Section 4.2.1..."), so a human can verify the answer in about ten seconds. The reader does not have to trust the AI. They can check.

That citation detail is not a nice-to-have. It is the difference between "an AI said so" and "an AI said so and here is the page." In a clinical, legal, or policy setting, that gap is everything.

What the answer sounds like

One of those you can take to a compliance officer. The other you cannot.

Why this beats fine-tuning

Fine-tuning hopes the model will remember enough facts to answer correctly. RAG just hands it the page. The first is gambling. The second is engineering.

Where this can still go wrong

RAG is not magic. It fails in predictable ways. The retrieval picks the wrong passages (chunk size, embedding choice, hybrid search, parts 6 through 11). The retrieved passages exceed the model's context window (the 2048-token gotcha, part 8). The model still hallucinates inside the passages it was given (less common, but possible; good prompting and citation discipline mitigate).

What if the answer is not in the documents at all? Then a well-built RAG system says so. That is test number two in the live demo: ask about a feature that does not exist and watch it admit the gap. The honesty is a feature.

Does the model still use its training at all? Yes, for language, reasoning, and how to structure an answer. It just does not use training as the source of facts. Think of training as "how to read and write," and retrieval as "what is true here."

Is this just search with extra steps? Sort of, yes. RAG is "do a smart search, then have a smart reader write up the result." That is most of what it is. The trick is doing both halves well.

Closed book to open book

The model stops guessing from memory and starts reading from your documents. This part is the concept. The next ones are the engineering.