A language model only knows what it absorbed during training, frozen at some point in the past. It has never seen your company’s documents, your notes, or anything that happened after its training cutoff. So how do AI tools answer questions about exactly those things? The most common answer is RAG, short for retrieval-augmented generation, and the idea is simpler than the name.
The problem RAG solves #
Ask a plain model “what is our refund policy?” and it has two options, both bad. It can admit it does not know, or it can make something up that sounds like a refund policy. It was never trained on your policy, so it genuinely cannot know.
You could try pasting your entire document library into every prompt, but that does not scale: it blows past the context window and costs a fortune in tokens. You need a way to put only the relevant piece in front of the model at the moment it is needed.
The core idea #
RAG splits answering into two steps:
- Retrieve. First, search your collection of documents for the pieces most relevant to the question. For “what is our refund policy?”, that means pulling the few paragraphs that actually discuss refunds.
- Generate. Then hand those retrieved pieces to the model along with the question, and ask it to answer using this material. The model reads the real policy you just gave it and answers based on that, not on its frozen training memory.
That is the whole trick. Retrieve the right context, then let the model generate an answer grounded in it. “Augmented” just means you augmented the prompt with retrieved information before generating.
How the retrieval actually works #
The generation half is a normal model call. The interesting engineering is in retrieval, and it usually looks like this:
- Chunk your documents. Long documents get split into smaller passages, so you can retrieve a relevant paragraph instead of a whole 80-page manual.
- Index them for meaning-based search. The common approach uses embeddings: each chunk is turned into a list of numbers that captures its meaning, so passages about “money back” and “refunds” land near each other even without sharing exact words. These go into a vector database built for finding the closest matches fast.
- Search at question time. When a question comes in, it gets turned into the same kind of numeric representation, and the system pulls the chunks whose meaning is closest.
- Stuff and generate. The top matches get inserted into the prompt, and the model answers from them.
You do not have to build all of this by hand anymore, plenty of tools package it up, but knowing the shape helps you understand why RAG systems behave the way they do.
Why RAG is everywhere #
RAG became the default way to make AI useful on real-world data because it hits several needs at once:
- Current and private information. It works on data the model was never trained on: your docs, today’s numbers, internal wikis, a product catalog.
- Fewer hallucinations. Grounding the answer in retrieved source material is one of the strongest ways to cut down on confident fabrication, because the facts come from your documents rather than the model’s guesswork.
- Citations. Because you know which chunks were retrieved, you can show the user exactly where the answer came from, which builds trust and lets people verify.
- No retraining. Update a document and the next question retrieves the new version. You do not have to retrain or fine-tune the model to teach it new facts, you just change what it can retrieve.
Where it falls short #
RAG is not magic. If retrieval pulls the wrong chunks, the model answers confidently from the wrong material, so the quality of the search matters enormously. If the answer is spread across many documents in a way simple retrieval misses, it can stumble. And it only knows what is in the collection you gave it. Garbage or missing documents in, garbage or gaps out.
Still, for the extremely common problem of “make an AI answer questions about my stuff,” RAG is the workhorse. Retrieve the relevant material, hand it to the model, let it generate a grounded answer. It is one of the most useful patterns in applied AI, and now you know what is happening behind it.