Why retrieval exists
A model knows what was in its training data, up to a cutoff date. It does not know your documents, your database, or anything from last week. Retrieval is how you close that gap at the moment of the question.
The three alternatives, and why they lose
Put everything in the prompt. Works until it does not. Your document set is almost certainly larger than any context window, you pay for every token on every request, and quality degrades as the window fills. Even with a million token window this is the wrong default.
Fine tune on your data. Fine tuning is good at teaching format, tone, and task shape. It is bad at teaching facts you need to be able to update. Retraining because a policy changed on Tuesday is not a workflow.
Just ask the model. It will answer confidently from nothing. This is the worst option and the easiest to fall into.
What retrieval does instead
Fetch the few relevant pieces at question time and put only those in the context.
- The user asks something.
- You search your content for passages relevant to that question.
- You put the top few into the prompt alongside the question.
- The model answers using them.
That is the whole idea. Everything else is implementation detail.
Why it wins
It updates instantly. Change the document, and the next answer uses the new version. No retraining, no redeploy.
It scales past the window. A million documents, five in the prompt.
It cites. You know which passages were used, so the answer can link to them. This is often more valuable than the answer, because it lets someone check.
It is cheap. You pay for five passages, not the whole corpus.
Where it is the wrong tool
Retrieval answers questions whose answers sit in a findable passage. It is poor at:
- Aggregation. "How many support tickets mentioned billing last month" is a database query, not a search.
- Whole document reasoning. "Summarise this contract" needs the contract, not three chunks of it.
- Anything needing every match rather than the best few.
Reaching for RAG when you needed SQL is one of the most common mistakes in this area.
The honest trade
Retrieval moves the hard problem rather than removing it. The model is no longer guessing, but the answer is now only as good as what search returned.
If retrieval surfaces the wrong passages, the model answers confidently from the wrong passages. Most of the work in a real system is in retrieval quality, which is why when RAG fails exists.
Next
Embeddings explained covers how the search step actually finds things.