RAG 101: How Retrieval-Augmented Generation Makes AI Models Smarter

Table of Contents

• The Problem RAG Solves
• What Retrieval-Augmented Generation Actually Is
• A Worked Example
• RAG vs Fine-Tuning vs Long Context
• The Anatomy of a RAG Pipeline
• Six Ways RAG Goes Wrong
• How to Tell If It Is Working
• Your First Build in 90 Minutes
• Where RAG Fits Alongside Agents
• Hybrid, Graph and Agentic RAG
• Should You Build One?
• Three Things People Get Wrong
• Frequently Asked Questions
• Final Thoughts

Ask any language model about your company’s refund policy and you will get a confident, well-written, entirely invented answer. The model has never seen your policy. It has seen ten thousand refund policies, and it will produce something that sounds exactly like one.

This is not a bug you can prompt your way out of. It is what the technology does. Retrieval-augmented generation is the standard fix, and understanding it properly is the difference between building something people trust and building something that embarrasses you in month three.

The Problem RAG Solves

A language model has two limitations that matter here, and they are structural rather than temporary.

It only knows what it was trained on. Your internal documents, your product specs, last week’s pricing change, the contract signed in March – none of it exists as far as the model is concerned.

It cannot tell you what it does not know. Asked a question outside its knowledge, a model does not stop. It generates the most plausible continuation, which reads identically to a correct answer. Confidence is not correlated with accuracy.

Retrieval-augmented generation addresses both by changing the question. Instead of asking the model what it knows, you find the relevant material first and ask the model to answer using only that.

What Retrieval-Augmented Generation Actually Is

Strip away the terminology and RAG is four steps.

1. Prepare. Your documents are split into chunks. Each chunk is converted into a vector – a list of numbers representing its meaning – and stored. This happens once, ahead of time.

2. Retrieve. A question arrives and is converted into a vector the same way. The system finds the stored chunks whose vectors sit closest to it.

3. Augment. Those chunks are inserted into the prompt alongside the question, with an instruction to answer only from the provided material.

4. Generate. The model answers, and because you know which chunks it received, the answer can cite its sources.

That is the whole idea. The term retrieval-augmented generation describes exactly what happens: retrieval, then augmentation of the prompt, then generation. The approach was formalised in a 2020 research paper and has since become the default architecture for grounding models in private data.

What makes it powerful is not sophistication. It is that updating your knowledge means editing a document, not retraining anything.

A Worked Example

Concrete beats abstract. Say an employee asks: how many days of parental leave do I get?

Without retrieval, the model answers from training data – a blend of policies from thousands of companies. It might say twelve weeks. It might say six. Both are wrong for you, and both sound authoritative.

With retrieval-augmented generation, the flow changes. The question becomes a vector. The system searches your indexed HR handbook and finds three chunks about leave entitlement. Those chunks go into the prompt. The model reads them and answers: eighteen weeks at full pay for the primary caregiver, citing section 4.2 of the handbook.

The employee can click through and verify. If the policy changes next month, someone updates the handbook and re-indexes it. Nothing is retrained, and the next answer is correct.

Notice what did the work. Not the model’s intelligence – the retrieval. If the search had returned the section on annual leave instead, the answer would have been fluent, confident and wrong.

RAG vs Fine-Tuning vs Long Context

These get compared constantly and they solve different problems.

	Retrieval-augmented generation	Fine-tuning	Long context
Teaches facts	Yes, reliably	Unreliably	Yes, within limits
Teaches style or format	No	Yes, very well	Partially
Updating knowledge	Edit a document	Retrain	Resend everything
Citations	Built in	Impossible	Possible but weak
Cost pattern	Storage plus per-query	Large upfront	High per query
Scales to large corpora	Yes	Yes	No

The short version: fine-tuning changes how a model behaves, retrieval changes what it knows. If you want a model that always responds in your brand voice, fine-tune. If you want it to know your documents, use RAG.

Long context deserves a note because it is frequently proposed as a replacement. Modern models accept enormous prompts, so why not paste everything in? Three reasons: cost scales with every token on every query, retrieval accuracy degrades when relevant material is buried in a very long prompt, and most real corpora are far larger than any context window. Long context complements retrieval-augmented generation – it does not replace it.

The Anatomy of a RAG Pipeline

Six components, and knowing what each does makes debugging enormously easier.

Document parsing. Turning files into clean text. Underrated to the point of negligence. A PDF with tables flattened into character soup poisons everything downstream, which is why engines built around serious parsing – our guide to RAGFlow covers one – exist at all.

Chunking. Splitting text into retrievable pieces. Too large and each chunk contains several topics, diluting its meaning. Too small and context is lost. Chunk on structure – sections, clauses, slides – rather than blindly on token count.

Embedding. Converting chunks into vectors using an embedding model – the MTEB leaderboard is the usual reference for comparing them. Your retrieval quality has a ceiling set here. A better embedding model improves results more reliably than a better generation model.

Storage and search. Where vectors live and how they are searched quickly. The options and trade-offs are covered in our comparison of vector databases for RAG.

Reranking. Optional, and the highest-return optional step in retrieval-augmented generation. Retrieve fifty candidates cheaply, then use a model that reads the question and each chunk together to pick the best five. Costs milliseconds, improves precision substantially.

Generation. The model, plus a system prompt that forbids answering beyond the supplied context. That instruction is not optional.

Six Ways RAG Goes Wrong

Almost every disappointing system fails in one of these ways, and none of them are the model’s fault.

1. The document was parsed badly. Tables destroyed, headings orphaned, scanned pages ignored entirely. Check what your pipeline actually extracted before blaming anything else.

2. Chunks are the wrong size or shape. Our guide to RAG chunking strategies covers the fix in detail. A definition separated from its term. A table split across two chunks. Retrieval cannot recover information that chunking destroyed.

3. No reranking. Vector similarity finds things that are topically near. It is imprecise about which is actually most relevant. Reranking fixes exactly this.

4. Semantic search cannot find exact terms. Error codes, part numbers, surnames. Dense vectors are bad at these. Hybrid search – combining vector similarity with keyword matching – is the standard remedy and the reason so many systems use it.

5. The knowledge base is stale. Superseded policies sitting alongside current ones, cited with equal confidence. This is a curation problem and no amount of tuning solves it.

6. The prompt permits guessing. Without an explicit instruction to say “this is not in the provided documents”, the model fills gaps. Every production system needs that instruction, tested with questions whose answers genuinely are absent.

How to Tell If It Is Working

Most teams evaluate by asking a few questions and forming an impression. That is how you end up shipping something that fails on the questions you did not think to ask.

Build a set of fifty real questions with known correct answers and known source documents. Then track three numbers.

Retrieval hit rate. How often the correct chunk appears among those retrieved. This isolates retrieval from generation, and it is the number to improve first because nothing downstream can compensate for missing it.

Answer accuracy. How often the final response is correct and properly sourced, judged by a person who did not build the system.

Refusal correctness. How often it declines when the answer genuinely is not there. A system that never refuses is guessing well, not performing well.

Run the set before and after every change. Change one thing at a time. This is unglamorous and it is the only way to know whether your retrieval-augmented generation pipeline is actually improving.

Your First Build, in About 90 Minutes

Reading about retrieval-augmented generation is considerably less instructive than building a small one. Here is a version you can complete in an evening.

Pick twenty documents you know well. Your own notes, your team’s handbook, product documentation. Familiarity is the point – you need to recognise a wrong answer instantly.

Write your twenty questions first. Before touching any tool. Include three whose answers are deliberately not in the documents, because testing refusal behaviour is how you find out whether the system is honest.

Use something that handles the plumbing. A ready-made engine gets you to a working retrieval-augmented generation pipeline in an afternoon instead of a fortnight. Building the components yourself teaches you more but delays the moment you learn whether the idea helps at all.

Accept every default on the first pass. Then run your question set and read the citations behind every wrong answer. The failures will point at exactly one of the six problems listed earlier, and usually the same one repeatedly.

Change one thing. Re-run. Compare. Repeat.

Ninety minutes gets you a working system and, more valuably, an intuition for where retrieval-augmented generation actually breaks. That intuition transfers to every future project regardless of which tools you use.

Where RAG Fits Alongside Agents

The two ideas get conflated constantly, so it is worth separating them.

Retrieval-augmented generation is a way of grounding an answer in specific material. An agent is a system that decides which actions to take to accomplish a goal. They are orthogonal – you can have either without the other.

Where they meet is that retrieval makes an excellent agent tool. Rather than baking one search into a fixed pipeline, you expose retrieval as something the agent can call when it decides it needs information. It might search, read the result, decide the answer is incomplete, and search again with different wording.

Retrieval exposed to an agent this way is usually delivered over MCP. That is more capable and considerably harder to reason about when it misbehaves. The practical advice is the same as everywhere else in this article: get deterministic retrieval-augmented generation working and measurable first, then hand the search decision to an agent if the questions genuinely demand it.

Most internal knowledge assistants never need to make that leap. A well-tuned fixed pipeline answers the overwhelming majority of questions people actually ask, and it fails in ways you can diagnose in minutes rather than hours.

Hybrid, Graph and Agentic RAG

Three variations worth knowing, in roughly the order most teams need them.

Hybrid search. Dense vector similarity plus keyword matching, scores fused. Nearly always an improvement, and the first upgrade to make on any corpus containing identifiers or proper nouns.

GraphRAG. Builds a knowledge graph of entities and relationships across the corpus rather than relying on similarity alone. Useful for questions requiring connections across many documents – which suppliers a policy affects, which projects a person touched. Expensive to build, unnecessary for straightforward lookup.

Agentic RAG. The model decides how to search rather than executing one fixed retrieval. It can reformulate a question, search several times, or check multiple sources before answering. More capable, slower, and harder to debug. This is where retrieval meets agent frameworks – the patterns in our guide to n8n AI agents apply directly.

Adopt these in order and only when a specific failure justifies them. Teams that start with agentic RAG usually have a slow, unpredictable system and no idea which component is responsible.

Should You Build One?

Yes, if you have documents people repeatedly ask questions about, answers need to be traceable, the material changes often enough that retraining is impractical, or the data cannot leave your infrastructure.

Probably not, if your knowledge fits comfortably in a single prompt, the questions are general rather than specific to you, or you have fewer than a hundred documents that nobody struggles to search already.

If you do build, start absurdly small. One knowledge base, twenty documents you know intimately, default settings. Get that trustworthy before adding anything. Our beginner guide to RAGFlow walks through a first deployment, and our look at enterprise RAG deployments covers what changes at scale.

Three Things People Get Wrong About RAG

That a bigger model fixes bad retrieval. It does not. If the correct passage was never retrieved, the model is answering from nothing regardless of how capable it is. Upgrading the model when retrieval is the problem is the single most common misallocation of effort in this field, and it is expensive in both directions – you pay more and improve nothing.

That more context is always better. Stuffing twenty chunks into a prompt instead of five frequently makes answers worse. The relevant passage competes for attention with nineteen mediocre ones, and the model has no reliable way to tell you which it actually used. Retrieve broadly, rerank aggressively, pass few.

That RAG is a product rather than a pipeline. There is no tool you install that gives you good retrieval-augmented generation. There are tools that handle the plumbing well, and there is the work of parsing, chunking, curating and evaluating – which remains yours regardless of what you install. Teams that expect a finished product tend to stall about a month in, when the defaults stop being good enough and nobody has built the evaluation set that would tell them what to change.

Frequently Asked Questions

What does RAG stand for?

Retrieval-augmented generation. Retrieve relevant material, augment the prompt with it, generate an answer from it.

Does RAG eliminate hallucination?

It reduces it substantially and does not eliminate it. A model given the right context and instructed not to guess is far more reliable, but it can still misread a passage. Citations matter because they let a reader verify rather than trust.

Do I need a vector database?

Not below roughly twenty thousand chunks – an in-memory index is fine. Beyond that, or when you need filtering and fast updates, a purpose-built store earns its place.

Which embedding model should I use?

Any current general-purpose model to start. Switching later means re-indexing everything, so if you have a specialised domain, test two or three against your own questions before committing.

How much does a RAG system cost to run?

Embedding is a one-off cost per document. Queries cost a small amount of model usage each. Storage is cheap. For most internal tools the dominant cost is the engineering time to maintain it, not the infrastructure.

Can I run retrieval-augmented generation entirely offline?

Yes. With a local embedding model and a local generation model served through Ollama, nothing leaves your machine. Quality trails hosted frontier models but is adequate for many internal use cases.

Is RAG going to be obsolete as context windows grow?

Unlikely. Larger windows reduce the need for retrieval on small corpora and do nothing for large ones, where cost and precision both favour retrieving the relevant few percent rather than sending everything.

Final Thoughts

The mental shift worth making is this: in a retrieval-augmented generation system, the model is the least important component. It is the last step in a chain, and it can only be as good as what the chain hands it.

Which means the work is not prompt engineering. It is parsing documents properly, chunking them sensibly, retrieving precisely, and curating what goes in. Do those four things well and a modest model produces excellent answers. Do them badly and the best model available produces confident nonsense with better grammar.

Start with twenty documents and fifty questions. Everything else is optimisation.