# RAG vs fine-tuning

> Canonical: https://www.overmindlab.ai/research/rag-vs-fine-tuning

- Author: Sam Morris
- Category: Product
- Tags: Data
- Published: 2026-10-26
- Updated: 2026-11-02

> RAG retrieves documents and adds them to the prompt, so the model has the right facts at the moment it answers. Fine-tuning trains the model on examples, so the behavior you want becomes its default. Use RAG for knowledge that changes or needs a source. Fine-tune when the model has the facts but gets the format, tone or decision wrong on a narrow task that repeats.

![RAG vs fine-tuning](https://cdn.sanity.io/images/gv4t9qa3/production/b925224b92beee098e6fa25fb1abf4a9da659361-2048x2048.jpg)

Most teams building on a large language model hit one of two walls. Either the model does not know something it needs, such as your pricing, your policies or last week's release. Or it knows enough and still gets the format, the tone or the call wrong. Retrieval-augmented generation (RAG) fixes the first wall. [Fine-tuning](https://docs.overmindlab.ai/how-to/train) fixes the second.

The two work on different parts of the system. RAG changes what the model reads. Fine-tuning changes the model itself.

## RAG vs fine-tuning, side by side

|  | RAG | Fine-tuning |
| --- | --- | --- |
| What changes | What the model reads at answer time | The model's weights |
| What it fixes | Missing, private or outdated knowledge | Wrong format, tone or decisions on a repeated task |
| Where knowledge lives | In your documents, fetched per request | In the model, fixed at training time |
| Freshness | As current as your index | As current as the last training run |
| What you need | Documents, an index and a retriever | A few thousand good examples of the task |
| Cost per request | Longer prompts on every call | Shorter prompts, and the model can be smaller |
| Sources | Can show the passage it used | Cannot point to where an answer came from |
| Typical failure | The retriever misses the right passage | Confident answers outside what it was trained on |

*RAG vs fine-tuning, side by side*

## When to use RAG

RAG adds a lookup step before the model answers. A retriever finds passages related to the request, often with embeddings, and puts them in the prompt next to the question. The model answers from what it was handed and does not change. Use RAG in four situations.

- **The facts change.** Prices, policies and docs move faster than any training schedule. Update the document and the next answer uses it.
- **The knowledge is private or too large to fit.** The base model never saw your help center, and no prompt holds all of it. Retrieval picks the few passages that matter.
- **Answers need a source.** The model answers from specific passages, so you can show users where each answer came from.
- **Access differs by user.** Filter retrieval by who is asking. Anything trained into the weights can surface for any user.

RAG has limits too. Answers are only as good as the passages the retriever returns, and every passage adds tokens and latency. RAG does little for behavior. If the model has the right facts and still gets the format wrong, more retrieval will not help.

## When to fine-tune

Fine-tuning continues training a model on examples of the job, each pairing an input with the output you want. After enough of them, the model produces that output without being told. It pays off in four situations.

- **The task is narrow and repeats.** Classification, extraction, routing and fixed-format replies are the classic cases.
- **The model has the knowledge but not the behavior.** It knows enough to answer, yet keeps breaking the output format or picking the wrong tool.
- **You have real examples.** A few thousand labeled runs, ideally from production traffic. [How to turn traces into a training dataset](https://www.overmindlab.ai/research/traces-to-training-dataset) shows how to build them.
- **Cost or latency matters.** A small fine-tuned model with a short prompt can replace a frontier model carrying a long one. [Open-weight vs frontier models](https://www.overmindlab.ai/research/open-weights-vs-frontier-apis) covers when that switch pays.

Fine-tuning is a poor way to add facts. Models learn new facts slowly this way, and the facts go stale the moment they change. Keep knowledge in retrieval and behavior in training.

## Where prompt engineering fits

Prompt engineering changes the instructions. RAG changes the facts. Fine-tuning changes the model. The usual order follows the cost. Start with the prompt, because it takes minutes and you can undo it instantly. Add RAG when failures trace back to missing knowledge. Fine-tune when failures are about behavior on a narrow task and better prompts have stopped moving your eval.

Read the failures to tell the gaps apart. Quoting an old price is a knowledge gap. Formatting the right price wrongly is a behavior gap. [Prompt engineering vs fine-tuning](https://www.overmindlab.ai/research/prompt-engineering-vs-fine-tuning) covers how to tell when prompting has hit its ceiling.

## Can you combine RAG and fine-tuning?

Yes, and agents that need both should use both. The two fix different gaps, so they stack. A support agent might retrieve the customer's plan and the right help article, then hand both to a model trained on thousands of resolved tickets.

Train on what the model will see in production. If your agent answers from retrieved passages, keep those passages in the input of each training example. The model then learns to answer from context the way your agent's best runs did. This is the idea behind [RAFT](https://arxiv.org/abs/2403.10131), short for retrieval-augmented fine-tuning.

Keep the split clean. Facts stay in retrieval, so a changed policy needs a document edit, not a training run. Behavior lives in the weights, so the prompt can shrink to the request and its passages.

## RAG, fine-tuning or both? Worked examples

| Problem | Start with | Why |
| --- | --- | --- |
| The agent quotes last quarter's prices | RAG | Prices change faster than you could retrain |
| Each customer should see only their own records | RAG | Filter retrieval per user. Anything in the weights can surface for anyone |
| A routing or extraction step runs on every request | Fine-tuning | Narrow task, high volume, and production gives you the examples |
| The right document is in the prompt, but answers misquote it or add details | Fine-tuning, on examples that include the retrieved text | The gap is in how the model reads, not what it reads |
| Replies are too long or the wrong tone | Prompt engineering | One instruction fixes it, and you can undo it instantly |

*Which technique fits which problem*

## What training adds when the document is already in the prompt

RAG decides what the model reads. Training changes how faithfully it reads. Overmind's benchmark, [When bigger isn't better](https://www.overmindlab.ai/research/when-bigger-isnt-better), measured that second part on three reading tasks. Each gives the model a document to answer from, the same step every RAG pipeline ends with.

The clearest case is contract review. Across 4,000+ questions on 100+ contracts, the model had to say whether a clause was in the contract and quote its exact wording. The benchmark scored the base model before and after training on real examples of the task, alongside a frontier flagship model.

|  | Base model, before training | Same model, trained with Overmind | Frontier flagship |
| --- | --- | --- | --- |
| Accuracy (F1) | 80.4% | 91.6% | 82.0% |
| Exact quote match | 4.2% | 59.6% | 8.5% |
| False alarm rate (clause claimed but absent) | 2.46% | 0.65% | 10.01% |
| Fabrication rate (quote that does not exist) | 3.36% | 0.12% | 3.36% |

*Contract clause detection, 4,000+ questions across 100+ contracts (from When bigger isn't better)*

Training took exact quotes from 4.2% to 59.6% and fabricated quotes from 3.36% to 0.12%. Better retrieval could not have done that, because the contract was already in the prompt.

The other two tests compared trained models with the frontier flagship, and the results pointed the same way. On biomedical papers (BioRED), the frontier flagship claimed relationships that were not there more than 7x as often as the trained model (a 48.2% hallucination rate against 6.6%). On NASA aviation incident reports, it added a detail that was not in the report to more than 1 in 3 synopses (33.4%), against 13.4% for the smallest model Overmind trained.

Because these tests hand the model the document, they measure the reading step only. They say nothing about retrieval quality.

## Fine-tuning a RAG agent with Overmind

Keep your retriever. Overmind trains the model that reads what it returns. The Overmind Python SDK can record each retrieval as its own span, so a trace shows what was fetched next to what the model answered. You [build a dataset from those production traces](https://www.overmindlab.ai/research/traces-to-training-dataset), and every row links back to the trace it came from. Then you [fine-tune an open-weight base model](https://www.overmindlab.ai/product/model-training) with [LoRA or full supervised fine-tuning](https://www.overmindlab.ai/research/types-of-fine-tuning).

Before you switch traffic, benchmark the trained model with your evaluators on a separate eval set. By default Overmind compares the base model before training with the trained model after. Comparing against the model your agent runs today is opt-in. The trained model is served through an OpenAI-compatible endpoint, and the weights are downloadable. [The training docs](https://docs.overmindlab.ai/how-to/train) cover each step.

**Have an agent in production?** Its own past work is the training data, and your coding agent can set Overmind up for you. Open the repo in Claude Code, Cursor, Codex or OpenCode and paste: `Set me up on Overmind: https://console.overmindlab.ai/setup.md`

## FAQ

**What is the difference between RAG and fine-tuning?**

RAG retrieves relevant documents when a request arrives and adds them to the prompt, so the model answers from current facts. Fine-tuning trains the model on examples, which changes its weights. RAG changes what the model knows in the moment. Fine-tuning changes how it behaves every time.

**RAG vs fine-tuning vs prompt engineering: which should you try first?**

Prompt engineering, because it takes minutes and you can undo it instantly. Add RAG when failures come from missing or outdated knowledge. Fine-tune when failures are about behavior on a narrow, repeated task and prompt changes have stopped improving your eval.

**Is RAG cheaper than fine-tuning?**

RAG is cheaper to start, because there is no training run. Fine-tuning can be cheaper to run, mostly because a fine-tuned model can be a smaller one. In Overmind's contract benchmark, answering 1,000 questions cost $1.03 with the trained model and $20.94 with the frontier flagship.

**Does fine-tuning add knowledge to a model?**

Only slowly and unreliably. [A 2024 EMNLP paper](https://arxiv.org/abs/2405.05904) found that models learn new facts from fine-tuning more slowly than facts they already know, and hallucinate more once they do learn them. Its authors conclude that models get most of their knowledge from pre-training, while fine-tuning mainly teaches them to use it.

**What is RAFT?**

RAFT, short for retrieval-augmented fine-tuning, is a training recipe from [a 2024 UC Berkeley paper](https://arxiv.org/abs/2403.10131). Each training example pairs a question with retrieved documents, some relevant and some distractors, and an answer that quotes the relevant passage. The model learns to find the right passage and ignore the rest.

**Does RAG stop hallucinations?**

No. The model can still misread the retrieved text or add details that are not in it. In Overmind's aviation benchmark, a frontier flagship given the incident report still added a detail the report did not contain to 33.4% of its synopses. Retrieval addresses missing knowledge. Training on real examples of the task improves how faithfully the model uses it.

## Related research

- [My summer at Overmind](https://www.overmindlab.ai/research/my-summer-at-overmind): A summer internship reflection on autonomy, hard problems, and building across the Overmind product.
- [Small is the new frontier](https://www.overmindlab.ai/research/small-is-the-new-frontier): Why specialized models trained on your own production data are becoming the practical path to owned intelligence.
- [When bigger isn’t better](https://www.overmindlab.ai/research/when-bigger-isnt-better): On specialized tasks, Overmind-trained small language models outperform frontier models on accuracy, hallucination, and cost.