Use RAG when the model needs to know things: your documents, policies and data, kept current and cited. Use fine-tuning when the model needs to behave differently: a fixed format, a house style, a narrow task done consistently. Most business projects need RAG. Few need fine-tuning, and those that do usually need RAG as well.
RAG vs fine-tuning: the difference
Retrieval-augmented generation (RAG) leaves the model as it is. When a question arrives, the system finds the relevant passages in your own content and hands them to the model with the question. The model answers from what it was just shown.
Fine-tuning changes the model. You train it further on examples until the behaviour you want is built in.
| RAG | Fine-tuning | |
|---|---|---|
| Changes | What the model can see | How the model behaves |
| Good for | Facts, documents, current information | Tone, format, a narrow repeated task |
| Updating knowledge | Update the documents | Retrain |
| Can show sources | Yes | No |
| Respects access rights | Yes, at retrieval | No, knowledge is baked in |
| Needs | Good content and good search | Hundreds or thousands of quality examples |
Why most projects should start with RAG
The usual request sounds like this: we want an assistant that knows our policies. That is a knowledge problem, and retrieval solves knowledge problems.
Three properties make RAG the right default for a business. Answers come with citations, so a person can check them in seconds. Knowledge is current the moment a document changes, with no retraining. And permissions carry over: someone who can't open a document doesn't get answers drawn from it.
That last one is decisive in regulated work. Train a model on confidential contracts and every user of the model can, in principle, get at what it learned. With retrieval, the access check happens on every question.
This is how we build private assistants: your content stays in your environment, and the model is called through your own account.
Where RAG goes wrong
RAG fails at search far more often than at generation. If the system retrieves the wrong passage, the model writes a fluent, confident answer from the wrong passage. People then blame the model.
So the unglamorous work decides the result: how documents are split, how tables and scans are handled, whether old versions are removed, how results are ranked. A messy document store produces a messy assistant.
Measure it. Collect real questions with answers your experts agree on, and test retrieval and answers against them before launch and after every change. Without that set you're judging quality by anecdote.
When fine-tuning is worth it
Fine-tuning earns its cost in a narrower set of cases than its reputation suggests.
- A strict output format the model keeps drifting from, even with careful prompts and examples.
- A specialised, repetitive task at high volume, such as classifying your tickets into your categories.
- A smaller, cheaper model that needs to match a larger one on that one task.
- A consistent voice across a very large amount of generated text.
Notice what's missing from that list: teaching the model your facts. Fine-tuning is poor at that. It blurs facts together, it can't cite them, and they go stale the day after training.
It also needs examples, and good ones. Assembling a few thousand clean input and output pairs is usually the real cost of the project, well above the training bill.
Try the cheaper things first
Before either, check whether you need them at all.
- A clear prompt with a few examples. This fixes more than people expect.
- A structured output setting, if format is the problem.
- RAG, if the model lacks knowledge.
- Fine-tuning, if behaviour is still wrong after all of that.
Models also keep improving. A weakness you fine-tune around this year may be gone in the next release, and then you own a custom model that's worse than the standard one. That risk is a good reason to keep your investment in your content and your test set, which carry over to any model.
Using both
They aren't rivals. A support assistant might retrieve the right policy with RAG and use a fine-tuned model to answer in the company's format every time. Start with retrieval, add tuning only where measurement shows a gap.
If you're not sure which problem you have, that's what our AI consulting work is for: we test candidates on your own data before you commit to a build. For the model choice itself, see how we use OpenAI and Claude.
