“An AI trained on our data” is usually a request for an assistant that answers from your prices, your policies, and your product catalogue rather than from the public internet. For that job the answer is almost always retrieval-augmented generation (RAG) rather than fine-tuning, because fine-tuning changes how a model behaves, and retrieval changes what the model has in front of it when it answers.
The mix-up is a reasonable one. “Training” is the word the whole industry uses for how models learn, so it is the word most people reach for when they want a model to learn about them. The two techniques solve different problems, though, and choosing the wrong one costs months and produces a system that is harder to keep accurate than the one you started with.
What each one changes
Retrieval searches your own content for the passages relevant to a question at the moment the question arrives, and hands them to the model along with it. The model is left as it was, and the material it answers from is what changes. How retrieval works, step by step has its own post.
Fine-tuning continues a model’s training on a set of examples you supply, so that its default behaviour shifts toward those examples. It alters the model itself, and nothing gets looked up when a question comes in.
That leaves one question to settle most of these decisions: is your problem what the model knows, or how it behaves?
RAG vs fine-tuning, side by side
| RAG (retrieval) | Fine-tuning | |
|---|---|---|
| What it changes | What the model has in front of it when it answers | How the model behaves by default |
| Best for | Facts: policies, prices, products, documents | Behaviour: a format, a voice, a narrow repeated task |
| When your information changes | Update the document, and the next answer uses it | Train again, and the old version answers until you do |
| Can it show where an answer came from | Yes, the passage it used | No, the knowledge is blended into the model |
| Can you limit who sees what | Yes, by searching only what that user may see | No, anything trained in is available to everyone using the model |
| What you need to start | Content you already have | Dozens to hundreds of worked examples of the ideal output |
| Tied to one model | No, it works with whichever model you use | Yes, and it is retired when that model is |
| Offered by your provider | Always | Not always |
Most of those rows follow from a single difference. With retrieval, your information stays in documents you control, so updating it, citing it, and restricting it work the way they already do. With fine-tuning the information is folded into the model, and each of those jobs now means another training run.
Why fine-tuning is a poor way to teach a model your facts
A model picks up patterns in training, and a fact you need quoted back exactly, such as a price, a returns window, or a warranty term, behaves badly when it is stored as a pattern. The research that has tested the two side by side keeps landing in the same place.
A Microsoft team found that retrieval consistently outperformed fine-tuning on the raw documents at getting knowledge into a model, both for facts the model had partly seen before and for entirely new ones. A Google team then looked at what happens when a model is fine-tuned on facts it did not already know, and found that it learns them slowly and grows more prone to inventing answers as it does. Their conclusion is the cleanest summary of the whole topic: models mostly acquire their factual knowledge in pre-training, and fine-tuning “teaches them to use it more efficiently.”
That second finding is the one with commercial weight. If the complaint is that an assistant makes things up about your business, training it on your business can make it make up more. The gap is also widest where a business sits, since a study of less common knowledge found retrieval ahead of fine-tuning by a large margin on the least popular facts, and to a general-purpose model your pricing is about as obscure as a fact gets.
In fairness, fine-tuning can be pushed into learning facts. Rewording each fact many times helps, and an agricultural study that trained a model on question-and-answer pairs generated from its source documents brought it level with retrieval. That is a great deal of preparation to reach parity, and the result goes stale the first time one of those facts changes.
The rest of the case is practical. A fact trained into a model cannot point to the document it came from, so nobody can check it against the source. It cannot be withheld from a user who should not see it, because whatever is inside the model is available to everyone the model talks to. And correcting it means another training run rather than an edit.
What fine-tuning is actually good at
Fine-tuning earns its place when the problem is behaviour that clear instructions and a few examples still cannot hold at volume. The model vendors draw the same line: OpenAI’s own guidance on accuracy treats missing, outdated, or proprietary knowledge as a job for context, and inconsistent formatting, the wrong tone, or reasoning the model does not follow consistently as a job for fine-tuning.
In practice that covers four kinds of problem:
- An output format that can never drift. Thousands of responses a day in an exact structure that another system reads.
- A house voice. A style that written instructions only approximate and dozens to hundreds of good examples pin down.
- A narrow judgement made all day. Routing, tagging, or classifying the same kind of item, where consistency matters more than range.
- A smaller model doing one job. Training a cheaper model to match a larger one on a single task is a real running-cost lever once volume is high, because the smaller model costs less per call and needs shorter instructions.
We have not needed to fine-tune a model on the AI work we have shipped, so that list comes from the providers’ guidance and the research rather than from a project of ours. Where we do have first-hand experience is the other side of the table.
Two jobs we solved without training anything
The streaming code generator we built ran into both of the problems people usually expect to train their way out of.
The first was facts. A general-purpose model knows a great deal about code and also confidently produces syntax that does not exist, which for a tool whose users paste the output into their own projects is worse than no answer at all. The fix was retrieval over versioned documentation, so that answers tracked a specific version of a reference rather than the model’s general impression of it. The documentation updates on its own schedule, and the model never has to be told.
The second was format. The interface needed structured output it could render reliably, where free text would have had to be picked apart on arrival, and that is the textbook case for fine-tuning a format. The provider’s structured-output feature handled it instead, and it removed an entire class of parsing bugs in the process. The same product serves different models to different subscription tiers, and a fine-tune would have tied each of those choices to a single model for as long as the fine-tune lived.
The order to try them in
When an AI feature’s answers fall short, these are the steps we take, cheapest first:
- Improve the instructions and add examples. Often enough on its own for format and tone.
- Add retrieval when the problem is facts the model does not have.
- Put code around the model wherever a rule has to hold every time: validate the output, enforce the structure, and retry or reject what fails.
- Test other models against the same scenarios. The model that suits a job is often not the one you started with, and trying another is cheap when the system was built to allow it.
- Fine-tune only when the earlier steps have had a fair run and a behaviour gap remains, and keep retrieval underneath it for the facts.
Step one does more than it looks like it should. On a client feature that drafts short written summaries, the prompt said “At most 1400 characters total. Count them. Going over is a failure.” and in testing, at least 48 of 98 runs came back over the limit. We replaced that one line with a request for three paragraphs of 60 to 75 words each, kept the same model and the same ten briefs, and the overshoot fell to 13 of 74 runs.
The typical miss it recorded had been about 45 characters, roughly seven words, so the model was trying to follow the rule and had no reliable way to count characters; words sit much closer to the units a model actually works in. That is a format that would not hold, the first case on the fine-tuning list above, and one changed line fixed most of it. The test measured whether the length rule held rather than whether the summaries got better, which is the next question an eval would ask.
You can try a change like that yourself, in the provider’s own tools, before paying anyone. What a single try cannot tell you is whether it worked, because one good-looking answer is a sample of one, and the difference between 48 of 98 and 13 of 74 only shows up when the same inputs run many times before and after the change.
Every step gets judged the same way, with an eval that runs a set of real questions and scores the answers, so that “it still isn’t good enough” turns into a specific failure you can point at. This is the sequence behind the AI integration work we ship.
Retrieval and fine-tuning can also be combined, and the evidence on whether that helps is mixed. In the agricultural study the gains stacked, while in the Microsoft one, on its current-events questions, adding fine-tuning to retrieval did worse than retrieval alone. That disagreement is the argument for measuring on your own questions rather than taking anyone’s word for it, ours included.
What fine-tuning commits you to
If the eval does point to fine-tuning, it helps to know what you are signing up for.
- One base model. A fine-tune is built on a specific model version, and when the provider retires that version the fine-tuned copy goes with it. Moving to a better or cheaper model means training again, which is vendor lock-in in its most literal form.
- A provider that keeps offering it. Not every major provider offers fine-tuning, and some that did are winding it down. Deciding you need a fine-tune can quietly become deciding which models you are allowed to use.
- A training set you now maintain. The worked examples are an asset with an owner, and they drift out of date like any other content as the task changes.
- A higher running rate. At one major provider, calls to a fine-tuned model typically bill at one and a half to two times the base model’s rate, on top of the cost of training it.
Is it what the model knows, or how it behaves?
When someone proposes to train a model on your data, the most useful thing you can ask is which of those two problems they think you have. If the answer is what the model knows, the proposal is for the slower and costlier tool, and the one that will be out of date first. If the answer is how the model behaves, fine-tuning may well earn its place, once the cheaper steps have had their turn and an eval has shown exactly what they could not fix.