RAG or Fine-Tuning: Which Does Your Team Actually Need?

RAG or fine-tuning: which does your team actually need?

If the model needs facts it does not have, use retrieval. If the model has the facts but gets the shape or style wrong, consider fine-tuning. That single split resolves most of these arguments. Teams get stuck because they treat the two as competing products rather than as fixes for different faults.

We see this question land in web and marketing teams more than it used to, usually attached to a support bot, an internal search tool, or a content system. The stakes are real: one path is a weekend of plumbing, the other is a data programme.

Here is the comparison as we run it, using what the model providers themselves publish.

What is RAG, in plain terms?

Retrieval augmented generation means fetching relevant text at question time and putting it into the prompt, so the model answers from documents rather than memory. Nothing about the model changes. You are changing what it can see.

The mechanics are ordinary. You split documents into chunks, turn each chunk into an embedding, store those, and at question time find the chunks closest to the question. You then hand those chunks to the model with the question attached.

The appeal is that your knowledge stays in your own store. Update a document and the next answer reflects it. Delete a document and the model stops quoting it. For anything that changes weekly, this is the only sane option.

What is fine-tuning actually for?

OpenAI's own model optimization guide describes fine-tuning as a way to make a model "consistently format responses in a certain way or handle novel inputs." It also notes you can give more example inputs and outputs than would fit in one request's context window, train on proprietary data without resending it each time, and use smaller, cheaper models that do one job well.

The same documentation lists three methods. Supervised fine-tuning uses examples of correct responses. Direct preference optimization supplies both correct and incorrect examples so the model learns a preference. Reinforcement fine-tuning has the model generate responses that an expert grader scores, and is available for reasoning models only.

Notice what is absent from that list: teaching the model new facts reliably. Fine-tuning shapes behaviour. It is a poor and expensive way to install knowledge that will change next quarter.

How do the two compare side by side?

The cleanest way to see it is against the questions a team actually asks when scoping the work. Cost, freshness, and what happens when the underlying information changes.

That last row is the one worth sitting with. A retrieval failure is visible and debuggable, because you can look at what was retrieved. A fine-tuning failure is baked into weights and much harder to trace.

Why does retrieval quality decide whether RAG works?

Because a model cannot answer from a chunk it never received. Most disappointing RAG systems are not model problems at all. They are search problems wearing a model costume.

Anthropic published useful numbers on this in its contextual retrieval research on 19 September 2024, measured across codebases, fiction, ArXiv papers and science papers using one minus recall at 20 as the metric. Contextual embeddings alone cut the top-20 chunk retrieval failure rate by 35 percent, from 5.7 percent to 3.7 percent. Combining contextual embeddings with contextual BM25 cut it by 49 percent, to 2.9 percent. Adding reranking on top cut it by 67 percent, to 1.9 percent.

Read those as a work order, not a benchmark. The gains came from better chunking context, a keyword method running alongside the vector method, and a reranking pass. None of that required a different model. If your retrieval is weak, upgrading the model will not save you.

When is fine-tuning the right call?

When you have a narrow, repetitive task with a clear right answer shape, and you have hundreds of real examples of that shape. Classification, structured extraction into a fixed schema, and rewriting into a house style are the classic cases.

It is also the right call when cost or latency is the binding constraint. OpenAI's guidance points at this directly: fine-tuning lets you use smaller, cheaper models that excel at specific tasks. If a job runs ten thousand times a day, moving it from a frontier model to a tuned small one is real money. Our piece on small models versus frontier models covers that trade in more depth.

The same documentation also offers the honest caveat that prompt engineering "may be all you need in order to get great results for your use case." We would put that stronger. Try the prompt first, properly, with real examples, before you commission a dataset.

What does each actually cost to maintain?

Retrieval costs you storage, an indexing job, and longer prompts on every call. The prompt length is the cost people forget. If every answer carries four chunks of context, every answer costs more than a bare question would.

Fine-tuning costs you the training run, which is usually the smallest part, plus the dataset. The dataset is the real bill. Someone has to produce, label and maintain examples, and when the task changes, someone has to do it again. That is a standing commitment, not a project.

There is a third cost on both sides: evaluation. Neither approach is safe to ship without a way to tell whether a change made it better or worse. We wrote about evaluating AI automations in production because this is where most of these projects quietly fail.

Can you sensibly use both?

Yes, and the combination is often the right destination rather than the starting point. A fine-tuned model that reliably produces your output format, fed by a retrieval layer that supplies current facts, is a strong architecture.

The order matters though. Build retrieval first, get it working, and only then look at whether a tuned model would make the output cheaper or more consistent. Doing it the other way round means tuning against behaviour you are about to change.

One practical warning. If you fine-tune on outputs that included retrieved context, and then run the model without that context, it will confidently invent the missing parts. Keep the training conditions and the runtime conditions the same.

What do most web and marketing teams actually need?

In our experience, retrieval plus a well-written prompt covers the large majority of what these teams want to build. Answering questions from your own documentation, drafting from your own brand guidelines, summarising your own tickets: all of those are knowledge problems.

The cases that genuinely earn fine-tuning tend to be high-volume and narrow. Routing inbound messages into a fixed set of categories. Pulling specific fields out of documents into a schema, every time, at scale. Those have the repetition and the clear right answer that tuning rewards.

If you are building an internal answer tool, start with the knowledge base itself. A retrieval system over a messy, contradictory document store will be confidently wrong, and no amount of model work fixes that. Our notes on building an AI internal knowledge base start from the same point.

Where should you start this week?

Write down ten real questions your system must answer, with the correct answer beside each one. Then run them through a plain prompt with the relevant document pasted in. If the answers are good, your problem is retrieval and you know exactly what to build. If they are bad even with the right document in hand, no retrieval system will rescue you.

That test costs an afternoon and settles an argument that otherwise runs for a quarter. It also gives you the beginning of an evaluation set, which you will need either way.

If you are weighing this up for a real system and want a second opinion before you commit, we are happy to walk through it. We are at phoenix.studio.