RAG Versus Fine-Tuning as an Architecture Decision
The usual framing of this comparison is “which one gives better results,” and it produces bad decisions because it treats two different mechanisms as competitors for the same job.
Retrieval supplies facts the model doesn’t have. Fine-tuning changes how the model behaves. Asking which is more accurate is like asking whether a filing cabinet or a training course improves an employee more. It depends entirely on whether the problem is that they don’t know something or that they do it wrong.
The useful framing is ownership: which one leaves you with a system you can correct quickly, cheaply, and without a specialist.
What each mechanism actually changes
Retrieval changes the input. The model is the same model; you’re handing it different material at inference time. Add a document and it’s available immediately. Delete one and it’s gone. The model’s behaviour — tone, format, reasoning style — is unchanged.
Fine-tuning changes the weights. You’re teaching the model a manner: a house style, a rigid output schema, a domain’s phrasing conventions, a classification boundary, a way of decomposing a problem. Facts learned this way are baked in, undated, and not individually addressable.
That last property is the crux. You cannot delete a fact from a fine-tuned model. You can only train a new one.
The decision table
| Criterion | Retrieval | Fine-tuning |
|---|---|---|
| Problem it solves | Missing knowledge | Wrong behaviour, format, or style |
| Update a single fact | Edit the document, reindex | Retrain |
| Time to correct an error | Minutes | Hours to days |
| Who can correct it | Anyone who can edit a doc | Someone who can run a training job |
| Attribution / citations | Natural — you have the sources | Not available |
| Per-request cost | Higher (longer prompts, retrieval hop) | Lower (short prompts) |
| Latency | Extra hop before generation | None added |
| Upfront cost | Pipeline engineering | Data curation + training runs |
| Portability to a new base model | Trivial — swap the model | Redo the training |
| Failure mode | Retrieves the wrong thing; visible in the trace | Confidently wrong with no trace |
| Knowledge cutoff | None — index is current | Frozen at training time |
Read the “portability” and “time to correct” rows together. Base models are replaced on a scale of months. A retrieval system survives that by changing one string in a config. A fine-tuned model has to be redone, on the new base, with the training data you’d better have kept.
When fine-tuning is clearly right
It’s a genuinely good tool, and it’s under-used for the things it’s actually for.
Rigid output format. If you need strictly-shaped output at high volume and prompting gets you to 95% compliance, fine-tuning is a reliable way to close the gap — and it lets you drop the long formatting instructions from every prompt, which pays back in tokens.
A domain’s manner of speaking. Clinical notes, legal drafting, a support voice with specific conventions about what never to say. This is style, and style is exactly what weight updates encode well.
Classification and extraction at scale. A tuned smaller model doing a narrow, well-specified task is frequently cheaper and faster than a large model with a long prompt, and the quality is often better on the specific task.
Cutting prompt length in a high-volume path. If you’re sending 2,000 tokens of instruction with every one of a million daily requests, training that behaviour in and dropping the instructions is real money.
Note that none of these are “the model needs to know our documents.”
When retrieval is clearly right
The knowledge changes. Anything with a version, an effective date, or a price. If the answer can go stale, it belongs in an index, not in weights.
You need citations. Regulated contexts, internal tools where trust is being built, anything where a user reasonably asks “where did that come from.” Retrieval gives you the source document as a byproduct. Fine-tuning cannot give you one at all, and a model asked to cite from parametric knowledge will produce plausible-looking references that may not exist.
The corpus is large. You can’t fine-tune your way into reliably recalling ten thousand documents, and attempting it produces a model that’s confidently approximate about all of them.
You need deletion. If a document must be removable — expired contract, withdrawn guidance, a GDPR request — retrieval makes that a DELETE and fine-tuning makes it a retraining project.
The combination
They compose, and the composition is usually better than either alone: fine-tune for behaviour, retrieve for facts. A model tuned to answer in your support team’s format, always citing, always refusing when the context doesn’t contain the answer — fed retrieved passages at inference.
But sequence matters, because most teams do this in the wrong order. Build retrieval first. Get the facts right, measure the system, and find out what’s actually broken. Then, if the remaining problems are behavioural — wrong tone, inconsistent format, doesn’t refuse when it should — consider tuning for those specifically.
Doing it the other way round means training on a moving target: you tune to compensate for retrieval problems, then fix the retrieval, and the tuning is now wrong.
The cost comparison people get backwards
Fine-tuning looks expensive up front and cheap to run. Retrieval looks cheap up front and expensive to run. Both are true per-request, and both mislead on the eighteen-month view.
Fine-tuning’s real cost is recurring and lumpy: data curation (the largest line item, and it’s human time), the training runs themselves, evaluation of each new checkpoint, and a full redo every time you change base models or the domain shifts. It is not a one-time cost. It is a cost that recurs unpredictably, requires a specific skill set, and tends to be deferred — which is how organisations end up serving a model tuned on last year’s product.
Retrieval’s real cost is recurring and steady: reindexing, corpus curation, keeping an eval set honest. Also substantial, also frequently unbudgeted, but it’s ordinary engineering that ordinary engineers can do, and it degrades gracefully. Both are broken down in what a RAG system actually costs to run.
The question to ask isn’t which is cheaper. It’s which recurring cost your organisation is actually able to keep paying.
The recommendation
Start with prompting. Genuinely. A well-constructed prompt with good examples solves a large share of what people reach for fine-tuning to solve, at zero infrastructure cost, and it’s the fastest way to find out what your real problem is.
If the problem is that the model doesn’t know things → retrieval. Nearly always. And if the corpus is small and stable, consider whether you even need the pipeline: RAG versus long context.
If the problem is that the model behaves wrong in a way prompting can’t fix, at volume where it matters → fine-tune, narrowly, for that behaviour.
If it’s both → retrieval first, then tune for the residual.
The tiebreaker, when it’s genuinely close: choose the one you can fix on a Tuesday afternoon without booking a specialist. Over eighteen months that property will matter more than any accuracy difference you can measure today.