What Each Technique Actually Changes

The comparison is usually framed as a choice between competing techniques. It is more useful to see that they operate on different parts of the system and mostly do not compete at all.

Fine-tuning continues training a model on your examples, adjusting its weights. What changes is the model's disposition: how it formats output, what tone it adopts, what conventions it follows, which of several plausible responses it prefers. The knowledge is distributed across parameters and cannot be inspected, updated or attributed.

Retrieval leaves the model untouched and changes its input. At query time you find relevant material and place it in the context window, so the model answers from text it can see. Nothing about the model changes, and the same base model serves any corpus you point it at.

The distinction that resolves most of the debate: fine-tuning changes how the model behaves; retrieval changes what it knows about right now.

So the diagnostic question is not which is better but what is actually wrong with the current output. If the model produces well-formed answers containing facts it does not have, that is a knowledge problem and retrieval addresses it. If it has the facts and presents them wrongly, that is a behaviour problem and retrieval will not help however good the retrieval is.

Getting that diagnosis wrong is the most expensive mistake in this area, and it is common.

The most common misconception is that fine-tuning on your documents teaches the model your documents. It does something, and not that, for four reasons.

Facts land unreliably. Training adjusts weights to reduce loss across examples, which is not the same as storing retrievable propositions. A fact appearing once in the training data may be absorbed weakly or not at all, and the model has no way to signal which facts it actually acquired.

The result cannot be updated. Correcting a fact means retraining, and there is no operation that removes a specific fact from weights. For any corpus that changes, this alone is disqualifying.

Attribution disappears. A fine-tuned model produces an answer with no source, so you cannot cite, verify, or satisfy the grounding requirements the hallucinations cursus describes.

And it can increase fabrication. Training on a corpus in a confident, authoritative register teaches the register as much as the content, so the model becomes more fluent about the domain without becoming more reliable, which is the worst combination for a reader.

There is also a permission consequence, from the company brain cursus. Facts baked into weights carry no access control, so per-user permissions become effectively impossible.

Retrieval has none of these properties: facts are updatable, citable, permission-filtered, and verifiably present or absent.

Having established what it is bad at, the cases where it is the right tool are specific and worth knowing, because retrieval cannot address them at all.

Output format and structure. Where you need consistent conformity to a schema, a house style, or a document template, a few hundred examples teach it far more reliably than instructions, and the instructions consume context on every request.

Domain register and vocabulary. Legal drafting, clinical notes, financial commentary each have conventions that are laborious to specify and easy to demonstrate.

Task specialisation. Classification, extraction and routing where a smaller fine-tuned model matches or beats a larger prompted one, at a fraction of the cost and latency. This is where fine-tuning frequently pays for itself commercially rather than qualitatively.

Implicit preferences that resist articulation. Which of several correct answers your organisation prefers, how much hedging is appropriate, when to escalate. Examples convey this; prompts struggle.

And behaviour under ambiguity, where the model must adopt a consistent default rather than asking.

The common thread: everything in this list is about form, style, structure or preference, not about facts. If you can state the requirement as here is what a good answer looks like across many examples, fine-tuning fits. If you can state it as the model needs to know X, it does not.

The sequence that resolves most cases, and it starts before either technique.

Diagnose first. Look at actual failing outputs and classify them: missing or wrong facts, or wrong form and behaviour. Teams that skip this choose a technique before knowing the problem.

Then try prompting. Better instructions, few-shot examples in context, and structured output constraints solve a surprising share of behaviour problems at zero training cost and with no new infrastructure. This is the step most often skipped because it feels insufficiently substantial.

For knowledge problems, retrieval. For behaviour problems that prompting could not fix, fine-tuning. For cost or latency problems on a narrow task, fine-tuning a smaller model.

And where the failures are of both kinds, which is common, both. They compose cleanly because they act on different parts of the system: a fine-tuned model that formats correctly, answering from retrieved documents it can cite.

The ordering matters because each step is cheaper than the next, and because doing the expensive thing first tends to obscure whether it was needed.

The step teams skip deserves its own treatment, because skipping it is how organisations end up with a training pipeline they did not need.

What prompting can do that people underestimate. Few-shot examples in context teach format and style effectively, often approaching fine-tuned quality for moderate consistency requirements. Structured output constraints, where the model is required to emit valid JSON against a schema, solve format conformity more reliably than training does, because it is enforced rather than learned. And explicit instructions handle a great deal of behavioural preference if written specifically rather than aspirationally.

What it cannot do. Consistency across very many examples at scale, since context is finite and few-shot examples consume it on every call. Deep register acquisition. And cost reduction, since a long prompt is more expensive per call, not less.

The economics are the interesting part. Few-shot prompting costs tokens on every request forever; fine-tuning costs a training run once and then produces shorter prompts. So at low volume prompting is cheaper, and at high volume fine-tuning is, with the crossover depending on your traffic and the length of the examples you would otherwise carry.

The practical instruction: get prompting to its ceiling and measure. That measurement is both the baseline that tells you whether fine-tuning helped, and frequently the answer.

The two options have costs that fall at different times and scale differently, which changes the decision more than most comparisons acknowledge.

Fine-tuning has a large one-off cost and low marginal cost. Assembling and cleaning a training set is the dominant expense and it is human labour, not compute. Training itself is comparatively cheap with parameter-efficient methods. Then inference is cheap, often cheaper than the base model if you fine-tuned a smaller one, and the prompt is shorter. But every corpus change means retraining, and every base model upgrade means redoing it.

Retrieval has a moderate setup cost and a permanent marginal cost. You build ingestion, embedding, an index and the retrieval layer, which the vector database cursus covers. Then every query pays for embedding, search, and substantially more input tokens, because retrieved context is large. That cost scales linearly with traffic and never goes away.

So the shapes invert. At low volume with a stable corpus, fine-tuning's one-off cost may never be repaid. At high volume, retrieval's per-query token cost accumulates into the dominant line item, and a fine-tuned smaller model can be dramatically cheaper.

The cost most consistently underestimated is the training data. Producing a few thousand high-quality examples is weeks of expert time, and it is the reason most fine-tuning projects stall.

Four realistic cases, showing how differently the same framework resolves.

CASE                          PROBLEM TYPE      ANSWER

Support bot answering from    knowledge:        RAG. Docs change weekly;
product docs                  facts change      fine-tuning cannot update.

Drafting contract clauses in  behaviour:        Prompt first. If consistency
the firm's house style        form and register  fails at scale, fine-tune.

Classifying 2M support        cost and latency  Fine-tune a small model.
tickets per month             on a narrow task   Prompting a large one costs
                                                 far more at that volume.

Internal assistant over       knowledge +       RAG for the facts and the
confidential HR documents     permissions        permission filter. Fine-tuning
                                                 would bake data into weights
                                                 with no access control.

The HR case is the one worth dwelling on. It looks like a natural fine-tuning candidate, since the corpus is proprietary and stable-ish. It is disqualified not by quality but by permissions: weights carry no access control, so a fine-tuned model would answer any employee's question from any document it absorbed. That is the company brain cursus's point arriving as a hard architectural constraint.

And the classification case is the clearest fine-tuning win, for reasons that are entirely commercial rather than about capability.

Six questions that between them resolve almost every real case.

Does the information change? If yes, retrieval, and the frequency determines how strongly. A weekly-changing corpus makes fine-tuning untenable regardless of anything else.

Must answers be attributable to a source? If yes, retrieval, since fine-tuned knowledge cannot be cited.

Does access differ by user? If yes, retrieval, because weights have no permission model.

Is the failure about form rather than facts? If yes, prompting first, then fine-tuning.

Is the volume high enough that per-query cost dominates? If yes, a fine-tuned smaller model may pay for itself, and this is a calculation rather than a judgement.

Do you have, or can you produce, a few thousand high-quality examples? If not, fine-tuning is not available to you whatever the other answers say, and this constraint ends more projects than any technical consideration.

One caution on how these interact. Answering yes to any of the first three settles it toward retrieval regardless of the rest, because those are architectural constraints rather than preferences. The remaining three are trade-offs you can reason about.

And the honest default for most organisations: retrieval addresses the problem you actually have, and fine-tuning is the answer to a narrower question than it is usually asked.