The Options Nobody Compares, and Changing Your Mind

As context windows grew, a natural question arose: if the model can read a million tokens, why retrieve at all? Put the whole corpus in the prompt.

For a small corpus this is genuinely correct and underused. Under roughly a hundred thousand tokens of stable material, stuffing everything into context beats building a retrieval pipeline. No index, no chunking decisions, no embedding model to version, no staleness, and the model sees everything so nothing is missed by a retriever.

For a large corpus it fails on four counts.

Cost scales with every token on every request. Sending a million tokens per query is expensive in a way that does not amortise, and it recurs for every user.

Latency scales too, since the model processes what you send.

Attention degrades over long contexts. Material in the middle of a very long prompt is used less reliably than material at the ends, which is the lost-in-the-middle effect covered in the context windows lesson. So capacity is not the same as effective use.

And it does not scale past the window regardless. A corpus of ten million tokens does not fit, whatever the model.

The honest position: long context has raised the threshold below which retrieval is unnecessary. It has not removed the need for it, and teams should check which side of the threshold they are on rather than assuming.

Prompt caching is the option most often absent from these comparisons, and it can invert the cost case entirely.

The mechanism: where many requests share a long common prefix, the provider caches the processed representation of that prefix so subsequent requests skip recomputing it. Cached input tokens are billed at a substantial discount and processed faster.

Why it matters here. A moderate corpus in context, previously uneconomic because you paid full price for it on every request, becomes affordable if it sits in a cached prefix. The same applies to long few-shot example sets, which is what makes prompting competitive with fine-tuning at higher volumes than the naive arithmetic suggests.

The design consequence is that prompt structure becomes a cost decision. Put the stable material, the corpus, the instructions, the examples, at the front where it can be cached, and the variable material, the user's query, at the end. A prompt assembled in the opposite order defeats caching entirely for the same content.

Two caveats. Cache entries expire, so low-traffic applications may miss frequently. And any change to the cached prefix invalidates it, so a system that rebuilds its instructions per request gets no benefit.

The practical instruction: before concluding that context is too expensive or that fine-tuning is needed for cost reasons, check whether restructuring the prompt for caching changes the answer. It frequently does.

Laying out the full set makes clear how narrow the usual framing is.

Prompting alone: zero infrastructure, immediate, limited by context and consistency.

Long context with the corpus inline: no retrieval infrastructure, bounded by corpus size and per-request cost, improved substantially by caching.

Retrieval: unbounded corpus, updatable, citable, permission-aware, at the cost of a pipeline you operate forever.

Fine-tuning: changes behaviour and reduces per-request cost, at the cost of a data project and re-work on every base model change.

Continued pretraining: training on a large domain corpus to shift the model's underlying distribution rather than its task behaviour. Genuinely useful for a specialised domain with unusual vocabulary, and it requires a very large corpus and real expense, so it sits outside most application budgets.

Most decisions should walk this list in order and stop at the first that works. Teams instead jump to the fourth, because it is the one that sounds like engineering.

A newer pattern that changes the retrieval side of the comparison, and which the agentic RAG lesson covers in more depth.

Classical retrieval runs once: embed the query, fetch k documents, generate. That works when the question maps to a single lookup and fails when it does not.

Agentic retrieval lets the model drive. It can decide whether it needs to search at all, issue several searches, reformulate after seeing poor results, search again based on what it learned, and stop when it has enough. The retrieval becomes a loop rather than a step.

What this buys. Multi-hop questions, where the answer requires combining facts that no single document contains, become tractable. Ambiguous queries can be clarified by an initial search rather than guessed at. And the model can recognise that the corpus does not contain the answer, which supports the abstention behaviour the hallucinations cursus argues for.

What it costs. Several model calls and several searches per question, so latency and cost multiply. Non-determinism increases, since the search path varies. And it is harder to evaluate, because the trajectory matters and not only the final answer, which is why agent evaluation frameworks exist.

Where it fits the decision in this cursus: if retrieval quality is your problem and single-shot retrieval is the constraint, agentic retrieval is usually a better next step than fine-tuning, because it addresses the actual failure rather than changing behaviour around it.

A decision that was right when made can stop being right, and four events should trigger a revisit rather than being absorbed silently.

The base model changes. A new model may do prompted what the old one needed fine-tuning for, which makes the adapter unnecessary rather than merely stale. Every fine-tune should be re-justified against the current base model, not carried forward by inertia.

Volume changes. The cost crossover between prompting and fine-tuning is a function of traffic, so a system that grew tenfold may now favour a fine-tuned smaller model where prompting was previously right. The reverse also holds for a system that shrank.

The corpus changes character. A stable corpus that starts changing weekly invalidates any fine-tuning-for-knowledge decision, and a growing corpus can cross the threshold where inline context stops working.

And the requirement changes. A system that becomes subject to an explanation obligation, or that starts serving users with different permissions, acquires constraints that settle the question toward retrieval regardless of what was previously optimal.

The practical arrangement is a review trigger rather than a calendar reminder, in the same spirit as the governance cursus. Bind the revisit to the events above, particularly the base model change, since that one happens on the provider's schedule and nothing in your process fires when it does.

Changing approach is easier in one direction than the other, which is worth knowing before committing.

Adding retrieval to a fine-tuned system is straightforward. The adapter keeps doing what it does, and you supply context alongside. The one caution: a model fine-tuned to answer from its own parameters may resist deferring to provided context, since that is a behaviour it was not trained for. If it contradicts the retrieved documents, that is a training-data gap, and the fix is examples demonstrating faithful use of context.

Removing a fine-tune is also straightforward, since the base weights were never modified. Detach the adapter and re-establish the prompted baseline.

Replacing retrieval with fine-tuning is the hard direction, and usually the wrong one. You lose updatability, attribution and permission filtering, and you acquire a data project. Where cost is the motivation, check prompt caching and a smaller model first, because those address cost without surrendering the three properties.

And migrating a fine-tune to a new base model means redoing it. The training data transfers, which is the asset, but the adapter does not, and results differ enough that the evaluation must be rerun rather than assumed.

The general lesson: retrieval is the reversible choice and fine-tuning accumulates commitment. Where genuinely uncertain, that asymmetry is a reason to start with retrieval.

Because the decision expires, it should be recorded in a form that lets a successor re-evaluate it, which is the architecture decision record practice from the company brain cursus applied here.

What to write down.

The failure you observed, with examples. Not we needed better quality but the model produced correct facts in the wrong format in roughly a third of outputs, with these samples.

The diagnosis: knowledge or behaviour, and the evidence for it.

The options considered, including the ones rejected and why. Particularly whether prompting was tried, and how far it got, since that is the baseline everything else is measured against.

The decision, and the measured comparison that justified it. The numbers, on the same evaluation set.

The conditions that would change it: the base model version this was evaluated against, the traffic volume assumed, the corpus stability assumed.

And the cost accepted, in build effort and in ongoing operation.

That last section is what makes the record useful rather than archival, because it names the triggers from the previous step in a form someone can check. A record saying we chose fine-tuning because prompting could not hold format consistency at 500 requests per day, evaluated against base model version X is re-evaluable. One saying we fine-tuned for better quality is not, and by the time anyone asks, the people who decided will have moved on.

Across three lessons, the argument compresses to this.

Diagnose before choosing. Look at failing outputs and classify them as knowledge or behaviour, because the two techniques address different failures and choosing without diagnosing is how the expensive mistake happens.

The framing hides options. Prompting, long context, caching, agentic retrieval and continued pretraining all sit alongside the two being compared, and several are cheaper. Walk the list in order of cost and stop at the first that works.

Three questions settle it toward retrieval on their own: does the information change, must answers be attributable, does access differ by user. Those are constraints, not preferences.

Fine-tuning's real cost is the training data, not the training run, and demonstrations are not documents. That single confusion ends more projects than any technical limitation.

The mature configuration is both, with the fine-tune teaching faithful use of retrieved context including when to abstain.

And the decision expires. Base models change, volume changes, corpora change, requirements change. Record the reasoning and the conditions, and bind a revisit to the base model version, because that one moves without anything in your process noticing.

Retrieval is the reversible choice. Where you are genuinely uncertain, that is reason enough to start there.