Building Both: LoRA, Data, and the Retrieval Pipeline

Full fine-tuning updates every parameter in the model. It works, and for most organisations it is out of reach.

The memory arithmetic explains why. Training requires holding the weights, the gradients, and the optimizer state simultaneously. With the Adam optimizer that means roughly four times the parameter count in additional state beyond the weights themselves, so a seven billion parameter model in 16-bit precision needs well over 100 gigabytes of accelerator memory to train fully, before activations.

That puts full fine-tuning on multi-GPU infrastructure for even modest models, and the resulting artefact is a complete copy of the model, so ten fine-tuned variants means ten full model copies to store and serve.

There is also a quality risk: catastrophic forgetting, where training hard on a narrow task degrades general capability. A model fine-tuned aggressively on your ticket classifications may become worse at everything else, including the reasoning the task incidentally relied on.

Parameter-efficient methods solve all three problems at once, which is why they are what people mean now when they say fine-tuning. Full fine-tuning remains appropriate when you are substantially changing a model's capability rather than adapting it, which is a frontier-lab activity rather than an application one.

Low-Rank Adaptation, introduced by Hu and colleagues at Microsoft in 2021, is the technique that made fine-tuning practical for ordinary organisations.

The insight is that the weight update a fine-tune applies has low intrinsic rank. You are nudging a model's behaviour, not rebuilding it, and a nudge lives in a much smaller subspace than the full weight matrix.

So instead of learning a full update matrix for a layer, LoRA learns two small matrices whose product approximates it. For a layer of dimension d by d, a rank-r adaptation learns two matrices of size d by r and r by d. With d of 4096 and r of 8, that is roughly 65,000 parameters instead of 16.7 million, a reduction of more than two hundred times.

The base weights are frozen. Only the small matrices train, so gradients and optimizer state are needed for a tiny fraction of the parameters, which is what collapses the memory requirement to something a single accelerator handles.

Three consequences that matter operationally.

The artefact is small, often megabytes, so storing many task-specific adapters is cheap.

Adapters can be swapped at inference, and serving frameworks can host many against one set of base weights, which makes per-customer or per-task adaptation economically viable.

And because base weights are untouched, catastrophic forgetting is much reduced.

QLoRA extends this by quantizing the frozen base to 4-bit, reducing memory further and putting larger models within reach of a single consumer accelerator.

This is where fine-tuning projects actually fail, and it is a data problem rather than a machine learning one.

What you need is examples of the behaviour you want: input paired with the output a good response looks like. Not documents. Not your knowledge base. Demonstrations.

That distinction eliminates the most common plan. A team with ten thousand support articles does not have training data for a support assistant, because articles are not question-answer pairs demonstrating how to respond. They have a retrieval corpus.

Quantity: a few hundred examples can shift format and style, and a few thousand is the usual range for a task adaptation that holds up. More helps, with diminishing returns.

Quality dominates quantity, and by a wide margin. A thousand carefully constructed examples beat ten thousand scraped ones, because the model learns whatever consistency exists in the data, including consistent mistakes. A dataset containing contradictory examples of how to handle a case teaches the model to be inconsistent about that case.

Sources, in descending order of reliability. Curated historical examples of genuinely good work, filtered by an expert. Expert-written demonstrations, which is expensive and best. Synthetic generation from a stronger model, reviewed by a human, which is now common and works when the review is real. And scraped historical outputs, which is fastest and encodes every past bad habit.

Budget accordingly: the data is the project, and the training run is an afternoon.

What each actually requires to build, which makes the effort asymmetry visible.

The fine-tuning pipeline is front-loaded and episodic. Collect and curate examples, split into train and held-out evaluation sets, choose a base model, train the adapter, evaluate against the held-out set and against the prompted baseline, then deploy the adapter. It runs once, then again whenever the requirement or the base model changes.

The retrieval pipeline is continuous and always running. Ingest documents, chunk them, embed, index, and then at query time embed the query, search, rerank, assemble context, and generate. Every stage is live infrastructure with failure modes.

The asymmetry that matters: fine-tuning is a project that ends, retrieval is a system you operate. Teams comparing them on build effort alone systematically underestimate retrieval, because they count the build and not the operating.

And note the shared prerequisite at the bottom. Both need an evaluation set, and neither can be tuned without one.

The mechanics, at the level a practitioner needs to make decisions rather than to implement from scratch.

Choose the base model on licence, size and how well it already does the task. A smaller model that fine-tunes well often beats a larger one prompted, which is the entire economic case for the technique.

Set the LoRA rank. Values of 8 to 16 suit style and format adaptation; 32 to 64 for more substantial task changes. Higher rank means more capacity and more overfitting risk. Start low, because it usually suffices and trains faster.

Choose which modules to adapt. Attention projection matrices are the conventional target, and adapting more modules increases capacity and cost.

Set a low learning rate, since you are adjusting rather than teaching, and train for few epochs. Two to four is typical, and more usually overfits, which shows up as excellent training loss and worse held-out performance.

Hold out a genuine evaluation set from the start, never seen during training.

And the step most often skipped: compare against the prompted baseline on the same evaluation set. Without it you cannot say whether the fine-tune helped, and a meaningful share of fine-tunes do not beat a well-constructed prompt. That comparison is the whole justification for the work, and it is cheap.

The two compose cleanly because they act on different parts of the system, and the combination is what most mature deployments run.

The arrangement: retrieval supplies the facts, and the fine-tuned model supplies the behaviour. The model is trained not on your documents but on examples of how to use retrieved context well, which is a different and more useful thing.

What fine-tuning teaches in this configuration. How to use provided context faithfully rather than departing from it. When to say the context does not answer the question, which is the abstention behaviour the hallucinations cursus argues for and which is genuinely trainable. How to cite at the claim level. And the output format the application needs.

The training examples therefore look like: a question, a set of retrieved passages, and the ideal response given those passages, including the ideal response when the passages are inadequate.

That last category is the valuable one and the one teams omit. A dataset containing only examples where the context was sufficient teaches the model to always produce an answer, which is exactly the failure you were trying to avoid.

The practical sequence: build retrieval first, since it addresses the knowledge problem and its output shows you what the model must handle. Run it with prompting. Then, if faithfulness or format remains inconsistent at volume, fine-tune on the traffic you have now collected.

Both techniques invite the same evaluation failures, and the discipline is the one the AI QA cursus establishes.

Hold out data properly. For fine-tuning, the evaluation set must never have been seen in training, and near-duplicates count as seen. For retrieval, the queries used to tune chunking and parameters should not be the ones used to report quality.

Compare against the right baseline. Not against nothing, and not against a deliberately weak prompt. Against the best prompted version you could produce, because that is the alternative you are actually choosing against.

Measure what the application needs. Training loss is not a quality metric. For a fine-tune, measure the behaviour you were trying to change: format conformity, style consistency, task accuracy. For retrieval, measure retrieval and generation separately, since a failure in either looks the same at the output.

Run several times. Both systems are non-deterministic, so a single comparison is weak evidence.

And watch for the specific regression each invites. A fine-tune can improve the target behaviour and degrade general capability, so test capabilities you did not train on. Retrieval can improve grounding and increase latency and cost past what the application tolerates, so measure those alongside quality.

The honest report states what improved, what regressed, and what it cost.

Six claims.

They change different things. Fine-tuning changes behaviour, retrieval changes available knowledge, and diagnosing which failure you have is the whole decision.

Fine-tuning is a poor way to add facts. They land unreliably, cannot be updated, cannot be cited, carry no permissions, and training on an authoritative corpus can raise fluency without raising reliability.

Any of three questions settles it toward retrieval on its own: does the information change, must answers be attributable, does access differ by user. Those are architectural constraints rather than preferences.

Try prompting first. It solves more behaviour problems than teams expect, costs nothing to try, and produces the baseline without which you cannot tell whether a fine-tune helped.

LoRA made fine-tuning accessible by learning a low-rank update against frozen base weights, which collapses the memory requirement and produces small, swappable adapters. But the project is the training data, not the training run, and demonstrations are not documents.

And the mature configuration is both: retrieval supplying facts, a fine-tuned model supplying faithful use of them, including trained abstention when the retrieved context does not answer the question.

The most common expensive mistake remains the first one. Fine-tuning on a document corpus to teach a model facts, and discovering it learned the register instead.