The Case for Self-Hosting, Examined Honestly

The usual case for self-hosting is that data never leaves your infrastructure. That is true and it is worth being precise about what it buys, because the precision changes the decision.

What sending data to an API provider actually involves, under a standard enterprise agreement: the data crosses a network boundary to a third party, is processed on their hardware, and is subject to their retention, their access controls, their subprocessors and their jurisdiction. Major providers contractually commit not to train on enterprise API traffic and offer zero-retention options, and those commitments are real and enforceable.

So the honest comparison is not private versus not private. It is your controls versus their controls plus a contract.

That framing matters because a self-hosted deployment is not automatically more private. A model running on a cloud instance you administer badly, with broad access, unencrypted volumes and logs nobody reviews, may expose more than a well-configured API integration under a zero-retention agreement.

Self-hosting changes who is responsible for the controls, not whether controls are needed. Organisations that adopt it expecting privacy as a property of the architecture, rather than as work they now own, get the exposure without the benefit.

The cases where it genuinely wins are narrower and clearer, and they are the subject of the next step.

Some situations do require the data to stay inside your boundary, and they are worth distinguishing from preference.

A legal or contractual prohibition on cross-border transfer or third-party processing. Some public sector work, some defence contracts, and some client agreements simply forbid it, and no amount of contractual assurance from a provider satisfies the term.

Data that cannot lawfully be shared with a processor at all, or where the transfer mechanism is unavailable for the jurisdictions involved. This is a data protection analysis rather than an assumption, and it is worth having the actual answer rather than the cautious guess.

Air-gapped environments, where there is no network path to an external service by design.

Regulated sectors with explicit residency or sovereignty requirements, where the obligation names where processing must occur.

And a client demand you cannot argue with, which is not a technical requirement and is frequently the real one.

What is not on this list, and is often given as a reason: general discomfort about confidential material, an unexamined assumption that regulators require it, or an internal policy nobody has re-read since it was written. Each of those is worth checking before committing to the operational burden, because the burden is substantial and permanent.

The test: can you name the specific obligation, and point to it? If not, the decision belongs on the other grounds in the next step.

Several arguments for self-hosting are stronger than the privacy one and get raised less often.

Version stability. A self-hosted model does not change unless you change it. This directly addresses the problem the AI QA cursus identifies as the change vector nobody's pipeline catches, and for a validated system in a regulated context it is a substantial advantage: your evaluation stays valid until you decide otherwise.

Latency floor. A model on your own hardware removes network round trips and provider queueing. For interactive applications where tens of milliseconds matter, or where you need predictable tail latency rather than good average latency, this is real.

Cost at sustained high volume. The economics invert past a threshold, covered below. A GPU running continuously at high utilisation can be cheaper per token than API pricing, and the crossover is a calculation rather than a matter of opinion.

Modification. Fine-tuning freely, adapting the serving stack, running unusual quantization, or modifying inference behaviour in ways an API does not expose.

Availability independence. No dependency on a provider's uptime, rate limits, or decisions about deprecating a model you depend on. That last one is underrated: an API model you built around can be retired on the provider's schedule.

Notice that only one of these is about data. If the honest reason is version stability or cost, say so, because it leads to different design choices than a privacy-driven deployment would.

The economics have a specific shape, and the crossover is calculable rather than a matter of belief.

An API costs per token, so the bill is linear in usage with no floor. Zero usage costs nothing.

Self-hosting costs the accelerator whether or not you use it, so the bill is a high floor that is flat in usage until you saturate the hardware. The per-token cost is therefore total infrastructure cost divided by tokens actually served, which falls as utilisation rises.

The consequence is that utilisation, not volume, is the variable that decides. A GPU at 10 percent utilisation is expensive per token however many tokens that represents. The same GPU at 80 percent may beat API pricing substantially.

Which makes bursty traffic the worst case for self-hosting. Capacity must be sized for the peak and is idle the rest of the time, so you pay peak cost for average usage.

And the infrastructure cost is not just the accelerator. Add engineering time, which is usually the largest line and the one omitted from the comparison.

Comparisons of API pricing against GPU rental almost always omit the largest term.

The accelerator is the visible cost and often the smaller one. What surrounds it.

Engineering time to build the serving stack, integrate it, and get it to acceptable throughput. Weeks initially, and it is specialist work.

Ongoing operation. Someone owns capacity, upgrades, incidents and the on-call rotation. An API has none of this, which is what you are actually buying.

Redundancy. A single accelerator is a single point of failure, so a production deployment needs at least two, which doubles the hardware line before serving any additional traffic.

Utilisation waste. Capacity sized for peak sits idle at trough, and that idle time is paid for.

Model evaluation and migration. Every model upgrade means re-benchmarking, re-tuning the serving configuration and re-validating quality, which is work an API consumer does not do.

And the opportunity cost, which is the real one: the engineers operating inference infrastructure are not building product.

A rough sanity check before committing: if the projected API spend is less than the fully-loaded cost of one engineer, self-hosting is very unlikely to be cheaper, whatever the per-token arithmetic suggests. That threshold surprises people, and it is why self-hosting for cost reasons makes sense at genuinely large scale and rarely below it.

The choice is not binary, and the intermediate arrangements are underused because the debate is framed as an either-or.

A managed open-weights endpoint. A provider hosts an open model and serves it via API. You get the model's licence terms, freedom to switch providers, and no infrastructure, while giving up the data boundary. Good when the motivation is model choice or avoiding lock-in rather than data residency.

A dedicated instance from a provider. Your own capacity within their platform, often with stronger isolation and residency guarantees. Reduces the multi-tenancy concern without you operating anything.

Virtual private cloud deployment. Some providers will run their model inside your cloud account, so data stays within your network boundary while they retain operational responsibility. This addresses a large share of genuine residency requirements at far lower operational cost than true self-hosting, and it is the option most often missed.

Hybrid routing. Sensitive requests to a small self-hosted model, everything else to an API. Adds routing complexity, and it means you only pay the self-hosting cost for the traffic that requires it.

And on-device for the narrow set of cases where a small model runs on the user's own hardware, which makes the privacy question disappear entirely.

The practical instruction: before committing to running inference yourself, check whether one of these satisfies the actual requirement. Frequently the requirement is data boundary rather than operational control, and the third option delivers it.

Open weights and open source are different things, and the difference has caught organisations out.

Some model weights are released under genuine open source licences such as Apache 2.0 or MIT, which impose essentially no restriction on commercial use, modification or redistribution.

Others are released under custom licences that permit a great deal while restricting something specific. Common restrictions include usage above a stated scale, competing uses, requirements to name the model in derivative products, and acceptable use policies that constrain application domains. These are frequently described as open, and they are not open source in the conventional sense.

And some weights are available for research only, which prohibits production use entirely.

Three practical consequences.

Read the actual licence before building on a model, not the announcement. The announcement says open; the licence says what you may do.

Check the acceptable use policy separately, since it can bind even under a permissive weights licence and can restrict exactly the domain you intend to operate in.

And note that licence terms flow to fine-tuned derivatives, so an adapter trained on a restricted base inherits the restriction.

This matters for the AI Act too. The Act's exemption for free and open-source AI systems is narrower than usually reported and does not extend to high-risk systems, prohibited practices, or those subject to Article 50 transparency duties, so an open licence does not remove the obligations.

A sequence that reaches a defensible answer.

Name the actual requirement. Data boundary, version stability, latency, cost, modification, or independence. Be specific, because each points to a different solution and several are satisfied without self-hosting.

Check whether a middle option satisfies it. Virtual private cloud deployment in particular addresses most genuine residency requirements at a fraction of the cost.

If the requirement is cost, do the arithmetic properly. Projected utilisation, fully-loaded infrastructure, engineering build and run, redundancy. Compare against actual API spend at your real volume, not list pricing at hypothetical volume.

If the requirement is privacy, ask what specifically must not leave, and whether a smaller sensitive subset could be routed separately rather than moving everything.

If the requirement is version stability, note that model pinning from a provider may achieve it, and check before building infrastructure for it.

And if you proceed, size the commitment honestly to whoever approves it: this is a permanent operational responsibility, not a project, and the team taking it on stops doing something else.

The honest default for most organisations is that self-hosting is not justified, and the ones for whom it is know precisely which of the reasons above applies. Where nobody can name the reason, the decision is usually being made on discomfort rather than requirement.