Operating It: Security, Upgrades, and Knowing When to Stop

Self-hosting removes a third party from the data path and adds an attack surface you are responsible for. Both halves are real.

The inference endpoint is the obvious one. It accepts arbitrary text, is expensive to serve, and is frequently deployed inside a network perimeter with weak authentication because it is treated as internal infrastructure. An unauthenticated endpoint is available to anything that can reach it, including a compromised internal service.

Model supply chain. Weights downloaded from a public hub are executable content in a meaningful sense, and older serialisation formats could execute code on load. Prefer safetensors, verify checksums, and pin versions rather than pulling latest, for the same reasons you would with any dependency.

Resource exhaustion. A model server is trivially expensive to attack: long prompts and long generations consume capacity, so an attacker who can reach the endpoint can deny service to everyone else cheaply. Rate limits and token caps are not optional.

Log exposure. Prompt and completion logs are frequently the most sensitive data store in the deployment, and they are usually created by default without anyone deciding to.

And the guardrail layer is now yours too. An API provider applies safety filtering you inherit; a raw open model applies none, so everything in the guardrails cursus becomes your build rather than your configuration.

This deserves its own treatment because it is the most consistently underestimated consequence of moving off an API.

Commercial API providers apply substantial safety work that arrives free with the endpoint: alignment training, input and output filtering, abuse detection, and continuous updating as new attack patterns emerge. A team building on an API has been running on top of that layer without necessarily noticing it.

An open-weights model served raw has whatever alignment its training gave it and nothing else. There is no input classifier, no output filter, no abuse detection, and nothing being updated as jailbreak techniques evolve.

So a migration from API to self-hosted that keeps the application code identical will produce a system that behaves differently on adversarial input, and the difference will not show up in functional testing.

What has to be rebuilt, from the guardrails cursus: input rails, output rails, and the architectural containment that matters more than either. Plus the red-team regression suite, since you no longer inherit a provider's ongoing work against new attacks.

Two practical notes. Open safety classifiers such as the Llama Guard family exist and are the sensible starting point rather than building from nothing. And this work is a real line in the cost comparison from the first lesson, and it is almost always omitted from it.

If the deployment is internal-only with trusted users, the requirement is lighter. If it is customer-facing, it is not.

Model upgrades are where self-hosting's version stability advantage turns into work, and the sequence matters.

The advantage is that nothing changes until you decide. The cost is that when you decide, the entire validation burden is yours, and there is no provider doing the compatibility work.

What actually has to happen. Evaluate the candidate on your own evaluation set against the incumbent, since benchmark improvements do not reliably transfer to your task. Re-tune the serving configuration, because memory profile, optimal batch size and quantization tolerance all differ. Re-run the red-team suite, since safety behaviour differs between models and a fixed jailbreak may reopen. Re-check prompts, because prompts tuned for one model frequently underperform on another. And re-validate anything downstream that depended on output format.

Only then does it go through the release gate and a staged rollout.

The honest framing: an upgrade is a project of days to weeks, not an afternoon. Which is why many self-hosted deployments run models that are considerably behind, and why that lag should be a conscious decision rather than a discovery.

The cost case from the first lesson depended on utilisation, so operating the deployment is substantially about defending that assumption.

Measure utilisation continuously, and treat it as the primary cost metric rather than total spend. A cluster at 15 percent utilisation is failing the business case whatever the absolute bill.

Size to a realistic peak, not the theoretical one. A peak that occurs for twenty minutes a week does not justify permanent capacity, and queueing briefly is usually acceptable where a graceful degradation path exists.

Use a fallback for overflow. Routing excess traffic to an API during peaks lets you size for the sustained load rather than the maximum, which materially improves the economics and requires the application to support both paths. This is the most useful architectural pattern in this lesson and it is underused.

Batch what can be batched. Non-interactive work, summarisation, classification, enrichment, can run in off-peak windows on capacity that is otherwise idle, which raises utilisation without adding hardware.

And revisit the decision on a schedule. The cost comparison depends on API pricing, hardware cost, model efficiency and your traffic, and all four move. A deployment justified two years ago on a cost argument may no longer be, and nobody will notice unless someone checks.

That last point is the one organisations skip, because the infrastructure exists and re-examining it feels like relitigating a settled decision.

Four categories, and the third is the one most often missing.

Infrastructure: accelerator utilisation, memory usage, temperature and throttling, and host health. Standard, and necessary.

Serving: queue depth, time to first token, inter-token latency, throughput, and rejected or timed-out requests. Queue depth is the leading indicator that capacity is becoming inadequate, and it moves before user-visible latency does.

Quality: the evaluation set run on a schedule, exactly as the AI QA cursus prescribes. This is the category teams omit, on the reasoning that the model is pinned so quality cannot change. That reasoning is wrong, because quality can move through a serving configuration change, a quantization change, a prompt change, a retrieval change, or a subtle upgrade to the inference framework that alters sampling behaviour.

Safety: guardrail firing rates, refusal rates, and the red-team regression suite on a schedule. Since you no longer inherit a provider's ongoing safety work, this is now yours to watch.

And cost per token, computed from actual spend and actual tokens served, which is the number that tells you whether the original justification still holds.

The general rule is the same one that runs through this catalogue: instrument the thing you claimed as the benefit. A deployment justified on latency should alert on tail latency, and one justified on cost should alert on utilisation.

Six ways self-hosted deployments go wrong, none of which appear in a proof of concept.

The proof of concept that became production. A model running on one instance, configured by hand, with no redundancy, no monitoring and no runbook, which is now load-bearing because it worked and nobody scheduled the hardening.

Single point of failure. One accelerator, and its failure takes the service down. Redundancy doubles the largest cost line, which is precisely why it gets deferred.

Utilisation collapse. Traffic did not grow as projected, and the cost case that justified the deployment quietly stopped holding, with nobody re-running the comparison.

The expertise concentration. One engineer understands the serving stack, and the deployment becomes fragile in a way that is invisible until they are unavailable.

Model staleness. Upgrades are a days-to-weeks project, so they get deferred, and the deployment runs a model that is materially behind while the team assumes it is current.

And silent quality decay, because nobody ran the evaluation set after a configuration change that seemed innocuous.

The common structure across all six: self-hosting is an operational commitment, and each failure is what happens when it is treated as a build. The countermeasure is unglamorous, which is that someone owns it, it has a runbook, and the things claimed as benefits are measured on a schedule.

Deployments are much easier to start than to retire, and the signals that one should be retired are worth naming in advance.

Utilisation stayed low. The cost case depended on it and it did not materialise, so you are paying a premium for capacity nobody uses.

The original requirement changed. The client that demanded on-premise processing left, the regulation was clarified, or the sensitive workload moved elsewhere. The deployment now exists because it exists.

A middle option now satisfies the requirement. Virtual private cloud deployment or a dedicated instance may have become available for your model and jurisdiction since you decided.

The model gap widened. Open models you can practically serve fell materially behind what an API offers for your task, and the quality difference now costs more than the deployment saves.

The expertise left. If nobody remaining can safely upgrade or debug the stack, the deployment is a liability regardless of its economics.

And the maintenance is displacing product work, which is the opportunity cost from the first lesson arriving as an observable fact.

Retiring is a legitimate outcome and it is treated as a failure, which is why deployments persist past their justification. The organisational move that prevents it is a scheduled review with the original justification written down, so the comparison is against what was actually claimed rather than against the sunk cost.

Five claims.

Self-hosting changes who owns the controls rather than whether controls are needed. A badly run local deployment can be less private than a well-configured API integration, so privacy is a reason only when you can name the specific obligation.

The reasons that hold up best are version stability, latency floor, cost at sustained high utilisation, freedom to modify, and independence from a provider's decisions. Only one of those is about data.

Utilisation, not volume, decides the economics, and engineering time is the largest cost line and the one omitted from comparisons. If projected API spend is below the loaded cost of one engineer, self-hosting is unlikely to be cheaper.

Serving is a real discipline. Generation is memory-bandwidth-bound, so continuous batching and KV cache management determine throughput, and the KV cache rather than compute is what limits concurrency.

And you inherited a safety layer from your API provider that you now have to build, which is the most consistently underestimated consequence of the move.

Check the middle options first. Virtual private cloud deployment satisfies a large share of genuine residency requirements at a fraction of the operational cost, and the debate is usually framed as an either-or when it is not.