The Release Gate: Deciding What Ships

A release gate is a defined set of checks with defined outcomes, run before a change reaches users, where failing has a consequence.

That last clause is the whole thing. A gate whose failure produces a discussion is not a gate, it is a report. The distinguishing property is that somebody has decided in advance what result blocks a release, and that the decision binds when it is inconvenient.

Why this matters more for AI features than for ordinary software: the failure modes are subtle, the evidence is statistical, and the pressure to ship is identical to any other project. Where results are a matter of interpretation, an unbound gate resolves every time in favour of shipping, because the person arguing to hold has to prove harm while the person arguing to ship only has to establish doubt.

Setting thresholds in advance is what removes that asymmetry. The argument happens once, when nobody is under deadline pressure and nobody knows which way the result will fall.

A gate also does something quieter that matters for the regulated-industries case: it produces a record that someone checked, against a standard, before release. That is the evidence a supervisor asks for, generated as a byproduct of a practice that is worth having anyway.

A workable gate for an AI feature has five categories, and each maps to a body of work from elsewhere in the catalogue.

Capability, against the labelled evaluation set: does the system still do the task, overall and on each reported segment?

Safety and policy, against the guardrail configuration: does it still refuse what it should, and has the false-positive rate on legitimate requests moved?

Groundedness, where the system is retrieval-augmented: what proportion of claims are supported by sources, and has the unsupported-claim rate risen?

Security, against the red-team regression suite: do previously fixed bypasses stay fixed, including on the indirect injection channel?

And operational: latency at the relevant percentile, cost per request, error rate, and behaviour when a dependency fails.

The fifth is included deliberately because it is the one most often left out of an AI-specific gate and the one most likely to cause an incident. A change that improves quality and triples cost per request is a change somebody should decide about consciously.

Not every change needs all five. The routing question is which of the five change vectors from the previous lesson fired, since a prompt change and a corpus change put different checks at risk.

A threshold chosen after seeing the result will be chosen to accommodate it. This is not dishonesty, it is how judgement works under pressure, and the defence is procedural rather than moral.

Three kinds of threshold, and they behave differently.

Absolute floors: the system must reach at least this level on this segment. Appropriate where a minimum standard exists independent of the current version, typically for safety and security checks. Non-negotiable by design.

Relative bounds: the new version must not be worse than the current one by more than this margin. Appropriate for capability, where absolute quality is hard to specify but regression is easy to detect. This is the workhorse.

And tolerance bands, which acknowledge non-determinism. A result within the noise band of the baseline is not a regression, and treating it as one produces a gate that blocks randomly and quickly loses authority. Establishing that band requires running the baseline several times, which is a setup cost worth paying once.

Set thresholds per segment rather than globally, because an overall figure holding while a specific population degrades is exactly the failure the gate exists to catch, and a global threshold cannot see it.

Write down the reasoning for each threshold. When someone later asks why the bar is where it is, the reasoning is what makes it defensible rather than arbitrary.

Not every check should block, and a gate that blocks on everything gets bypassed within a month.

The useful split is by whether a failure is unambiguous. Security regressions and safety floor breaches are unambiguous: a previously fixed injection bypass returning is a defect regardless of context, so it blocks automatically.

Capability regressions beyond the tolerance band are ambiguous, because a small drop may be acceptable alongside a large gain elsewhere. These route to a named decision-maker rather than blocking, and the decision is recorded.

Movements inside the tolerance band pass, with the result logged so the trend remains visible even when no single run trips.

And operational changes, cost and latency, route to whoever owns the budget, since the trade is theirs to make.

The design principle: block on the things where no reasonable person would ship, route the genuine judgement calls to someone accountable, and log everything so a slow drift is visible before it crosses a line.

The hardest case is the one the previous lesson identified: a provider updates a model, and nothing in your pipeline fires.

The first requirement is detection. Where the provider offers version pinning, use it, and treat moving the pin as a deliberate change that goes through the gate like any other. Where they do not, you need to notice: run the evaluation set on a schedule, and monitor output distribution in production, since a shift with no input explanation is the signature of a model change.

The second is a defined response. When a scheduled run trips a threshold and no change of yours explains it, that is an incident rather than a curiosity. Someone should be paged, the possibility of a silent model change investigated, and the vendor asked.

The third is contractual, which is where the procurement cursus does its work: notification of material change with lead time, a version pin or a delayed rollout channel, and a right to re-evaluate.

And the fourth is honesty about the residual. Where a vendor ships continuously with no pin and no notice, you cannot gate their changes, only detect them afterwards. That is a real limitation and it belongs in the risk register with a named accepter rather than sitting unexamined, because the alternative is discovering it during an incident and finding that nobody had decided to accept it.

A gate is a point check, and point checks miss things that only appear under real traffic. Staged rollout is the complement, and it changes what the gate has to catch.

Shadow mode: the new version runs on live traffic while the current one serves users, and outputs are compared. Real distribution, no exposure, and it produces the disagreement data that a static evaluation set cannot.

Canary: a small proportion of real traffic, with the segments you care about monitored separately and an automatic rollback condition defined in advance.

Progressive rollout with monitoring at each step, holding at each level long enough for a signal to appear rather than advancing on a timer.

And a rollback path that has been tested. A rollback nobody has exercised is a plan, not a capability, and AI systems complicate it, because reverting a model version may not revert everything if a prompt or retrieval change went out alongside it.

The division of labour is worth stating. The gate catches what a labelled set can catch, which is regression against known cases. Staged rollout catches what only real traffic reveals, which is the long tail of inputs nobody thought to label. Neither substitutes for the other, and organisations that invest in one and skip the other are covered against half the problem.

Gates decay in predictable ways, and each has a countermeasure.

The evaluation set ages. Usage moves away from the cases it contains, and a high pass rate comes to mean the suite is stale rather than the system is good. Countermeasure: refresh from production traffic on a cadence, retiring cases that no longer represent anything.

Thresholds get quietly relaxed. A gate that blocks is inconvenient, and the easiest resolution is to adjust the bar. Countermeasure: treat a threshold change as a change requiring review, versioned like code, with the reasoning recorded. A threshold moved to unblock a release should be visible as exactly that.

Overrides become routine. The first override is a considered exception; the twentieth is the process. Countermeasure: count them, report the count, and treat a rising override rate as a signal that either the thresholds are wrong or the gate is not being respected. Both are worth knowing.

The gate becomes theatre. Everyone knows the result will not block anything, so nobody reads it. Countermeasure: the test from the regulated-industries cursus, which is to ask what happened the last time the gate blocked a release. If the answer is that it never has, the gate is not a control.

A gate with occasional, respected blocks is functioning. A gate with none is either measuring nothing or being ignored.

An evaluation suite is itself a production system with a bill, and the economics shape what is possible.

Every case costs at least one generation, and non-determinism means several runs per case. Add a model judge and the cost roughly doubles. A thousand cases run five times with judged output is a meaningful expense per gate execution, and a gate too expensive to run often gets run rarely, which defeats it.

Four techniques keep it affordable.

Tier the suite. A small fast set on every commit, the full set nightly, and an extended set before a release. Most changes do not need the full suite, and matching depth to risk is what makes frequent running possible.

Use cheap checks where they suffice. Deterministic assertions, schema validation and entailment checks cost a fraction of a judge and cover a surprising amount.

Cache aggressively. Where neither the prompt nor the model version changed, the previous result is still valid, and a large share of a suite is often unaffected by a given change.

And sample rather than running everything every time, provided the sample is stratified across segments so a small population cannot vanish from the run.

The budget itself is worth stating explicitly, because an unfunded evaluation practice degrades silently as someone quietly reduces how often it runs.

Beyond the ship or hold decision, a gate run leaves artefacts that are useful elsewhere, and capturing them costs almost nothing.

A dated record that the system was assessed against a defined standard before release, with the result. This is the pre-deployment evidence the regulated-industries cursus asks for, generated automatically.

A performance baseline for the version that shipped, which is what makes the next comparison possible and what drift monitoring measures against.

A record of accepted regressions: where a decision-maker allowed a change through despite a degradation, with the reasoning. This is a governance decision, and unrecorded it becomes an oversight failure.

And trend data across releases, which reveals slow degradation that no single run trips. A capability sliding two percent per release passes every gate and is a serious problem by the tenth.

That last one is the argument for logging everything rather than only failures. A gate that records only its blocks cannot see the drift that never blocked, and drift that never blocks is precisely the failure mode a point check is blind to.

Which is the summary for this lesson: the gate is the decision, and the record it leaves is worth as much as the decision itself.