Validating a System Whose Behaviour Is Statistical

Independent validation, in the model risk tradition, asks three questions, and they remain the right three for AI systems even where the methods have to change.

Is the approach conceptually sound? Is this the right kind of system for this problem at all, are the assumptions defensible, and does the design match the stated purpose? This is the question that catches the most expensive mistakes, because a technically excellent implementation of the wrong approach fails no test until it fails in production.

Is it implemented correctly and performing? Does the code do what the design says, does it perform adequately on data representative of real use, and does it hold across the populations it is used on?

Is it still working? Ongoing monitoring, and outcomes analysis comparing what the system predicted against what actually happened.

The first question is where validation adds most value and where organisations spend least effort, because it requires domain judgement rather than measurement and cannot be automated. A validator who only checks the second question is performing quality assurance, not validation.

All three must be performed by someone independent of the developers. That is not a comment on developer integrity; it is that people cannot reliably find the flaws in reasoning they themselves produced.

For a conventional predictive model, a credit scorecard, a fraud classifier, a churn model, the established validation method applies almost unchanged, and organisations should use it rather than inventing something AI-flavoured.

There is a defined output, a probability or a class, so accuracy has a meaning. There is a ground truth that eventually arrives, so outcomes analysis is possible. Performance can be disaggregated across subgroups. Benchmark and challenger models can be built and compared. Statistical tests for drift are well established.

This is the comfortable case, and the practical advice for a regulated organisation is to recognise when you are in it. A great deal of what gets called AI in a bank is a gradient-boosted model producing a score, which is a model in exactly the sense the tradition means, and it should go through the existing process rather than a new one.

What has genuinely changed for these systems is scale and opacity, not kind. More features, less interpretable structure, faster retraining. Those stress the process without breaking it: they argue for stronger disaggregated testing, for explainability tooling, and for tighter change control, all of which the framework accommodates.

The discontinuity comes with generative systems, which is the next step.

Four properties of generative systems defeat the classical validation approach, which is the substantive reason SR 26-2 declined to bring them into scope.

The output space is unbounded. There is no set of possible outputs to reason over, so coverage in the usual sense is unavailable.

There is often no ground truth. A summary is not right or wrong in the way a default prediction is. Quality is multi-dimensional and partly a matter of judgement, so outcomes analysis has nothing to compare against.

Behaviour is non-deterministic. The same input produces different outputs, so a single test result establishes little and every measurement needs repetition to characterise a distribution rather than a value.

And the system changes without your involvement. A provider updates the model and behaviour shifts with no deployment on your side, which breaks the change-control assumption the entire framework rests on.

The wrong response is to declare validation impossible and proceed. The right response is to change what is validated. If you cannot validate outputs exhaustively, validate the process that produces and constrains them: the retrieval, the guardrails, the oversight, the escalation, and the monitoring. That is a defensible position and it is documentable, which is what the gap in the first lesson requires.

The shift in what gets validated when the output cannot be exhaustively checked.

Instead of one question about the model, you evidence a chain of narrower claims, each of which can be tested.

The inputs are what you think they are: retrieval is permission-aware, the corpus is current, provenance is known.

The constraints operate: guardrails are configured as documented, and the regression suite from the red-teaming lesson passes.

The output is grounded: sampled outputs are supported by retrieved sources, measured rather than assumed.

Oversight functions: the four conditions hold and were tested, not asserted.

Escalation works: incidents are recognised and routed within their deadlines.

And the whole thing is monitored, with thresholds set in advance and a defined response when one is crossed.

Each link is individually verifiable. Together they support a claim about the system that no single test of the model could.

What should exist before a system in a regulated context goes live, assembled as a pack rather than scattered across teams.

A statement of purpose and scope, precise enough that a change of use would be visible as a change, plus the decisions the system informs and who is affected.

The conceptual soundness assessment: why this approach, what alternatives were considered, what assumptions it rests on, and what would invalidate them. Written by the team, challenged by someone independent, with the challenge recorded including what it disputed.

Performance evidence, disaggregated across the populations the system is used on, against metrics chosen and thresholds set before the results were seen.

The limitations register: where it performs worse, on whom, and under what conditions. Explicitly including what was not tested.

The control design: guardrails, oversight arrangement with its four conditions evidenced, escalation route, and the logging that supports reconstruction.

The monitoring plan: metrics, thresholds, cadence, owner, and the defined response when a threshold is crossed.

The residual risks accepted, each with a named accepter.

And the independent validation opinion, stating what was examined, what was found, and what was not covered.

That last item is the one that distinguishes a pack a supervisor will accept from a folder of documents.

Validation at a point in time is a snapshot of a system that will move. Monitoring is what makes the snapshot meaningful, and four things are worth instrumenting.

Input drift. Is the data arriving still like the data the system was assessed on? This is measurable without waiting for outcomes, which makes it the earliest warning available.

Output distribution. Has the proportion of approvals, high scores or refusals shifted without a corresponding change in the population? A shift with no explanation upstream means something moved in the system.

Outcomes, where they arrive. Comparing predictions against what actually happened is the strongest evidence there is, and its weakness is latency: for a credit decision, ground truth may be a year away, so it validates the past rather than the present.

And control health, which is the part most often omitted: override rates, guardrail firing rates, escalation volumes, and complaint volumes. These reveal a degrading control before the outcome data does.

Two design rules. Set thresholds in advance, because a threshold chosen after seeing a concerning number will be chosen to be comfortable. And define the response before the trigger fires, since deciding what to do while a metric is breaching produces the slowest and worst decisions.

And note that the deployer feedback channel from the procurement cursus is what makes provider-side monitoring possible at all.

The framework assumes you control when the system changes. For a purchased model you do not, and reconciling that is one of the genuinely unsolved operational problems in this area.

The exposure is specific: a provider updates the model, behaviour shifts, and your validation opinion now describes a system that no longer exists. Nothing in your change management fired, because nothing on your side changed.

Four partial mitigations, none complete.

Contractual notification of material change with lead time to re-evaluate, and a version pin or delayed rollout channel where the vendor offers one. This is the procurement lever, and it is why those clauses matter.

A standing regression suite, run on a schedule rather than only on your own deployments, so a silent change surfaces as a test failure rather than as an incident.

Monitoring tuned to detect behavioural shift, since output distribution moving without an input explanation is the signature of a model change.

And a documented position on materiality: what magnitude of change triggers re-validation, decided in advance so it is not negotiated under pressure.

What none of these achieve is the control a supervisor would expect over an internally built model. That residual gap should be recorded as an accepted risk with a named accepter, rather than left unstated, because a supervisor will identify it and the question will be whether you had.

For a generative system in a regulated firm, you are in the gap identified in the first lesson: outside SR 26-2's formal scope, and outside the AI Act's high-risk regime until December 2027 unless the use case sits in Annex III.

That absence is not permission. Sector conduct rules, non-discrimination law and data protection all apply, and the general expectation that a firm manages its risks never had a technology carve-out.

What a defensible position looks like when there is nothing to cite.

State the gap explicitly. Record that you assessed which frameworks apply, found the system outside the formal scope of the obvious one, and say why. A supervisor discovering the gap themselves reads very differently from a firm that identified and addressed it.

Apply the principles rather than the rules. Effective challenge, independent validation, monitoring and documented governance are principles, not procedural requirements tied to a particular letter. Applying them to a system outside formal scope is exactly what a supervisor would expect of a well-run firm.

Document the reasoning at each judgement. Where no standard specifies the answer, the reasoning is the evidence.

And revisit deliberately, since regulators are actively soliciting input on governance for these systems and the position will change. Set a review trigger on it rather than waiting to be told.

Three lessons, one argument.

A supervisor cannot verify a statistical system's behaviour directly, so they verify whether the organisation could have caught a problem. That makes independence, challenge and documentation the substance rather than the paperwork around it.

Human oversight is the control everything else leans on, and it is nominal far more often than organisations realise. It becomes real through four evidenced conditions and it can be tested, which almost nobody does.

And validation, built for models that output a number, has to be redirected for systems that output prose: validate the pipeline and the controls when you cannot validate the outputs, and say plainly that this is what you did and why.

The honest position for a regulated organisation deploying generative AI today is that no settled standard covers it, the deadlines that will are still ahead, and the sector rules that always applied still do. That is an uncomfortable place to operate and it is where the work actually is.

The firms that come out of an examination well will not be the ones that found a framework to cite. They will be the ones that identified the gap, applied the principles anyway, wrote down why, and can show someone independent checked.