Why QA Breaks on AI Systems

Software quality assurance is built on three assumptions, and an AI feature violates all of them at once.

There is a specification. For ordinary software, correct behaviour is defined in advance and a test asserts it. For a summarisation feature there is no specification of the correct summary, only a range of acceptable ones and a much larger range of unacceptable ones.

Behaviour is deterministic. The same input produces the same output, so a passing test means the behaviour is right rather than that it was right once. Run the same prompt twice against a model and you may get different answers, which means a single test result carries far less information than QA practice assumes.

The system changes only when you change it. A regression suite protects against your own edits. When a provider updates a model, behaviour shifts with no commit, no deployment, and nothing for a CI pipeline to trigger on.

Each of these individually would be manageable. Together they mean that the core artefact of software QA, a suite of assertions that pass or fail, does not straightforwardly apply.

The response is not to abandon QA. It is to change what is being asserted, which is what this cursus is about.

The conceptual shift is from asking did this pass to asking what does the distribution of behaviour look like, and has it moved.

A deterministic test asserts one thing about one execution. An AI evaluation samples behaviour and characterises it: this configuration produces acceptable output on this proportion of a representative input set, with these failure modes concentrated in these conditions.

That changes several practices at once.

A single run establishes little, so cases run several times and the result is a rate rather than a boolean.

A pass threshold replaces a pass condition. Not every case must succeed; a defined proportion must, and the threshold is set by consequence.

Comparison becomes relative rather than absolute. The useful question is rarely is this good enough in the abstract but is this better or worse than the version we are running, which makes a stable baseline the most valuable artefact the function owns.

And variance itself becomes a metric. A configuration that is right eighty percent of the time consistently is different from one averaging eighty percent while swinging between sixty and ninety-five, and the second is much harder to operate even though the mean matches.

Teams that carry deterministic habits into this produce suites that fail constantly for reasons nobody investigates, which trains everyone to ignore them.

The scope is wider than model accuracy, and drawing it wrongly is the most common structural mistake when standing up the function.

Capability: does the system do the task adequately on inputs representing real use, including the hard ones?

Safety and policy: does it refuse what it should, and does it avoid outputs the organisation cannot stand behind? This is the guardrail configuration, and testing it is testing a control rather than a model.

Groundedness: are claims supported by the sources the system was given, which is the hallucination work in its testable form.

Security: does it withstand adversarial input, which is the red-team regression suite.

Regression: has any of the above degraded relative to the baseline, across model changes, prompt changes, retrieval changes and configuration changes.

And the conventional software concerns that do not go away: latency, cost per request, error handling, and behaviour when a dependency is unavailable.

The frequent error is scoping the function to the first item alone. A team measuring only capability ships a system that answers well and leaks data, or that answers well until a provider update, because nobody owned the other five.

Cost belongs on that list deliberately. It is the quality attribute most often discovered in an invoice.

Ordinary software has one change vector: your commits. An AI feature has five, and only one of them triggers your pipeline.

Code and prompt changes are yours, and standard CI catches them.

Configuration changes, thresholds, temperature, retrieval parameters, guardrail settings, are often made outside the code path, in a console or a config service, by someone who does not think of it as a deployment.

Corpus changes happen continuously in any retrieval system. Documents are added, edited and deleted by people with no relationship to the AI feature at all.

Model changes come from the provider, on their schedule.

And data drift is the world changing underneath a static system.

The design consequence: a suite bound only to your commits covers one fifth of the surface. Evaluation has to run on a schedule as well as on commit, because three of the five vectors will never trigger a build.

Four numbers that look like quality signals and are not, each of which will be requested by someone.

Test count. In deterministic QA, more tests weakly indicates more coverage. Here it indicates how many cases someone wrote, and a thousand near-identical cases sampled from the same easy distribution measure one thing a thousand times.

Pass rate on a suite you tuned against. If cases were added because the system failed them and then fixed, the suite measures what you already addressed. Rising pass rate on a static suite is at least as likely to mean the suite aged as that the system improved.

Aggregate quality scores from a model judge. Useful as a trend, misleading as a level, and dangerous when averaged across dissimilar tasks. A single number spanning summarisation, extraction and open questions describes none of them.

And user satisfaction alone. It measures whether people liked the output, which correlates with fluency more strongly than with correctness. A confidently wrong answer rates well.

The common failure underneath all four: each is an aggregate that lets a serious failure in a small population hide inside an acceptable average. The reporting that survives is failure-mode-shaped, naming what goes wrong and under what conditions, rather than score-shaped.

The single most valuable thing an AI QA function builds is not a framework or a dashboard. It is a well-constructed set of cases with known good answers, and everything else derives from it.

Four properties distinguish a good one.

Real provenance. Cases drawn from actual usage, support tickets, and logged queries rather than invented by the team. Synthetic cases are phrased the way the system's builders think, which quietly removes the vocabulary mismatch that causes real failures.

Deliberate hard-case weighting. A random sample of production traffic is mostly easy, so it measures the easy path repeatedly. Oversample ambiguity, unusual formats, edge conditions and underrepresented populations, and report those segments separately rather than blending them in.

Stable labels. Someone competent has decided what a good answer is, and the reasoning is recorded so a later reviewer can apply the same standard.

And held-back portions. Keep a slice the team building the system does not see, or you end up measuring how well the system was tuned to the cases rather than how well it works.

A set of a hundred well-chosen labelled cases outperforms ten thousand scraped ones, and it is the artefact that makes every comparison in this cursus possible. It is also the thing that decays if nobody owns it.

AI QA sits between disciplines, and the staffing question determines what the function can actually assess.

A traditional QA engineer brings the discipline that matters most: systematic coverage thinking, the instinct to test the boundary rather than the middle, and the habit of asking what happens when this is wrong. What they typically need is enough understanding of non-determinism and sampling to stop treating variance as flakiness to be eliminated.

A data scientist brings evaluation methodology, metric selection and statistical literacy about sample sizes and confidence. What they typically need is the adversarial and coverage mindset, since evaluation training emphasises measuring performance rather than hunting for failure.

A domain expert is the one who can say whether an output is actually correct, which neither of the others can for a specialist task. This is the role most often missing and least often budgeted.

And a security-minded tester covers the red-team dimension, which the other three will not naturally reach.

For most organisations this is not four people. It is one person with one of these backgrounds, plus structured access to the others, and the practical question is which single background to hire first. The usual answer is the QA engineer, because coverage discipline is harder to acquire informally than evaluation methodology is.

It is worth being clear that most of software QA still applies, because teams sometimes conclude that AI requires an entirely new discipline and discard practice that was working.

The system around the model is ordinary software. Input validation, error handling, timeouts, retries, rate limits, authentication and authorisation all behave conventionally and fail conventionally. A large share of production incidents in AI features are ordinary bugs in this layer rather than model behaviour.

Integration testing is unchanged. Whether the retrieval service is called correctly, whether responses are parsed, whether the schema holds, whether a failure propagates sensibly, all deterministic.

Performance and load testing are unchanged in method, though the cost profile differs, since concurrency limits and token throughput replace CPU as the constraint.

And the fundamentals of coverage thinking transfer directly. Boundary conditions, empty inputs, unusual encodings, very long inputs, adversarial structure. These are QA instincts and they apply.

So the accurate framing is that AI QA is conventional QA plus a new layer for the probabilistic component, not a replacement for it. Organisations that staff the new layer and neglect the old one ship features that fail for entirely traditional reasons, which is both common and avoidable.