Keeping It Alive: Incidents, Redress, and Measurement

Organisations have incident processes, and AI harms mostly do not enter them.

An incident process is built around service disruption. Something broke, someone noticed, an alert fired, and a team responded. The defining property is that the organisation experienced the failure.

An AI harm frequently has none of those properties. The system worked as designed. Nothing broke. No alert fired. The organisation experienced nothing at all, because the cost fell entirely on someone outside it: an applicant who was ranked low, a customer whose claim was routed to a slower queue, a person a moderation system removed.

So the harm is invisible to every mechanism the organisation uses to detect problems. It will not appear in monitoring, it will not page anyone, and the affected person often does not know a system was involved and would not know where to complain if they did.

This is the structural reason responsible AI needs its own detection route rather than relying on existing operational processes. What is required is a definition of harm that does not depend on the organisation noticing, plus a channel that lets the affected person tell you.

Without both, a programme can run cleanly for a year while producing exactly the outcomes it exists to prevent, and report success throughout.

Write the definition before you need it, because during an incident the definition gets negotiated by people with an interest in the answer.

Six categories that between them cover most of what matters.

A person received a materially worse outcome than they should have, because of a system error or a limitation the system has.

A group received systematically worse outcomes than another, whether or not any individual case looks wrong. This is the one that only measurement finds.

Someone was subject to an automated decision without a route to challenge it, or without knowing one was made.

The system was used outside its declared intended purpose, which is why the out-of-scope section of the system card matters.

Personal or confidential information reached somewhere it should not have.

And a person was told something false with material consequences, which is the hallucination cursus arriving as a governance category.

Two drafting notes. Include near-misses, since a harm caught by a reviewer is the cheapest learning available and reporting it should be rewarded rather than treated as a defect record. And write the definition so it does not require intent or negligence, because most of these arise from a system working exactly as built.

Four detection routes, and they differ enormously in latency and in what they can catch.

Measurement is the only route that finds systematic group-level disparities, because no individual case in such a pattern looks wrong. It is proactive, and it only finds what you thought to measure.

Internal reporting catches what staff notice, and depends entirely on people knowing what to report and believing that reporting is safe.

The redress channel catches what affected people tell you, and its yield depends on whether they know a system was involved and whether the route is findable.

And external discovery, a journalist, a regulator, a researcher, a viral post, catches what the other three missed. It has the longest latency and by far the highest cost.

The design implication is direct. Every harm that reaches the fourth route passed through the first three undetected, and investment in the earlier routes is what moves detection left.

The people best placed to tell you a system harmed them are the people it harmed, and most organisations make that impossible without intending to.

Four requirements for a route that functions.

The person has to know a system was involved. Where the AI Act's transparency duties apply this is an obligation, and beyond that it is the precondition for everything else. A decision presented as institutional rather than automated cannot be challenged on the grounds that it was automated.

The route has to be findable, at the moment of the decision, in the communication conveying it, in language a non-specialist reads. A process buried in a policy page is not a route.

A human has to reconsider, with the authority to change the outcome. A challenge that returns the same automated answer is not redress; it is the same decision, repeated.

And the outcome has to be capable of changing. Which requires the reviewer to have the information the audit trail lesson describes, since reconsidering a decision whose inputs cannot be reconstructed is not possible.

The part organisations resist is the volume implication, and it is worth confronting honestly. A findable route generates contacts, most of which will not reveal a harm. That cost is the price of the detection channel, and an organisation unwilling to pay it has chosen not to find out.

A technical postmortem asks what broke. An AI harm review has to ask something harder, because usually nothing broke.

The questions that produce learning.

What happened to the person, described concretely. Starting with the human consequence rather than the system behaviour keeps the review anchored to what matters.

Which decision made this possible? Trace back to the choices from the first lesson: the purpose, the data, the objective, the threshold, the autonomy level, the interface. Almost always one of those, made months earlier by someone not thinking about ethics, is the cause.

Which control should have caught it, and why did it not? Either a control was missing, or it existed and did not function, and those have different remedies.

How long did detection take, and through which route? A harm found by external discovery indicates three failed detection mechanisms, and that is a finding in its own right.

What would have surfaced this at the design gate? This is the question that improves the gate, and it is the one most often skipped because it implicates a decision someone present defended.

And what changes as a result? A review that changes no control, no threshold and no check has documented an event rather than learned from it, which is the same test the guardrails cursus applies to red-team findings.

A responsible AI programme should be able to say whether it is working, and the metrics organisations reach for measure activity rather than effect.

What not to report as success. Number of assessments completed, which measures process volume. Training completion rates, which measure attendance. Number of principles published. All three rise while nothing changes.

What is worth reporting.

Decisions changed. How many systems were narrowed, modified or not built as a result of a gate. This is the direct measure of whether the machinery binds, and a programme reporting zero over a year has an answer to the question from the first lesson.

Detection latency and route. How long between a harm occurring and being known, and which of the four routes found it. A shift toward measurement and away from external discovery is the clearest evidence of maturity.

Coverage of consequence. What proportion of high-tier systems have current disaggregated measurement, a named owner, and a redress route. Better than counting assessments because it is tied to the systems that matter.

Redress volume and outcome. How many challenges, how many upheld. A route generating no contacts is probably not findable rather than evidence of no harm.

And override rate on the gates, since a rising rate means either the thresholds are wrong or the gate is being ignored, and both need knowing.

Five ways a functioning programme becomes a formality within about a year, each gradual enough that no one notices the moment it happens.

The gate stops blocking. Assessments are completed, concerns are noted, launches proceed. Within two quarters teams treat it as a form. This is the same death the AI QA function suffers and it has the same cause: no consequence attached to the finding.

The checklist grows. Each incident adds questions, nobody removes any, and a fifteen-minute exercise becomes ninety minutes of low-attention box-ticking. Length is inversely related to thoughtfulness here.

The measurement lapses. Disaggregated performance was measured at launch and never again, so the programme reports a state of the world from eighteen months ago.

The function is reorganised under delivery. Often presented as improving collaboration, and it removes the independence the whole thing rested on. The structural point recurs across this catalogue because it is the one that decides everything else.

And the people leave. This work attracts people who care about it, and they leave when it stops mattering, which accelerates the decline and removes the institutional memory of why the controls existed.

The countermeasures are in the metrics above. Each failure shows in the numbers before it is visible in the process, provided someone reports them honestly.

Three lessons, one argument, statable in a paragraph.

Principles do not bind because a value is not a decision rule. What binds is a named population, a chosen measurement, a threshold set in advance, and a consequence for missing it. That conversion is small work, and it is the work.

The machinery has to sit where decisions are actually made, which is inside the ordinary build process rather than in a review at the end, because by the end the available outcomes are recommendations. And someone has to be able to stop a launch, reporting somewhere other than to the function whose launch it is.

And the programme is judged after launch, by whether harms are detected through measurement rather than through a journalist, whether the person affected has a route to challenge, and whether reviews change controls rather than documenting events.

The honest test, at any point, is the one from the first lesson: name a decision that came out differently. If the answer is none, the programme has produced documents.

The reason to do any of it is the empty seat. Every other interest in the room has an advocate. The person the system is used on does not, and a responsible AI function is the organisational answer to that asymmetry rather than a compliance exercise that happens to concern ethics.