Diligence and Evaluation: Testing the Claim

For any purchase where individuals are affected, request a defined set of documents rather than accepting a sales deck. The request itself is diagnostic: how a vendor responds tells you as much as what they send.

The instructions for use, if the system is high-risk. This is the Article 13 document and it is the most valuable item in the pack.

The vendor's own classification of the system and the reasoning behind it, including any derogation relied on.

Performance results, disaggregated across the populations the system will be used on, against a metric appropriate to your decision.

A description of the training data at a level sufficient to judge representativeness for your context: sources, time period, geography, languages, and known gaps.

A statement of known limitations and failure modes.

The security position, including handling of AI-specific attacks.

Data handling terms: retention, training use, subprocessors, deletion.

And the incident and change history: what has gone wrong, and how the system has changed.

A vendor selling into regulated European markets should have most of this assembled. One that treats the request as unusual is telling you where they are in their own compliance work.

For a high-risk system, Article 13 requires the provider to supply instructions for use that are concise, complete, correct and clear, and it specifies their contents. That specification is what makes the document so useful to a buyer.

It must state the intended purpose, which fixes the boundary beyond which you become the provider yourself.

It must state the levels of accuracy, including the metrics used, along with robustness and cybersecurity levels. So the number you would otherwise have to extract from a salesperson is a required disclosure.

It must describe the system's performance regarding the specific persons or groups it is intended to be used on, which is the disaggregated performance question answered by obligation rather than by negotiation.

It must state known or foreseeable circumstances that may lead to risks to health, safety or fundamental rights.

It must specify the input data required, and describe the human oversight measures including the technical measures that help a person interpret output.

And it must describe the logging mechanisms, which determines whether you can meet your own retention duty.

So the practical instruction is short: ask for it during evaluation, not after signing. If the system is high-risk and the vendor cannot produce it, either they do not consider it high-risk, which is a question worth pursuing, or they are not ready to place it on the market.

Understanding what sits underneath the product changes what you can rely on and who else is in the relationship.

Is this a model the vendor trained, or a third-party model they build on? If the latter, you have an unnamed party in your supply chain. Ask which one, and what happens to your data on the way through.

Where the vendor builds on a general-purpose AI model, the model provider has its own obligations, including supplying downstream providers with information about capabilities and limitations. That is what makes your vendor's answers possible, and a vendor who cannot describe the base model's limitations has not collected what they are entitled to.

What data was the model trained or fine-tuned on? You are unlikely to get a full inventory and do not need one. You need enough to judge fit: which populations, which languages, which time period, which geography, and what is known to be underrepresented.

Does the system continue to learn in deployment? If so, on whose data, and within what bounds. A model learning from your users' behaviour is a different risk profile, and the Act's feedback-loop requirement applies to systems that continue to learn.

And how often does the model change, with what notice? This is the question that determines whether the thing you evaluated is the thing you will be running in six months.

Standard security diligence covers the infrastructure. AI systems add attack classes that a normal questionnaire will not ask about, and Article 15 names several of them explicitly.

Data poisoning: manipulating training data to alter behaviour. Ask how training data is validated and whether customer-supplied data can influence a shared model.

Model poisoning: interfering with the model itself or its supply chain, including through third-party weights or dependencies.

Adversarial examples: inputs crafted to cause a mistake. Ask whether the vendor tests for this and what mitigations exist.

Model evasion: crafting inputs to avoid detection, which matters particularly for fraud, moderation and screening systems where an adversary has an incentive.

Confidentiality attacks: extracting training data or model parameters through the interface, which is a data protection issue where training data included personal data.

And prompt injection for anything built on language models: instructions hidden in content the system processes, which redirect its behaviour. Any system that reads documents, emails or web content on your behalf has this exposure, and a vendor who has not thought about it has not thought about the class.

The useful question across all of these is not do you protect against X but how do you test for X and what did the testing find.

A pilot is only evidence if it was capable of producing a negative result. The structure that achieves that is fixed before the vendor is involved.

The threshold comes first, from the scoping specification in the previous lesson. Then a held-out sample of your own data, representative of the population the system will be used on, with labels the vendor has never seen.

Run the system against it, disaggregate the results, and compare against the threshold set in advance.

Running in shadow mode is the strongest form: the system produces outputs on live cases while humans decide as usual, and nothing the system says affects anyone. You get real-distribution performance with no exposure, and you can measure the disagreement between system and human directly.

The decision at the end is against the pre-set threshold, not against the alternatives. A tool that beats two worse tools and misses your bar has not passed.

Four design choices separate a pilot that produces evidence from one that produces enthusiasm.

Use your data, not the vendor's sample. Performance on their distribution is not a prediction about yours, and the gap is often the entire finding.

Hold the test set back. If the vendor tunes against the data you send, the result measures their tuning rather than the system. Keep a portion they never see and evaluate on that.

Include the hard cases deliberately. A random sample under-represents exactly the situations where errors are costly: the ambiguous applications, the unusual formats, the underrepresented groups. Oversample them and report separately.

Measure the human alongside the machine. The relevant comparison is not the system's accuracy in isolation but whether the human-plus-system outcome beats the human alone. A system that is individually accurate but induces automation bias can make the combined outcome worse, and only measuring the pair reveals it.

And run it long enough to see variation. A two-week pilot in a quiet period tells you about quiet periods.

One discipline that costs nothing: write down, before starting, what result would cause you to reject the tool. If no such result exists, the pilot is a demonstration and should be described as one.

Eight vendor responses that should slow or stop a procurement. None is conclusive alone, and several together are.

Cannot state their role under the AI Act, or classifies confidently without reasoning.

Declines to provide disaggregated performance, or provides only aggregate accuracy and treats the request as unusual.

Claims the system is unbiased, which is not a property any system has, rather than describing what was measured and what was found.

Cannot describe any failure mode. Every deployed system has them, so this indicates either inexperience or evasion.

Will not permit evaluation on your data, or insists the pilot use their environment and their sample.

Cites certifications that do not exist in the regime. There is no general AI Act certification for most Annex III systems, since those are self-assessed, so a claim of being certified compliant warrants asking exactly what was certified and by whom.

Treats compliance as your problem entirely, when the deployer duties depend on information only they hold.

And will not commit to notifying material model changes, which means the evaluated system and the deployed system may diverge silently.

The unifying signal: a vendor who has done the work can answer these comfortably, because they had to answer them internally first.

Scoring matrices tend to reward completeness of response rather than quality of system, and the weighting decides the outcome more than the scores do.

Three adjustments improve them.

Weight by what would actually cause failure. If the realistic failure mode is a subgroup being systematically disadvantaged, subgroup performance should dominate the matrix rather than sit alongside integration effort at equal weight.

Score evidence quality separately from the claim. Two vendors claiming the same accuracy are not equivalent if one produced a held-out evaluation on your data and the other produced a marketing figure. Record how you know, not only what you were told.

Include the do-nothing baseline as a scored row. It is the option that keeps the comparison honest, and its absence is why procurements that should have concluded with no purchase conclude with a purchase.

One further caution about integration cost. It is genuinely important and it is the criterion most likely to be overweighted, because it is easy to estimate and falls on the team doing the estimating. Performance disparities fall on people outside the room and are harder to quantify, so a matrix built by the implementing team will systematically underweight them unless someone deliberately corrects for it.

The output of diligence is not only a decision. It is four artefacts that feed everything downstream, and capturing them here is much cheaper than reconstructing them later.

The measured performance, disaggregated, on your data. This becomes the baseline against which drift is detected once the system is running, and without it you cannot tell later whether the system got worse or you got unlucky.

The documented limitations. These are the input to human oversight design, since oversight has to be built against the specific ways this system fails rather than against failure in general.

The classification and role assessment, with reasoning, which populates the inventory entry and satisfies the record-keeping the Act's judgement calls require.

And the list of things you accepted. Every procurement involves residual risk, and the difference between a governance decision and an oversight is whether someone named accepted it in writing.

Organisations that treat evaluation as a gate discard all four the moment the contract is signed, and then rebuild them badly a year later when a question arrives. Treating it as a data-collection exercise that happens to end in a decision is the cheaper posture.

The next lesson turns these into contract terms and an ongoing relationship.