Evaluation as Decision Evidence

On the same 38,455 hospitalizations at Michigan Medicine, the Epic Sepsis Model scored an area under the curve of 0.63, a sensitivity of 33%, a specificity of 83% and a negative predictive value of 95%. Every one of those numbers is correct. All four were computed on the same patients, at the same alert threshold of 6 the hospital was actually using. A buyer handed the last of them learns nothing about the two thirds of sepsis patients the model failed to identify.

Evaluation begins by naming the decision the evidence has to support. A metric becomes useful only once its population, unit, time window, workflow and acceptable failure modes are explicit. As often as not, it becomes useful only once somebody outside the vendor has computed it.

Teams often compress several claims into the sentence “the model performs well.”

Google's retinopathy screening model went to eleven clinics in Thailand, and there the claims came apart. In laboratory evaluation it had been reported at over 90% accuracy and described as specialist-level. Beede and colleagues followed it into the clinics in 2020. More than a fifth of the images the nurses captured were refused by the system's own quality threshold, and slow internet connections added further delay. MIT Technology Review put the clinic conditions in one sentence: “With nurses scanning dozens of patients an hour and often taking the photos in poor lighting conditions, more than a fifth of the images were rejected.”

The predictive claim travelled to Thailand intact. The system claim did not. No amount of additional accuracy would have recovered a patient whose photograph the model declined to read.

Each link narrows what the final numbers mean, and every failure in this lesson is a link that nobody wrote down. The Epic Sepsis Model had an operating point — a score of 6 — before it had a local validation. Ofqual's standardisation model had a defensible claim about centres and an indefensible one about students. Google Flu Trends got its first post-release check when Lazer and colleagues ran one, eight years in.

Take the Michigan Medicine figures and put each of the four claims to them in turn. The dataset never changes. The verdict does.

Use this before calculating any headline metric. Three regulators already require most of it in writing. On 27 October 2021 the U.S. FDA, Health Canada and the UK MHRA jointly issued ten guiding principles for Good Machine Learning Practice for Medical Device Development. Principle 7 refuses to accept a model score on its own: “Where the model has a 'human in the loop,' human factors considerations and the human interpretability of the model outputs are addressed with emphasis on the performance of the Human-AI team, rather than just the performance of the model in isolation.” Principle 8 requires test plans covering the intended patient population, important subgroups, the clinical environment, measurement inputs and potential confounders. Steps 2, 3 and 4 below are not this lesson's advice. They are a regulatory expectation with a date on it.

Step 2 is the one that gets skipped, and England's summer 2020 A-levels show the price. With exams cancelled, Ofqual awarded grades from its Direct Centre Performance standardisation model. That model was built to reproduce a centre's grade distribution, and it was evaluated at that unit. The decision it was actually making was about an individual student's university place. On results day, 13 August 2020, 39.1% of entries were downgraded from the centre assessment grade and only 2.2% were raised. Ofqual's own evaluation records 39% of entries lowered and just over 2% raised: “in most instances where there were differences, teachers' assessments of grades were higher than the results produced by the standardisation method (39% of entries)”. The calculated grades were abandoned four days later. A model can be right about the unit it was measured on and catastrophic at the unit it decides.

The final report should say what was tested, on whom, under which protocol, with what uncertainty, and for which decision. It should also state what remains unknown.

That discipline stopped being a matter of professional taste on 13 June 2024, the day the EU adopted the Artificial Intelligence Act. Article 15(3) of Regulation (EU) 2024/1689 provides: “The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.” Both halves are named. Not the level alone, which is the marketing number. Not the metric alone, which is the technical alibi. And they travel with the product, to the person deploying it. Article 15(2) then tasks the Commission with encouraging the development of benchmarks and measurement methodologies for those levels — a legislature conceding that the metric worth declaring is not yet settled.

That requirement prevents a narrow score from becoming a broad marketing claim, and it makes later monitoring possible.