Evaluation Harnesses, Reporting, and Release Governance

A dashboard shows F1 = 0.82, but no one can identify the dataset snapshot, threshold, label version, or code commit that produced it. The number may be correct and still be unusable evidence. An evaluation harness turns a number somebody computed into a reproducible record linked to a specific claim and decision.

That record has a name and a shape. Model cards were proposed in 2018 by Mitchell and colleagues: “short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type)”. The same cards, they wrote, “also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information”.

Every element of that description is a denominator, a condition, or a procedure. None of it survives the trip to a dashboard tile. The card ships with the release. It is not a summary written afterwards from a dashboard.

Each layer should be immutable or explicitly versioned. The layer teams most often leave unversioned is the metric implementation. In the June 2023 MMLU comparison discussed below, it was the only layer that changed. It put LLaMA-65B at 0.637, 0.636 and 0.488 on the same benchmark. That was enough to reverse which model looked best.

Useful evaluation tests include more than happy-path examples, because the reference data can move a score as easily as the model can. In 2019 Recht and colleagues rebuilt the CIFAR-10 and ImageNet test sets, following the original collection procedures, then re-scored models that nobody had touched. Their abstract reports what came back: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.” The models were fixed. The measuring stick was not.

The follow-up matters as much as the finding. Engstrom and colleagues re-analysed the replication in 2020 and corrected for statistical bias in the way the new test sets had been built. Of the original 11.7% ± 1.0% ImageNet drop, only 3.6% ± 1.5% still had no explanation. So the reference data moved the number, and then the method used to rebuild the reference data moved most of that movement back. A harness that cannot separate those two effects will report both of them as model quality.

Good reports answer questions that summary dashboards omit, and the difference between the two forms can be measured on a real system. The Epic Sepsis Model was a proprietary vendor model deployed at hundreds of US hospitals, and its reported denominators had never been published. Somebody else published them. In 2021 Wong and colleagues validated the model externally at Michigan Medicine, on 27,697 patients with 38,455 hospitalizations between 6 December 2018 and 20 October 2019.

The hospitalization-level AUC was 0.63 (95% CI, 0.62–0.64). Sepsis occurred in 2,552 hospitalizations and the model missed 1,709 of them, 67%. An alert threshold of 6 or higher fired on 6,971 of the 38,455 hospitalizations, 18%. Their conclusion: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.” A second group repeated the exercise in 2023 at two county emergency departments, on 145,885 ED encounters. Sensitivity there was 14.7%, aligned with prior external validations and slightly worse than Epic's published analysis.

Nothing in those two paragraphs is a better metric than the vendor's. Every item is a population, a date range, a support count, an interval, or an alert burden — the parts a scorecard drops. In the EU those parts are now compulsory for high-risk systems. The EU AI Act of 2024 states it in Article 15(3): “The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.” Article 17(1)(d) requires the provider's quality management system to document the “examination, test and validation procedures to be carried out before, during and after the development of the high-risk AI system, and the frequency with which they have to be carried out”. Article 12(1) requires the system to record events automatically over its lifetime. The declared metric, the procedure behind it, and the log that lets it be recomputed all ship with the product.

A release can proceed despite a missed criterion when stakeholders accept a documented risk and mitigation, and the exception should name an owner, scope, expiration, monitoring trigger, and rollback condition.

Otherwise temporary waivers become permanent hidden policy and weaken every future gate.

Medical devices show what this discipline looks like when it is binding rather than advisory. FDA's final guidance on predetermined change control plans, issued in 2024 and reissued in 2025, lets a manufacturer pre-authorise specified model changes without a new marketing submission. The plan has to carry three parts: a Description of Modifications, a Modification Protocol and an Impact Assessment.

Five guiding principles for such plans were set out in October 2023 by the FDA, Health Canada and the MHRA: Focused and Bounded, Risk-based, Evidence-Based, Transparent, Total Product Lifecycle Perspective. Under the first of them, a plan must characterise the “plans in place to safely modify the device within the bounds of the PCCP, including methods for verifying and validating the changes and mechanisms to detect and revert or stop implementation of a change that fails to meet specified performance criteria”.

That is the template for an internal exception too. Note the order of operations: what may change, how each change will be verified and validated, and the pre-agreed trigger for reverting it. All of it written down before the change ships, not after it fails.

Automate routine evidence while preserving human accountability. The shape of this workflow is not merely local practice. Congress directed the Director of NIST, in the National Artificial Intelligence Initiative Act, to “work to develop, and periodically update ... a voluntary risk management framework for trustworthy artificial intelligence systems”. The result arrived on 26 January 2023: the AI Risk Management Framework, 48 pages, known as AI RMF 1.0.

Steps 1 to 3 have a line in it: “MEASURE 2.1: Test sets, metrics, and details about the tools used during TEVV are documented.” Step 3 has a second one. MEASURE 2.13 requires the effectiveness of the TEVV metrics themselves to be evaluated and documented — the measurement system is inside the scope of measurement. Step 5 is MANAGE 1.1: “A determination is made as to whether the AI system achieves its intended purposes and stated objectives and whether its development or deployment should proceed.” A disposition is a recorded determination with a name attached, not the absence of an objection.

A strong harness reduces manual copying, preserves protected test access, and generates consistent reports. It should also resist false certainty, by requiring support, intervals, and known limitations.

Leakage is not a rare accident, and somebody counted it. Kapoor and Narayanan surveyed “literature in fields that have adopted ML methods”, looking for reproducibility failures rather than for results. They found “17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions”. They set out “a detailed taxonomy of eight types of leakage, ranging from textbook errors to open research problems”. The pipeline was the fault, in eight documented ways. Their remedy is procedural rather than statistical: researchers should “test for each type of leakage by filling out model info sheets”.

Attaching a sheet like that to the workflow changes behaviour measurably. NeurIPS 2019 introduced a reproducibility program: a code submission policy, a community reproducibility challenge, and the Machine Learning Reproducibility checklist attached to every submission. Of 6,743 submissions, 21.1% were accepted, 40% carried a code link at submission and 74.4% by the camera-ready deadline. At ICML 2019 the same two figures were 36% and 67%. At NeurIPS 2018, code at camera-ready was under 50%. The report carrying those numbers starts from the premise this lesson has been arguing throughout: “Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings.”

No one was persuaded of anything new. The checklist made the traceable path the path of least resistance, and the numbers moved within a single conference cycle. The goal is not to automate judgment away. It is to ensure that human judgment is applied to complete, traceable evidence.