Experiment Tracking and Decision Records

Put the state-of-the-art GAN algorithms side by side, give every one of them the same hyperparameter search and the same random restarts, and the ranking collapses. That is the comparison the field had not run for itself. Lucic, Kurach and three colleagues ran it in 2018.

Each algorithm got one shared budget instead of whatever its own authors had happened to spend on it. The abstract reports what survived: “We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes.” They found no evidence that any tested algorithm consistently outperforms the original non-saturating GAN.

Every one of those published wins had been logged somewhere. What no tracker recorded was the asymmetry that produced them.

An experiment tracker holds every parameter and every metric from six months of work. Nobody can still say which runs used the same data split, whether the baseline received equal tuning, or why the selected candidate was preferred. The data is all there. The comparison is not.

The cost of that has been measured. Edward Raff tried to re-implement papers by hand, from the text alone, without looking at any released code — papers published between 1984 and 2017. “After this selection process, we are left with 255 papers, of which 162 (63.5%) were successfully replicated and 93 were not.”

Two record-keeping facts predicted the outcome. One was whether the paper specified its hyperparameters (p = 8.45 × 10⁻⁶). The other was whether the authors replied to questions (p = 6.01 × 10⁻⁸). Of 24 papers whose authors did not respond, replication succeeded once. Of 26 whose authors did respond, it succeeded 22 times.

Tracking is useful only when runs belong to a comparison protocol. The record must preserve the question, the controlled variables, the budget, the evidence and the conclusion — negative results included.

One dashboard cannot answer every question. A run record supports reproduction and debugging of one execution. An experiment report supports comparison across controlled alternatives. A decision record supports accountable selection and future reconsideration.

For high-risk systems the third artifact is no longer a matter of team taste. The Artificial Intelligence Act — Regulation (EU) 2024/1689, published in the Official Journal on 12 July 2024 — makes it a legal artefact. Annex IV is the technical documentation required by Article 11(1). Point 2(b) obliges the provider to document “the design specifications of the system, namely the general logic of the AI system and of the algorithms; the key design choices including the rationale and assumptions made, including with regard to persons or groups of persons in respect of who, the system is intended to be used; the main classification choices; what the system is designed to optimise for, and the relevance of the different parameters”. Point 2(g) adds the validation and testing procedures, the metrics used to measure accuracy and robustness, and “test logs and all test reports dated and signed by the” responsible persons.

Rationale, assumptions, the optimisation target, the trade-offs made, and a dated signature on the test report. That is a decision record, written into law rather than recommended in a style guide.

Comparison begins before the first training job starts. State the question. Fix the controlled design. Capture the execution record. Analyse uncertainty and cost. Then record the decision and the evidence that justified it.

NeurIPS 2019 turned part of that anatomy into a submission requirement: every submitted paper had to answer the Machine Learning Reproducibility checklist. Pineau and seven co-authors, the conference's reproducibility chairs, reported on what came back across 6,743 submitted papers. Of those, 40% carried a link to code at submission and 74.4% by camera-ready. NeurIPS 2018 had been under 50% across 4,856 papers; ICML 2019, 36% rising to 67% across 3,424 papers.

The execution record improved quickly. The analysis stage did not, and the checklist answers say so in the report's own words: “In particular, it seems surprising to have 87% of papers that see value in clearly defining the metrics and statistics used, yet 36% of papers judge that error bars are not applicable to their results.”

Uncertainty is the field a community waives first. It is also the field a decision rests on.

A run records what executed. A decision record explains what alternatives were compared, which evidence mattered, what uncertainty remained, and why the team chose a path.

Annex IV point 2(b) asks for exactly the second kind of content: the key design choices with their rationale and assumptions, and what the system is designed to optimise for. No quantity of run logs supplies it. A log records a value, not the reason it was chosen.

Keeping the two records separate prevents selection logic from disappearing into dashboards. It also makes it possible to revisit a decision when the product contract or the population changes.

A short decision memo exposes hidden advantages and unresolved risks. Restate the question. Verify that data, split, tuning budget, compute and evaluation protocol are comparable. Summarize the evidence with its uncertainty and its cost. Record the rejected alternatives. Set the conditions that would reverse the choice.

Step one — stating the question before the run rather than after it — has already been tested at the scale of a whole field. Kaplan and Irvin examined all 55 large NHLBI-funded randomised trials of drugs or dietary supplements for cardiovascular disease between 1970 and 2012. That period spans the years in which declaring the primary outcome in advance became the norm. They reported in 2015 that “17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000” (χ² = 12.2, df = 1, p = 0.0005). Their Results add that “Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings”.

Nothing about the treatments changed on either side of 2000. What changed was that the outcome had to be named before the data were seen. The positive-result rate went from 57% to 8%. That is the size of the effect a written question has on a result — and it is the size of the effect a question chosen afterwards has on a promoted model.

A well-recorded failure prevents the next team from repeating an expensive experiment under the same assumptions. It can also reveal that the limiting factor is a product constraint, not model capacity.

The reproduction figures point the same way from the other side. Of the 255 papers Raff tried to re-implement, replication succeeded for 22 of the 26 whose authors answered questions, and for one of the 24 whose authors did not. What a team writes down, and whether anyone can still explain it later, decides whether the work can be built on at all. Track decisions and invalidated hypotheses with the same care as promoted artifacts.