Automated Reproducibility Has a Problem Statement Problem

AI summary

Overview

• Research area: AI methodology and metascience — specifically the automation of scientific reproducibility using large language models, with a formal framing grounded in the scientific method.
• Technical level: Intermediate. The paper is conceptual (a problem definition plus a proof-of-concept extraction pipeline), but it assumes familiarity with reproduction/replication terminology, LLM prompting, and text-similarity metrics such as Levenshtein distance.
• Scope (one sentence): The paper proposes a general, paper-independent problem statement for automated reproducibility, and tests whether an LLM can automatically extract that structure (hypotheses, experiments, interpretations) from 20 published empirical AI studies, as judged by the original authors.

What This Paper Is About

Efforts to automate the reproduction of scientific studies have used many different evaluation metrics — per-paper rubrics, data-availability-dependent scores, and custom alignment metrics — which makes results across systems hard to compare and leaves the underlying problem formally undefined. The authors argue that what is missing is a single, generalisable problem statement: a description of what "reproducing a study" actually means, applicable to any empirical study. They propose representing a study as a graph of hypotheses, experiments, outcomes, analyses, and interpretations, then test whether an LLM can extract that graph automatically from PDFs and whether the original authors agree the extraction is representative of their work.

Key Contributions

1. A formal problem definition for reproducibility that generalises across empirical AI studies, intended to enable generalised (rather than per-instance, handcrafted) metrics.
2. A proof-of-concept extraction method that automatically derives the problem statement of reproducibility for an empirical AI study from its publication, using Google Gemini 2.5 Pro with few-shot prompting at temperature t = 0.0.
3. A dataset of 20 published papers with the LLM's extracted representations, author corrections, and author evaluations on 5-point and 7-point Likert scales, released via a GitHub repository.
4. An empirical evaluation by the original authors, quantifying which parts of the representation were captured well and which were not.

Main Findings

• All elements captured in most studies: For 75.00% of the studies, the method correctly captured all elements — all hypotheses and experiments were represented at least to some degree.
• Hypotheses were often edited but only slightly: Authors wished to adapt the extracted hypothesis in 65.52% of cases (19 instances), but the average correction was 43 characters, corresponding to 14.90% of the statement on average.
• Interpretations were more accurate than hypotheses: 24.32% of interpretation statements (9 instances) were edited, with an average edit distance of 35 characters, or 4.79% per statement. The authors attribute this to heavier quoting and paraphrasing of the original paper.
• Missing hypotheses in six cases: In six cases the method failed to capture the full set of hypotheses (while still capturing one or more correctly). For Bosman et al. (2025), 7 of 9 hypotheses were captured; for Benjamins et al. (2025), 1 of 2.
• Missing experiments in two cases: The method failed to capture an entire experiment for Benjamins et al. (2025) and Berger et al. (2025).
• Token count is an imperfect explanation: The two papers with missing hypotheses had substantially larger token counts, but Skaf et al. (2025) and Eimer et al. (2023) — the largest studies in the dataset by token count (11 095 and 11 869 tokens) — showed no missing experiments, indicating token count alone does not explain the failures.
• Experiment details were the weakest area: Experiment results were missing or incorrect in 69.63% of cases (1103 instances), experiment metrics needed correction in 46.88% (15 instances), experiment strategy in 31.25% (10 instances), and experiment statistics in 28.12% (9 instances). The authors note the general spirit and goal of experiments was still captured.
• Links were mostly correct: Experiment-to-hypothesis links were wrong in 18.75% of cases (6), interpretation-to-experiment links in 5.41% (2), and interpretation-to-hypothesis links in 0.00% (0).
• Visual results are a persistent weakness: The LLM often defaulted to extracting results from text rather than figures, especially when images were not vectorised or rasterised within the PDF, making extraction of visually presented outcomes unstable.
• Merging of experiments noted as minor: For Eimer et al. (2023), the first author reported that the approach merged three experiments into one, which the author regarded as a minor issue given the problem statement.
• Latent hypotheses are post-hoc: Because empirical AI studies rarely state explicit hypotheses (authors state research questions and findings instead), the extracted hypothesis is treated as a post-hoc hypothesis: the expected outcome of reproduction is drawing the same conclusions as the original investigators.
• Levenshtein caveat: Edit distance was calculated with the Levenshtein distance and rounded up for averages; it does not capture semantic changes to statements.

Methodology in Plain English

The authors start from a definition of reproducibility adapted from Gundersen (2021): based on the documentation provided by the original authors, independent investigators can conduct similar experiments whose outcomes can be analysed and interpreted to support the original hypotheses. They turn this into a graph structure — a study contains one or more hypotheses; hypotheses are linked to experiments; each experiment has input data and a method/strategy producing outcomes; outcomes are analysed with metrics and statistics or tests; interpretations of those analyses support the hypotheses. They deliberately allow a flexible structure (one experiment linked to multiple hypotheses, multiple analyses per outcome, interpretations spanning experiments), with interpretations treated as relatively static compared to the flexible reading a human replicator might apply.

To test the structure, they built a relatively simple prompt with few-shot examples and hints about which sections and keywords might contain the target information, and presented it along with each paper's PDF to Google Gemini 2.5 Pro at temperature t = 0.0. They iteratively queried the model on three candidate publications (Dettmer et al. 2024; Gundersen et al. 2025; Snelleman et al. 2024) to improve the prompt; the authors note this is not few-shot learning because the model received no feedback. They then ran the extraction on the 20 publications listed in Table 1 (token counts ranging from 1 291 for Fehring et al. 2025 to 11 869 for Eimer et al. 2023).

For evaluation, they asked the first author of each paper to correct mistakes in phrasings, links between hypotheses, experiments and interpretations, and experiment details (measured outcomes, applied statistics, strategies, and how tests assess outcomes), and to rate each element on a 5-point Likert scale (hypotheses used a 7-point scale). The authors of each article were informed about the problem formulation and the extraction objective beforehand. The full prompt, outputs, review form and outcomes are in the project's GitHub repository.

Why This Matters

Impact on research. Automated reproducibility work has been fragmented, with each system inventing its own rubric, metric, or score, making cross-system comparison difficult. This paper supplies a shared problem definition and a unified graph structure, so different automated systems can be measured on the same axis: how much of the graph they can reproduce to uphold the authors' hypotheses. It also flags an uncomfortable finding for science as a whole — studies that do not state hypotheses or research questions clearly are harder to capture, whether by humans or machines.

Real-world applications:

• Benchmarking and comparing automated reproduction agents on a common, generalisable metric rather than per-paper rubrics.
• Screening published literature at scale to check whether claims are backed by extractable hypotheses, experiments and results.
• Producing structured, machine-readable summaries of empirical AI papers as input to downstream agents (the authors explicitly envision distributing structured tasks to multiagent systems).
• Review and editorial tooling that flags publications whose hypotheses or experimental details are not clearly stated.

Industry relevance. Organisations that depend on reproducing third-party machine learning results — model evaluation teams, internal research groups, and teams doing due diligence on published or vendor-reported benchmarks — face the cost of manual reproduction. A structured, extractable representation of a study's claims and evidence would allow automated systems to identify, at scale, which claims are and are not supported by documented experiments.

Future Directions

• Improve the extraction method with extensive prompt engineering or post-training of LLMs on this task, using the published dataset as training data.
• Handle multimodal results, giving clear instructions to the LLM on how to read figures (box-plots, line graphs, histograms) directly rather than falling back on the surrounding text.
• Produce more structured and fine-grained output, since the current approach missed details such as experiment results, metrics, statistics and strategies in a substantial fraction of cases.
• Extend and stress-test generalisability beyond empirical AI studies, and test whether token count/length effects on accuracy can be mitigated.
• Use the formulation as a base for larger multiagent systems, where the structured extraction supplies sub-tasks to collaborating agents — a gap the authors identify in prior single- and dual-agent approaches.

Target Audience

Researchers and practitioners in AI methodology, metascience and reproducibility; developers building LLM-based or multiagent systems for automated scientific reproduction; and reviewers, journal editors and research evaluators interested in defining what counts as a successful reproduction. The paper is most valuable to readers who already understand LLM prompting and reproducibility terminology, and to those seeking a shared problem definition to evaluate automated reproduction systems against.

Authors’ abstract

Background. Reproducibility is essential to the scientific method, but reproduction is often a laborious task. Recent works have attempted to automate this process and relieve researchers of this workload. However, due to varying definitions of reproducibility, a clear problem statement is missing. Objectives. Create a generalisable problem statement, applicable to any empirical study. We hypothesise that we can represent any empirical study using a structure based on the scientific method and that this representation can be automatically extracted from any publication, and captures the essence of the study. Methods. We apply our definition of reproducibility as a problem statement for the automatisation of reproducibility by automatically extracting the hypotheses, experiments and interpretations of 20 studies and assess the quality based on assessments by the original authors of each study. Results. We create a dataset representing the reproducibility problem, consisting of the representation of 20 studies. The majority of author feedback is positive, for all parts of the representation. In a few cases, our method failed to capture all elements of the study. We also find room for improvement at capturing specific details, such as results of experiments. Conclusions. We conclude that our formulation of the problem is able to capture the concept of reproducibility in empirical AI studies across a wide range of subfields. Authors of original publications generally agree that the produced structure is representative of their work; we believe improvements can be achieved by applying our findings to create a more structured and fine-grained output in future work.