Tampered ML: Evaluating LLM Judgment of Machine Learning Research

AI summary

Overview

Research area: AI alignment and LLM evaluation (Natural Language Processing), specifically benchmarks for judging the quality of machine learning research.

Technical level: Intermediate. The framing is conceptual rather than mathematical; readers need some familiarity with ML experimental design and benchmark evaluation.

Scope: The paper introduces Tampered ML, a benchmark of 1,858 paired experiment proposals in which one member of each pair contains a deliberately introduced methodological flaw, and reports how well frontier LLMs can identify the sounder proposal.

What This Paper Is About

Automating parts of alignment research requires models that can be trusted on tasks that are hard for humans to supervise, such as spotting flaws in a research proposal or judging whether a proposed experiment actually supports its stated claim. The paper asks whether current LLMs can make that kind of methodological judgment. It builds a benchmark where two versions of an experiment proposal differ only by a targeted flaw, and tests whether models can tell which one is sounder.

Key Contributions

1. A new benchmark, Tampered ML, consisting of 1,858 item-pairs. Each pair contains an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual version that introduces a targeted methodological flaw.
2. A two-order evaluation protocol. Models must identify the more methodologically sound proposal in both presentation orders, which exposes position bias rather than allowing a model to score well by consistently picking one slot.
3. A human validation study on a stratified sample of items, estimating that 86% of items are valid (95% CI: 81–91%), i.e., that the intended proposal really is the sounder one and the edit really is a flaw.
4. A baseline evaluation of frontier models, showing that they exceed chance but fall well short of reliability, and characterizing how they fail rather than only how often.

Main Findings

• Above chance but not reliable: Frontier models perform better than chance on the benchmark but remain far from dependable. The best figure reported in the abstract is Opus 5 at 77.9% accuracy.
• Strong order sensitivity: Model judgments are highly sensitive to the order in which the two proposals are presented, meaning performance depends partly on presentation rather than purely on methodological content.
• Systematic preference for surface plausibility: Models tend to favor the flawed designs when those designs appear more specific, more transparent, or more conventional, suggesting they are swayed by stylistic cues rather than methodological validity.
• Benchmark quality is measured, not assumed: The human validation study supports the benchmark's construct validity at an estimated 86% of items, with a reported confidence interval of 81–91%.
• Details not in the abstract: The abstract does not report which other models were tested beyond the Opus 5 figure, per-flaw-type breakdowns, or how accuracy was aggregated across the two presentation orders.

Methodology in Plain English

The researchers started from published machine learning papers and reconstructed the experiment each one describes as a standalone proposal. For each proposal they created a near-copy that differs by a small, deliberate edit introducing a specific methodological flaw, so the two versions are close enough that telling them apart requires actual methodological reasoning rather than spotting obvious errors. Models are shown both versions and asked which is more methodologically sound; because each pair is presented in both orders, a model that simply prefers the first or second option will not score well. Finally, humans checked a stratified sample of the pairs to confirm that the unedited proposal genuinely is the sounder one and that the edit genuinely is a flaw, giving an estimate of how much of the benchmark is trustworthy. The abstract does not specify the taxonomy of flaw types, the number of models evaluated, or the exact scoring procedure beyond accuracy.

Why This Matters

The paper targets a specific prerequisite for automating alignment research: the ability to evaluate research quality on tasks where humans cannot easily check the model's work. Benchmarks that measure factual knowledge or reasoning accuracy do not capture this kind of judgment, so a benchmark built around minimally edited counterfactuals isolates methodological discernment from surface-level plausibility. The finding that models are drawn to designs that look specific, transparent, or conventional is a caution for anyone planning to rely on LLM critique.

Real-world applications:

• Peer review and internal review triage: flagging submissions or internal proposals whose experimental design does not support their claims.
• Red-teaming research plans: having a model adversarially check an experiment protocol before compute is spent.
• Evaluating and monitoring AI research agents: measuring whether systems that propose or execute experiments can also critique them.
• Model selection and procurement: giving organizations a signal for whether a model is suitable as a research assistant rather than only as a writer or coder.

Industry relevance is direct for AI labs and research-heavy organizations deploying LLMs as reviewers or R&D assistants: the results suggest current models can catch many flaws but are unreliable enough, and biased enough toward confident-sounding designs, that unchecked use would let bad methodology through.

Future Directions

• Reducing order sensitivity: understanding whether position bias is a prompting artifact or a deeper property of model judgment, and testing mitigations.
• Explaining the surface-cue bias: investigating why specificity, transparency, and conventionality are rewarded over validity, and whether training or prompting can shift this.
• Broadening the benchmark: extending beyond the initial set of papers and flaw types, and reporting per-flaw-category performance so failure modes are diagnosable.
• Linking benchmark scores to real utility: establishing whether performance on Tampered ML predicts usefulness as an actual research critic, and whether the benchmark can serve as a training or reward signal.

Target Audience

Alignment and AI-safety researchers working on scalable oversight and the automation of research; benchmark and evaluation researchers interested in construct-valid, bias-resistant evaluations; AI lab safety and red-team staff assessing LLM research assistants; and meta-science or peer-review researchers interested in machine-assisted methodological critique.

Authors’ abstract

One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim. To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw. Models must identify the more methodologically sound proposal in both presentation orders. A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81-91%). Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional.