Sci$Π$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

AI summary

Overview

• Research area: AI for science, specifically the evaluation of large language model (LLM) research agents and the scientific judgment they need to work autonomously.
• Technical level: Intermediate. The protocol is conceptually simple (a forced choice between two plans), but the paper assumes familiarity with LLM agents, benchmarking, reasoning-effort settings, and preference data.
• Scope: The paper builds and analyzes SciΠvot, a 248-case benchmark drawn from human corrections made during six months of real, multi-project LLM-agent research, and measures whether frontier models can identify the correction a senior researcher made at a pivotal decision point.

What This Paper Is About

LLM agents can now keep scientific experiments running for weeks or months, but the authors found that the real bottleneck is not execution. It is judgment: deciding whether to scale up or first check a measurement, whether a failed attempt invalidates an idea or only its implementation, and whether the current experiment actually answers the research question.

The paper turns those moments into a measurable benchmark. Senior researchers intervened repeatedly while supervising agents on 26 real projects, and each intervention became a case in which a model must choose between the agent's original plan and the human-corrected alternative, using only the project record available before the correction.

Key Contributions

1. A 248-case benchmark of "PI Judgment" built from real human corrections. The cases come from 26 projects across six scientific fields, supervised by senior researchers over roughly six months, and each pairs an original LLM plan with a human-corrected plan reconstructed from the completed project history.

2. A retrospective, evidence-grounded evaluation protocol. Evaluators see the project record frozen at a commit before the intervention, plus both plans, and must choose which is more likely to advance the scientific question while the outcome is still unknown. Choosing the human-corrected plan counts as correct.

3. A taxonomy of recurring agent failure modes. Human corrections are grouped into judgment categories: planning (123 cases), focus (122), anomaly awareness (88), adapting (65), evidence sufficiency (47), and repetition (4), with overlapping primary and secondary labels.

4. Controlled comparisons across models, reasoning effort, context access, and reconsideration. The paper reports results for five frontier models (Astra, Opus, Grok, Sol, Terra), five requested effort settings, an options-only condition without the project record, and two rounds of no-feedback reconsideration, alongside a human baseline from 13 fresh PhD students and postdocs.

Main Findings

• Models often reject the human-corrected plan. At medium reasoning effort with the project record, four of the five models select the rejected original plan more often than the reference plan; for all four, both case-level and project-cluster bootstrap intervals fall below 50% at the upper bound. Accuracy is Astra 59.5%, Opus 38.9%, Grok 33.1%, Sol 35.1%, and Terra 29.2%, giving a five-model average of 39.2% against a 50% chance level.

• Humans outperform the best model setting. The highest project-condition score across all tested models and effort settings is Astra's 63.5% at xhigh, compared with 73.4% for fresh researchers, who had an agent assistant available and no prior exposure to the projects.

• The project record lowers accuracy. At medium effort, every model scores lower with the project record than without it: 39.2% versus 55.8% on average. Paired differences are Astra −4.2 points (interval includes zero), Opus −20.0, Grok −21.0, Sol −11.7, and Terra −26.6. The authors attribute this partly to inherited explanations: in the Transformer case, Opus treats a progress note's claim that scoring faults were excluded as established evidence, supporting the original mistake rather than checking it against the code and observations.

• More deliberation does not reliably help, and can restore errors. Across models, going from low to max effort helps four models, while Sol declines from 36.1% to 29.2%. Terra rises from 29.2% to 52.0% after the first review, then falls to 42.7% after the second. Mean accuracy across the initial answer, first review, and second review is 39.2%, 44.0%, and 42.3%. The first review repairs 422 errors and introduces 302; the second repairs 264 and introduces 305.

• Deliberation can "re-trap" a model. In the Transformer case, Opus first chooses the larger model, then questions the scorer and picks the reference plan, and on its second review returns to the larger model with a premise the frozen code contradicts. The authors name this failure mode re-trapping.

• Model agreement is highly polarized on selected cases. On 49 cases, all five models choose the rejected plan in both option orders; on 16 cases, all five choose the reference plan.

• Performance varies by judgment type. Astra reaches 72.2% on anomaly-awareness cases but 56.1% on planning cases and 54.9% on focus cases. Two cases show the tension: in case 21 an untested setup calls for a pilot, yet models retain a 200-item-per-cell batch in seven of ten responses; in case 13 a full-length training run is needed, yet models retain three short gated runs in six of ten responses.

• The benchmark is diagnostic, not a general accuracy estimate. Cases were retained using 120 model responses per case if responses varied across the two plans, following challenge-set precedent. The authors state that the scores measure recognition on decisions where models disagree rather than accuracy on research decisions in general.

• Writer identity does not explain the results. Grok 4.6 wrote all released cases. In a controlled pilot on 13 cases with identical frozen commits across three writers, Grok was never the top scorer on its own cases.

Methodology in Plain English

The researchers did not invent hypothetical research dilemmas. They first ran LLM agents on real, PhD-level projects for about six months, with senior researchers reviewing work several times a day and stepping in when an agent's plan would waste time or miss the scientific point.

For each intervention, a case writer inspected the full project history and reconstructed two things: the agent's original plan at the moment before the human stepped in, and the human-corrected plan, inferred from the correction plus everything the project later did. The writer then froze the project state at a commit before the intervention.

To evaluate a model, the researchers handed it that frozen record — research idea, proposal, progress notes, code, and logs, with later history and Git history removed — plus the two plans in identical "Goal:" and "Plan:" format. The model had to pick one and explain why, with no network access and no permission to run experiments. Cases were tested in both option orders, and each case counted equally.

The team then varied three things: the requested reasoning effort (low through max), whether the project record was supplied, and whether the model got two additional chances to reconsider its answer without any feedback. A separate human baseline used 13 PhD students and postdocs who had never seen these projects and could use an agent assistant while reading the record. Confidence intervals came from bootstrapping cases (10,000 samples) and, because several cases come from the same project, from bootstrapping whole projects.

Why This Matters

The paper reframes autonomy in AI-driven science as a judgment problem rather than an execution problem. Running experiments around the clock is already feasible; deciding which experiment deserves weeks of compute is not. By converting senior researchers' corrections into a scored benchmark, the work gives labs a way to measure that gap instead of discovering it when a project stalls.

• AI research agents: Labs building autonomous research systems can use the benchmark to test whether a new agent recognizes when to check a measurement before scaling up, or when a search has hit an artifact of its own configuration.
• Scientific computing and simulation: The applied-mathematics and mechanics projects (PDE solving, symbolic regression, cloth simulation) map directly onto decisions about experiment scale, parameter fitting, and whether a proxy metric actually answers the research question.
• Biological imaging and earth sciences: The cell-lineage tracking and rainfall-downscaling projects show the same judgment failures in data-analysis pipelines, where a silent bug or a confounded measurement can invalidate weeks of work.
• Robotics and control: The humanoid motion, programmatic control, and navigation projects illustrate decisions about error budgets and whether a controller is failing for the reason the agent assumes.

Industry relevance: Organizations deploying agents on long-horizon technical work face the same supervision bottleneck: one human can only oversee so many projects. The benchmark's cases are also formatted as preference pairs, matching the training format of direct preference optimization, which points toward using human corrections as reward signal for agentic reinforcement learning. The paper's own framing ties this to recursive self-improvement, since better research agents could improve the AI systems that build them.

Future Directions

• Training on the preference pairs. The authors propose training a preference scorer on the 248 human-corrected comparisons to supply rewards for research actions in agentic reinforcement learning, then testing whether this reduces human intervention on new projects.
• Does judgment transfer across projects and domains? The benchmark covers only theoretical and computational work on CPUs and GPUs; wet-lab execution and specialized commercial software such as COMSOL are explicitly outside it, leaving open whether these judgments transfer to those settings.
• Why does the project record hurt? Every model does worse with context at medium effort. Separating retrieval, interpretation, and client-interface effects, and designing agents that check inherited claims against code and logs, is an open engineering question.
• Beyond no-feedback reconsideration. The effort and review experiments test only the controls the model interfaces expose; the authors note they do not test controlled compute scaling or verifier-guided search, and requested effort levels are not matched token budgets.

Target Audience

Researchers and engineers building LLM agents for long-horizon scientific or technical work; benchmark designers interested in expert-grounded, retrospective evaluation; AI-for-science teams at labs and companies deciding where human oversight is still required; and scientists who supervise AI agents and want a concrete account of which decisions agents get wrong. Readers new to the area will find the protocol accessible, while the statistical appendix and condition tables give specialists the detail needed to reuse the benchmark.

Authors’ abstract

Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct Sci$Π$vot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while Sci$Π$vot supports systematic evaluation of scientific judgment for LLM agents.