ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

AI summary

Overview

Research area: AI agents for automated research replication, specifically in the social and behavioral sciences (SBS), spanning LLM agent benchmarking, computational social science, and research integrity.

Technical level: Advanced. The paper assumes familiarity with LLM agent frameworks, tool use, sandboxed code execution, and the distinction between reproducibility and replicability in empirical research.

One-sentence scope: The paper introduces ReplicatorBench, an end-to-end benchmark of 39 human-verified replication instances with 3,128 gradable checkpoints across three stages, plus a baseline agent (ReplicatorAgent) evaluated across four underlying LLMs and different design choices for code access and programming language.

What This Paper Is About

Existing benchmarks for AI agents that assess research claims focus almost entirely on the computational side of the problem: they give the agent the paper's code and data and ask whether the outcomes can be reproduced. That setup does not reflect real replication, where a researcher must first locate or construct a new data sample, and it also lacks ground-truth diversity because it includes only fully reproducible or replicable papers. The goal of this paper is to build a benchmark that mirrors the full human replication workflow, including the retrieval of new data, the design and execution of computational experiments, and the interpretation of results, and to establish a baseline for how well current LLM agents perform on that workflow.

Key Contributions

1. ReplicatorBench, a benchmark of 39 instances, each corresponding to a paper with a human expert replication report produced in the SCORE project. It mimics the real-world setting where the new data for replication is not given, and it goes beyond a single binary outcome by decomposing replication into three stages covering a total of 3,128 gradable checkpoints that allow partial credit.
2. ReplicatorAgent, a tool-using agentic framework plus a sandbox orchestration setup. It is equipped with tools such as web search and iterative interaction with sandboxed environments, and can inspect directories, read and edit files with minimal diffs, resolve dependencies, and rerun analyses until it produces interpretable outputs or reaches a justified conclusion (criteria met or unmet).
3. A baseline evaluation of ReplicatorAgent on ReplicatorBench across four underlying LLMs and several configurations, including the choice of programming language and the level of code access, highlighting where agents succeed and fail and the root causes of failures.
4. An analysis of a practical trade-off in replication agents: executing the native non-Python code (e.g., R) supplied by original authors or human replicators versus executing Python code translated from non-Python code.

Main Findings

• Agents are strong at computation, weak at data retrieval: The findings reveal that while current LLM agents are capable of effectively designing and executing computational experiments, they struggle with retrieving new data, which is necessary to replicate a claim.
• Three-stage decomposition enables partial credit: Rather than only evaluating the final reproducibility or replicability outcome, ReplicatorBench evaluates the process through 3,128 gradable checkpoints across Extraction, Generation, and Interpretation, giving AI agents partial credits.
• Binary outcome aligned with human replication: Each instance's outcome falls into criteria met (preregistered criteria satisfied on the replication sample) or criteria unmet. Agent performance is measured by alignment between the agent's concluded outcome and the ground-truth human outcome from SCORE.
• Code-language trade-off: Using native non-Python code streamlines replication by reducing installations of language compilers and software dependencies, which resulted in fewer execution failures. The trade-off is that translating code to Python may introduce information loss that produces incorrect outputs, which in turn negatively affects the reliability of the replicability assessment.
• Data packaging to avoid error propagation: The new-sample data are curated and packaged for the agent rather than collected live from the open web, specifically to avoid error propagation in later stages.
• Specific LLM identities not reported here: The paper states that ReplicatorAgent was evaluated across four underlying LLMs, but the model names are not included in the provided content (the comparison table's "Evaluated LLMs" row is truncated).

Methodology in Plain English

The researchers start from the SCORE project, a DARPA program run in partnership with the Center for Open Science that coordinated large-scale extraction of social-behavioral claims and human reproduction and replication efforts. Because SCORE documents replication as an end-to-end process — starting with a preregistered replication plan containing criteria for replicability, Internet-retrieved data sources, and proposed methodology, then the replication study with documented code and data, and finally a report assessing replicability — the authors can score an agent at each stage rather than only at the end.

They assemble 39 instances, one per paper with a human expert replication report, and break each into three stages:

1. Extraction — gather relevant information about the focal claim and retrieve data resources for replication.
2. Generation — design, generate, and execute code for replication, testing computational capacity.
3. Interpretation — read the computational outputs and conclude whether the focal claim is replicable.

Then they build ReplicatorAgent as the baseline: a tool-using agent with web search and iterative interaction with a sandboxed environment, able to inspect directories, read and edit files with minimal diffs, resolve dependencies, and rerun analyses until it either produces interpretable output or reaches a well-justified conclusion. They run this agent across four LLMs and vary practical design choices — programming language and level of code access — and compare the agent's concluded outcome against the human ground truth while also awarding partial credit for the intermediate checkpoints. Evaluation combines automatic scoring and LLM-based scoring in a 3-stage aggregated score.

Why This Matters

Impact on research: Replication is essential to academic and research integrity but is resource-intensive, making it a natural target for autonomous agents. Because existing benchmarks assume the new data sample is already available, they cannot measure the step that human replicators actually find hardest. By introducing data retrieval as a first stage and by including both replicable and non-replicable claims, ReplicatorBench creates a testbed for evaluating an agent's ability to identify non-replicable research — a capability that outcome-only, fully-reproducible benchmarks cannot assess.

Real-world applications:

• Meta-science and research integrity offices screening large volumes of published social and behavioral science claims.
• Funding agencies and journals seeking scalable triage of claims for replication studies.
• Individual research teams wanting automated assistance reproducing or replicating prior work before building on it.
• Benchmark and agent developers needing a process-oriented, partial-credit target for building better retrieval-capable scientific agents.

Industry relevance: Organizations that apply behavioral science findings (product, policy, health, marketing) depend on whether those findings generalize to new populations and time periods. The paper's identification of data retrieval — not code execution — as the current bottleneck tells industry builders where the remaining engineering effort lies. The finding about translated versus native code is also directly relevant: automating translation to a single convenient language reduces setup failures but risks silent information loss that undermines the reliability of downstream conclusions.

Future Directions

• Improve data retrieval for replication: The paper's central failure mode is agents' inability to retrieve the new data needed to replicate a claim. Building agents that can responsibly locate or construct replication samples from open sources is the clearest next step, since the benchmark deliberately packages the data instead of requiring live collection to avoid error propagation.
• Resolve the native-versus-translated code trade-off: The trade-off between fewer execution failures from native non-Python code and potential information loss from Python translation is unresolved; studying when translation is safe is an open question.
• Broaden coverage beyond the current instances: ReplicatorBench contains 39 instances from peer-reviewed journals with SCORE human replication reports; expanding the pool of papers and claims is a natural extension.
• Extend the model and configuration sweep: The paper evaluates four LLMs and varying programming language and code access levels; the evaluated model list is not present in the provided content, and more models and access configurations remain to be tested.

Target Audience

Researchers working on LLM agents and tool-using autonomous systems; computational social scientists and meta-science researchers studying replication and reproducibility; benchmark designers interested in process-oriented, checkpoint-level evaluation rather than outcome-only scoring; and research integrity practitioners at journals, funders, and open-science organizations who want to understand both the promise and the current limits of automated replication.

Authors’ abstract

The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on the computational aspect of this task, testing agents' ability to reproduce or replicate research outcomes when having access to the code and data. This setting, while foundational, (1) fails to capture the inconsistent availability of new data for replication as opposed to reproduction, and (2) lacks ground-truth diversity by focusing only on reproducible papers, thereby failing to evaluate an agent's ability to identify non-replicable research. Furthermore, most benchmarks only evaluate outcomes rather than the replication process. In response, we introduce ReplicatorBench, an end-to-end benchmark, including human-verified replicable and non-replicable research claims in social and behavioral sciences for evaluating AI agents in research replication across three stages: (1) extraction and retrieval of replication data; (2) design and execution of computational experiments; and (3) interpretation of results, allowing a test of AI agents' capability to mimic the activities of human replicators in real world. To set a baseline of AI agents' capability, we develop ReplicatorAgent, an agentic framework equipped with necessary tools, like web search and iterative interaction with sandboxed environments, to accomplish tasks in ReplicatorBench. We evaluate ReplicatorAgent across four underlying large language models (LLMs), as well as different design choices of programming language and levels of code access. Our findings reveal that while current LLM agents are capable of effectively designing and executing computational experiments, they struggle with retrieving resources, such as new data, necessary to replicate a claim. All code and data are publicly available at https://github.com/CenterForOpenScience/llm-benchmarking.