FrontierCS 2: A Benchmark for Computer Science R&D

AI summary

Overview

Research area: Machine Learning (cs.LG) — evaluation of LLM agents on open-ended computer-science research and development.

Technical level: Advanced. The benchmark design is explained plainly, but the tasks themselves draw on FPGA circuit design, private information retrieval, video diffusion attention kernels, and item-response-theory scoring.

Scope (one sentence): The paper introduces FrontierCS 2, a 48-task, 13-domain benchmark in which an LLM agent works inside a real paper's repository with the paper's contribution deleted, and is graded on whether its final patch beats the authors' own code on held-out workloads.

What This Paper Is About

Most open-ended coding benchmarks hand an agent a problem statement, let it write a single program, and score that program once at the end. The authors argue this misses what research and development actually looks like: a researcher inherits a working codebase, changes one component, tests it on known workloads, and is judged against the paper's own implementation. FrontierCS 2 asks whether an LLM agent can do the same — make a genuine contribution to frontier computer-science R&D by beating the code that researchers released with their paper, inside that code's own repository.

Key Contributions

1. Tasks built by subtraction. Each task is made by locating the code that implements a paper's key contribution, deleting it, and leaving everything it depends on intact. The deleted code becomes a private "human reference" that sets the bar — an oracle that already runs in the repository on the paper's own workloads.

2. A judge that scores patches during the run while keeping secrets. The agent can submit a patch to its allowed files at any time and receives a development-suite score plus a sanitized message per patch. The evaluator, the held-out final suite, and the human reference stay private with the judge.

3. 48 tasks across 13 domains, spanning databases, machine learning, MLSys, HPC, programming languages, architecture, cryptography, networks, security, robotics, bioinformatics, software engineering, and theory. A private set of more than 100 additional tasks is held back for future release.

4. Two new measures: the FrontierCS Capabilities Index (FCI) and FrontierCS Difficulty Index (FDI), which place agents and tasks on a single scale using every per-metric outcome and score margin from a run, rather than one summary number.

Main Findings

• The best agent passes 58% of its runs. Claude Opus 5.5, run in Claude Code, leads with a 58% pass rate. Every other agent passes fewer than half of its runs.

• Sixteen tasks are unsolved by every agent. No run passes on any of 16 tasks, out of the 48.

• A higher score is not the same as a pass. Most runs whose final score exceeds the human reference's still miss a threshold on at least one metric or workload. Conversely, some passing runs score at or below the reference's final score, because a pass requires every metric to hold simultaneously while the final score is a single number on each task's own scale.

• The median task is roughly as hard as the strongest agent can handle. Tasks span a wider difficulty range than agents do on the shared FCI/FDI scale. Individual tasks are not ordered by FDI because each task's FDI is uncertain.

• The leader depends on the cost budget. Under test-time scaling, DeepSeek V4.1 Flash leads at the smallest budgets (with the caveat that its costs are lower bounds, so this depends on token pricing), while Claude Opus 5.5 leads at the largest budgets.

• Case study — QuarterPIR (private lookups): Muse Wonder 1.0 replaces parity values in each reply with two RAID-6-style checksums, shrinking bytes per lookup far more than any other passing run on the 67-million-entry database. Neither of the paper's measures distinguishes this: GPT-6.1 Sol scores slightly higher with a far smaller saving.

• Case study — BBQ (packet scheduler): Only Gemini 4 Argon, Claude Opus 5.5, and GPT-6.1 Sol pass. Claude Opus 5.5's circuit has the shortest clock period on the largest tree (4,096 priorities) but uses the most lookup tables among passing circuits and does not serve trees as large as BBQ was designed for. The final score ranks it first because it rewards speed and ignores circuit size.

• Case study — SVG-EAR (sparse video attention): No run passes. GPT-6 Astra comes closest — faster than the human reference on every held-out clip and, on average, closer to full attention — but fails image quality on two clips. Speed was not the obstacle; no run stayed within every image-quality band (latency at most 0.96× the reference, PSNR within 0.25 dB, SSIM within 0.01, LPIPS within 0.005).

• FCI and pass rate largely agree. Claude Opus 5.5 leads FCI, followed by GPT-6 Astra and GPT-6.1 Sol, with Step 5 Preview last; the ordering is almost the same as by pass rate. The intervals separate only agents that sit far apart on the scale.

Methodology in Plain English

Building a task. The authors take a published paper and the code its authors released. They find the code implementing the paper's central idea, delete it, and leave the surrounding repository working — the deleted functions are still declared and called, so the agent must write them. The agent may edit only the files that held the contribution. The removed code is saved as a private "human reference" patch.

Setting the bar. The reference is measured on the workloads the paper itself reported, and each metric's pass threshold is set from those measurements. A run passes only if its final submission beats the reference on every metric across every held-out workload — including held-out instances the agent never sees, such as different seeds, queries, or input graphs.

Running the agent. The agent gets a working repository and five hours. It edits code, submits patches whenever it likes, and gets back a development-suite score and a message counting how many development workloads met each threshold — without the reference score or the evaluator's raw output. On GPU tasks it may start a development GPU through the driver, while only the judge uses the GPU that measures against the human reference.

Keeping things separate. Agent and judge run in separate sandboxes linked only by a driver. The agent's sandbox reaches no network except the driver, so it cannot download the authors' code. Some tasks give the judge additional test sandboxes — for example, the Continuum task runs SWE-bench episodes against the patched vLLM serving system.

Choosing the final patch. At the end, the agent names one submission as final, and the judge scores only that one on the held-out suite. The benchmark does not pick the agent's best development-suite submission for it, because choosing the patch is treated as part of the research work. A run with no saved final submission scores 0.

Scoring agents on one scale. The FCI adapts Epoch AI's Epoch Capabilities Index, an item-response model, fitted jointly to per-metric outcomes (which metrics a run beat) and to score margins over the reference. Both come from the same measurements and share one fitted run effect. Each agent gets a strength in logits, reported at ten points per logit, with Kimi K2.7 anchored at 0 as a thirteenth, unreported agent. A task's FDI is the FCI at which an agent passes it half the time. Intervals are 90% bootstrap intervals over tasks. The fit uses 624 runs and 115 metric groups.

Why This Matters

Impact on research. The benchmark shifts evaluation from "write a program that solves a stated problem" to "improve a real system against the code its authors shipped." It also shows that a single pass/fail number hides important differences — a run that saves a few bytes and one that saves the most both count as a pass — so the authors argue benchmarks like this need to report side costs (circuit size, memory, image quality) next to the pass.

Real-world applications (drawn from the domains and tasks the benchmark covers):

• Database and query engines: approximate graph pattern mining, acyclic join processing, subgraph counting, and blockchain transaction execution.
• Systems and hardware performance: GPU kernels (FlashLib K-Means, KNN, DBSCAN, IVF-PQ), communication-avoiding matrix multiplication, parallel radix sorting, and FPGA packet schedulers.
• Security and privacy: privilege control for AI agents, regex vulnerability repair, private information retrieval, and CKKS homomorphic-encryption noise estimation.
• ML and robotics engineering: diffusion policies, MuJoCo Playground, KV-cache compaction, and multi-turn agent scheduling with KV cache time-to-live.

Industry relevance. The cost axis matters as much as the capability axis: agents are compared by pass rate against mean cost per run, and the leader changes with the budget. Teams deploying coding agents can use the pass rate, the mean margin over a human baseline, and the FCI to decide which model-plus-harness combination is worth its price, and the "score is not a pass" finding warns against optimizing a single scalar reward.

Future Directions

• Releasing the private task set. More than 100 additional tasks are held back and planned for future announcement, which will let the community test whether the current difficulty picture holds.
• Reporting side costs alongside passes. The three case studies each show a measure hiding something — bytes saved on QuarterPIR, circuit size on BBQ, per-clip image quality on SVG-EAR — pointing to benchmark designs that record these costs explicitly.
• Adopting FCI and FDI more broadly. The authors recommend these measures for benchmarks of this kind, since they use every metric outcome and score margin and come with intervals showing which differences the data actually support.
• Open question: what "beating the reference" should mean. The paper notes that some passing runs score at or below the reference's final score, and that most runs beating the reference on the combined score still fall short on some metric or workload — how to reconcile a single score with a multi-metric bar remains unresolved.

Target Audience

Researchers and engineers who build or evaluate LLM coding and research agents; benchmark designers interested in constructing tasks from real artifacts rather than problem statements; and ML systems practitioners who want a realistic read on whether current agents can improve production-grade code in databases, compilers, kernels, hardware design, security, and robotics. Readers need no background in item-response theory to follow the main results, but the case studies assume familiarity with the specific systems (PIR, FPGA priority queues, video diffusion attention).

Authors’ abstract

Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becomes the human reference that sets the bar. Earlier open-ended benchmarks give each agent a problem statement and grade one program after the run. FrontierCS 2 places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns a development-suite score but keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics. Beyond these 48 tasks, FrontierCS 2 keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.