FrontierCS: Evolving Challenges for Evolving Intelligence

AI summary

Overview

Research area: Evaluation of large language models on code generation and open-ended computer-science reasoning (Machine Learning, cs.LG; arXiv:2512.15699v1, published 17 December 2025).

Technical level: Advanced. The paper assumes familiarity with LLM benchmarking, competitive programming, NP-hard optimization, and systems research workflows.

Scope (one sentence): FrontierCS is a benchmark of 156 open-ended computer-science problems, split into an algorithmic track and a research track, where no solution is known to be optimal but every submitted program can be scored deterministically, and it is used to compare frontier reasoning models against human experts.

What This Paper Is About

Most code and reasoning benchmarks use closed-form tasks with a single optimal answer and a pass-or-fail criterion, which top models now largely saturate. FrontierCS targets problems whose global optimum is unknown or practically unattainable, yet whose solutions can be run as programs, checked for validity, and assigned an objective score. The goal is to provide an expert-curated, cross-domain benchmark that measures whether models can discover high-quality algorithms and system designs rather than merely producing workable code.

Key Contributions

1. A 156-problem open-ended benchmark. FrontierCS combines 107 algorithmic problems and 49 research problems. The algorithmic track contains 29 optimization, 27 constructive, and 51 interactive problems. The research track spans Operating Systems (8), High-Performance Computing (19), Artificial Intelligence (6), Databases (7), Programming Languages (5), and Security (4).
2. A three-stage expert curation pipeline (Proposal, Implementation, Review). Algorithmic problems are proposed by experts with qualifications equivalent to ICPC World Finalists and rewritten from contest problems into partially scored, open-ended variants. Research problems are proposed by CS PhD students and shipped with a reproducible environment containing resources, set_up_env.sh, evaluate.sh, and a README. Review requires that no single optimal solution exists, that partial scoring meaningfully reflects progress, and that the environment is deterministic and isolated.
3. Deterministic evaluators with a 0-100 score, using human references and trivial baselines. Each problem includes a generator for test cases, a deterministic verifier, an expert reference solution, baseline implementations, and an evaluation harness. Runtime and memory act as feasibility constraints rather than scoring components, so higher scores reflect better strategies rather than more compute. No LLM is used as a judge.
4. A benchmark update policy that preserves comparability. FrontierCS can evolve by (1) adding new tasks, (2) increasing difficulty of existing tasks without changing the written problem statement, by tightening time/memory budgets, changing workloads or datasets, or adjusting objectives, and (3) refining human reference solutions and evaluation thresholds.

Main Findings

• Frontier models lag far behind human experts on algorithmic problems. Human experts reach Score@1 of 95.41, while the reported models reach 29.37 (Gemini 3.0 Pro), 14.95 (Claude Opus 4.5), 13.67 (Grok 4), 12.65 (DeepSeek 3.2), 12.53 (Gemini 2.5 Pro), 11.80 (GPT 5.1 Thinking), 10.87 (GPT 5 Thinking), 6.91 (Claude Opus 4.1), and 5.84 (Claude Sonnet 4.5).
• More sampling helps but does not close the gap. On the algorithmic track, Score@5 improves over Score@1 by 6.40 to 22.69 points across models (for example, Gemini 3.0 Pro: Score@1 29.37, Score@5 52.06). On the research track, Score@5 improves over Score@1 by 6.72 to 21.04 points.
• Research-track scores are higher than algorithmic-track scores. Claude Opus 4.5 has the best one-shot research result (Score@1 29.40) and GPT 5.1 Thinking leads under multi-sample evaluation (Score@5 47.21).
• Pass rates exceed raw scores, indicating workable but under-optimized solutions. On the algorithmic track, Pass@1 ranges from 28.04% (Claude Sonnet 4.5) to 65.42% (Gemini 3.0 Pro), while scores stay low.
• Higher reasoning effort does not monotonically help. For GPT 5 Thinking, average score rises from 7.903 at low effort (4,389 average tokens) to 15.336 at medium effort (11,554 tokens), then falls to 12.626 at high effort (19,763 tokens). GPT 5.1 at high effort averages 20,402 tokens and 12.508.
• A misleading micro-optimization trap. On Polyomino Packing, GPT 5-Thinking produced invalid code in about 30% of attempts and scored only 20-70 in the remaining 70%, because it used the output transformation list as its internal data structure. Adding one instruction to use a 2D array and convert only at the end dropped the zero-score rate to about 10% and produced scores of 80-85 in nearly 80% of cases.
• Human experts beat GPT-5 on the illustrative polyomino case. Both produced valid packings, but the human achieved 87% density versus 47% for GPT-5.
• A research-versus-engineering trade-off in Claude models. Claude Sonnet 4.5 often produced workable solutions that compiled and ran but fell below the trivial baseline on the algorithmic track, earning zero. Claude Opus 4.5's workable solutions performed relatively well on research problems, for example earning around 50 points on Symbolic Regression problems that require correctly configuring the PySR package.
• Worked example, Problem 1 (World Map, adapted from IOI 2025): on an instance with N=5 countries and M=6 adjacency requirements, the human expert achieved K=7 while GPT 5 produced K=15; the only known solution achieves R' = 1.5.
• Worked example, Problem 2 (Treasure Packing): with C=12 categories and 1-second and 1024 MB limits, GPT 5's greedy plus branch-and-bound approach scored 74 points against a human expert score of 100.

Methodology in Plain English

The authors collected 156 problems and turned each into an executable challenge: the model receives a text-only problem statement plus any required I/O or API stubs, and must write a self-contained program. A deterministic evaluator runs that program on generated instances and assigns a score between 0 and 100. A solution scoring at or below the trivial baseline gets zero; one that matches the human expert reference or a known nontrivial bound gets full credit; intermediate scores are mapped by problem-specific rules. Because each specification is parametric, one problem can generate many instances of varying difficulty, which reduces leakage and overfitting.

Evaluation was single-round: the model's output is final, with no code execution, unit-test feedback, editor, Python environment, or external tools, and the request timeout is 20 minutes. GPT-5, GPT-5.1 Thinking, Grok 4, and Claude Opus 4.5 used reasoning_effort = high; Claude Opus 4.1 and Sonnet 4.5 used max_tokens = 32,000 and reasoning_budget = 20,000; Gemini 2.5 Pro and Gemini 3.0 Pro used thinking_budget = -1. The paper states that nine frontier models were tested, though the setup text names eight and the result tables also report GPT 5.1 Thinking and Grok 4 (listed as Grok 4 Fast in the research table).

Metrics are Score@1, Avg@5, Score@5, Pass@1, and Pass@5. Score@k is the maximum score across k trials; Avg@k is the average; Pass@k is the fraction of problems with a non-zero score, meaning the solution exceeds the most trivial baseline. The research track runs on infrastructure built with SkyPilot, scaling from a single node to distributed cloud clusters and using spot instances and region arbitrage to keep costs low.

Why This Matters

Impact on research. Existing benchmarks are reaching saturation, or they cover a narrow slice of problem types authored by a small set of recurring setters. FrontierCS provides continuous, objectively verifiable scores on problems without known optima, which makes it usable not just for evaluation but also for training and ablation, since the scoring functions can serve as reward signals for reinforcement learning or self-play.

Real-world applications.

• Compiler and systems research, including OS, databases, programming languages, and security workflows.
• High-performance computing and GPU kernel optimization, where throughput, latency, and memory trade-offs dominate.
• Cloud infrastructure tasks such as multi-region spot-instance scheduling and cloud transfer path optimization.
• Accuracy-constrained engineering problems, such as designing a vector database index that minimizes query latency while maintaining at least 95% accuracy.

Industry relevance. The results indicate that models which are strong on closed-form software engineering, such as the SWE-bench Verified performance noted for Claude models, may still fail to produce high-performance solutions for open-ended algorithmic problems. Teams deciding where LLM-generated code is safe to deploy, and how to design agents that iterate on their own outputs, can use these findings to calibrate expectations.

Future Directions

• Agentic and iterative evaluation. The paper leaves multi-round, agentic, tool-call-assisted, and evolve-style frameworks to future work, since all reported results are single-round with no code execution or feedback.
• Reconciling high reasoning budgets with performance. Because GPT 5 Thinking drops from 15.336 at medium effort to 12.626 at high effort, the authors call for more effective ways to use high reasoning effort on complex open-ended problems.
• Refining thresholds as models improve. The third update mechanism requires periodically rebuilding human reference solutions, scoring rubrics, and evaluation thresholds so that capable models remain distinguishable.
• Broader and deeper problem coverage. The update policy allows adding new tasks and increasing difficulty of existing ones without rewriting problem statements, which raises the question of how far the benchmark can track advancing capabilities before thresholds need replacing.

Target Audience

Researchers and engineers working on LLM evaluation, reasoning, and code generation; benchmark designers who need partially scored, non-binary evaluation methods; competitive programming and algorithms experts interested in open-ended problem formulation; and systems researchers in operating systems, HPC, databases, programming languages, AI, and security who want tasks drawn from real research workflows.

Authors’ abstract

We introduce FrontierCS, a benchmark of 156 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts on both the algorithmic and research tracks, that increasing reasoning budgets alone does not close this gap, and that models often over-optimize for generating merely workable code instead of discovering high-quality algorithms and system designs.