$Φ$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

AI summary

Overview

Research area: Evaluation of large language models on systems-level software engineering — specifically, the GPU kernels, training frameworks, and inference serving stacks that power LLMs themselves.

Technical level: Advanced. The paper assumes familiarity with GPU kernel programming, distributed training, inference serving systems, and LLM agent evaluation. The high-level conclusions are accessible to a general technical reader, but the task definitions and metrics are aimed at systems and ML researchers.

Scope: The paper introduces Φ-Bench, an 85-task benchmark spanning nine LLM infrastructure domains and three task formats of increasing open-endedness, and uses it to measure how well eight frontier LLMs can engineer the infrastructure that runs them.

What This Paper Is About

Existing benchmarks for "LLMs doing systems work" mostly hand the model an isolated GPU kernel, a fixed function signature, and a predefined optimization target. That is not what infrastructure engineering actually looks like. Real work involves reading a large unfamiliar codebase, finding the bottleneck, forming hypotheses, making changes across multiple files, profiling, and iterating under noisy measurements. Φ-Bench is built to test that harder, longer-horizon, open-ended version of the problem — and to measure how far current frontier models are from doing it reliably.

Key Contributions

1. A new benchmark spanning the LLM infrastructure stack. Φ-Bench contains 85 tasks across nine infrastructure categories (training, inference and serving, compression, kernels, I/O, hardware and edge, data infrastructure, system optimization, system assurance), with tasks grounded in real repositories and published systems research rather than synthetic prompts.

2. A three-tier task format ladder. Tasks come in three formats with increasing scope and open-endedness: Kernel Function Completion (55 tasks), Long-Horizon Implementation (20 tasks), and End-to-End Optimization (10 tasks). Each level removes more scaffolding — from a specified single-file function to a bare system-level objective with full repository edit access.

3. A scalable, taxonomy-guided construction pipeline. The authors assembled a three-level coverage taxonomy from 2,260 systems papers and 1,852 repository artifacts, yielding 410 fine-grained tags clustered into 62 middle-level topics and nine top-level categories. Tasks are synthesized through three routes: reconstructing real pull requests and issues, an agent loop that mines repositories and auto-generates tests for uncovered execution paths, and expert curation for problems with no repository history.

4. A systematic evaluation with trajectory-level analysis. Eight frontier models were evaluated, and the paper goes beyond leaderboard numbers to analyze error modes, iteration behavior, reasoning-budget sensitivity, and qualitative differences in how strong and weak models conduct optimization.

Main Findings

• Ceiling is low. Claude Opus 5 leads at 36.53% overall, followed by Kimi K3 (28.12%) and Qwen3.8 Max (27.73%). Even with full repository access and visible test cases, the best model scores roughly one-third of the maximum.

• Performance is uneven across domains. Claude Opus 5 leads in five of nine categories, but no model is uniformly strong. Kimi K3 is best on Inference & Serving and System Optimization; Qwen3.7 Max on Hardware & Edge; GLM 5.2 on System Assurance.

• Hardware & Edge is the weakest area by far. The best score in that category is 5.4%, indicating frontier models lack meaningful understanding of hardware-level infrastructure concerns.

• Long-horizon repository work is harder than kernel work. Scores on LHI are consistently lower than on KFC for every model. Claude Opus 5 achieves 37.16% on KFC but 21.60% on LHI, while scoring 62.94% on E2EO — a format where the small task count and objective-driven scoring may make partial credit easier to earn.

• Iterative refinement separates strong from weak models. On a nanoGPT mixture-of-experts training task scored by validation bits-per-byte, Opus 5 started strong and improved steadily; Qwen3.8 Max and Kimi K3 started poorly but converged well; DeepSeek V4 Pro, Qwen3.7 Max, GLM 5.2, and GPT 5.6 Sol plateaued and never reached competitive BPB.

• More reasoning budget does not reliably help. Across 20 LHI tasks, all three tested models peaked at maximum reasoning effort, but gains were non-monotonic. GPT 5.6 Sol was notably unstable at intermediate effort levels. Kimi K3 lost about 45% of its score at low effort, meaning its strong showing depends heavily on large test-time compute.

• Error volume correlates with success, not failure. The top models produced more errors, reflecting more rounds of trial, diagnosis, and correction. Weaker models made fewer errors, apparently because they gave up earlier or settled for simpler solutions. Python runtime errors dominated for most models; Opus 5 had a notably smaller share of these, concentrating its errors in CUDA execution instead — suggesting better first-pass repository-level Python correctness.

• Cheating was effectively suppressed. Soft network disconnection (hijacking pip and repository URLs) plus explicit prompting limited violations to three benign PyTorch-website requests from DeepSeek V4 Pro, confirmed by a proctor agent as non-hacking.

• Qualitative behaviors distinguish strong models. In a case study, Opus 5 ran cheap local validation experiments before spending a submission, controlled for measurement noise and compilation-cache artifacts before attributing performance changes, and maintained a coherent multi-round optimization plan. Weaker models used formal submissions as simultaneous debugging and hypothesis testing, changed multiple variables at once, or mistook noise for signal.

Methodology in Plain English

The authors started by asking what LLM infrastructure engineering actually consists of. They gathered systems papers from top conferences (2023–2026) and issues and pull requests from widely used LLM infrastructure repositories, then used an LLM to filter for relevant, nontrivial work. That produced 4,112 sources, which they organized into a three-level topic taxonomy to make sure the benchmark would cover the whole field rather than a narrow slice.

Tasks were then built three ways. The most grounded approach takes a real repository change — the code before the change becomes the starting point and the code after becomes the reference solution; existing unit tests become the grading harness. A second approach uses an agent to scan repositories for valuable implementation sites and automatically generate tests, tracking which code paths existing tests cover and adding cases for the uncovered ones. A third approach has human experts design tasks from influential papers when no repository history exists.

Each task ships with a specification, the full repository, executable workloads, and an evaluation harness that the agent cannot see. Grading depends on the task type: performance-oriented tasks measure the candidate's speedup ratio against a baseline using paired measurements, then convert it to a reward that is zero if the candidate merely matches the reference and rises logarithmically toward 1.0; functional tasks get binary credit (build, pass all tests, respect edit constraints). Correctness gates everything — no reward is awarded for code that does not work.

The authors evaluated eight models with high reasoning settings and generous context, running each task on 8 CPU cores, 32 GiB of memory, and one NVIDIA H20 GPU, allowing 16 submissions for the longer tasks and taking the best.

Why This Matters

Impact on research. The paper reframes LLM-for-systems evaluation away from isolated kernel puzzles toward the messy, open-ended reality of infrastructure work: navigating unfamiliar codebases, profiling, forming and testing hypotheses under measurement noise, and maintaining a coherent multi-round strategy. It also supplies a reusable construction pipeline — taxonomy plus agent-driven synthesis — that other benchmark builders can adopt, and it provides the first systematic evidence that "more reasoning tokens" is not a reliable lever for this class of task.

Real-world applications:

• Automating performance engineering. Teams maintaining training and inference stacks could use models to identify bottlenecks and propose optimizations, provided the reliability gap shown here is closed.
• GPU kernel development. Kernel-level tasks (the weakest area alongside Hardware & Edge) map directly to writing and tuning CUDA/Triton kernels for new hardware or new model architectures.
• Serving system optimization. Inference and serving tasks correspond to improving throughput and latency for deployed LLM endpoints, where small efficiency gains translate into large cost savings.
• Repository maintenance and feature work. The LHI format mirrors real feature requests, suggesting a path toward models that can contribute nontrivial multi-file changes to systems projects.

Industry relevance. Infrastructure cost is a dominant expense for any organization training or serving large models. If models can reliably assist with even a fraction of infrastructure optimization, the payoff is measured in GPU-hours and dollars. The low absolute scores and the Hardware & Edge weakness indicate that current deployments should treat model-generated infrastructure changes as proposals requiring rigorous human review, not as autonomous improvements. The paper's emphasis on anti-cheating infrastructure — network isolation, rule-based scanning, and an inspecting proctor agent — is also directly relevant to anyone running agentic coding evaluations in production.

Future Directions

• Closing the long-horizon execution gap. LHI scores lag KFC scores across every model, so the central open problem is repository-scale reasoning: understanding module interactions, planning multi-file changes, and debugging across them.

• Improving hardware understanding. With a 5.4% ceiling on Hardware & Edge, it is unclear whether this reflects a data gap (few hardware-level training examples), a reasoning gap, or an interaction gap where models cannot connect high-level objectives to hardware constraints.

• Making reasoning budget useful. Since performance did not improve monotonically with effort and one model depended heavily on maximum compute, research is needed on allocating reasoning adaptively rather than simply spending more of it.

• Teaching experimental discipline. The case study suggests that the distinguishing skills are methodological: cheap local validation, variable and noise control, and resisting premature causal attribution. Whether these behaviors can be trained or scaffolded deliberately is an open question.

• Extending and hardening the benchmark. Only 10 E2EO tasks exist, and the format where Opus 5 scored highest (62.94%) is also the smallest and least precisely graded. Expanding that tier and strengthening anti-hacking detection for more capable future models would sharpen the signal.

Target Audience

This paper is most valuable to:

• ML systems and infrastructure researchers evaluating whether LLM agents can contribute to training and serving stacks.
• Benchmark and evaluation researchers interested in task synthesis pipelines, agentic evaluation harnesses, and anti-cheating design.
• LLM agent developers who want to understand which agent behaviors — iterative refinement, controlled experimentation, cautious attribution — actually separate strong from weak performance.
• Practitioners running LLM infrastructure at scale who need a realistic read on current model capabilities before deploying them on optimization work.
• General ML researchers seeking a clear, quantitative picture of where frontier models still fall short on long-horizon engineering.

Authors’ abstract

Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present $Φ$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories, $Φ$-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.