The Pulse
ProgramDistill Builds 4,063 Coding Tasks From 26 Working Web Applications
The arXiv abstract contains no attributable statement by a named person; no quotation was available after checking the source.
AI.info Team ·
A new benchmark called ProgramDistill turns the behavior of working web applications into coding tasks designed to test whether agents can reconstruct features from interaction rather than from source code, issue descriptions, or written instructions.
The paper, submitted to arXiv on September 16, 2026, describes a setting in which a coding agent must infer how a fully functional reference application behaves and implement those behaviors in an incomplete application. The benchmark focuses on practical software work in which the desired result may be visible in a running product but not fully specified in prose.
Working Applications as Specifications
ProgramDistill factorizes applications into features at different levels of granularity. Each feature is linked to a replayable behavior that can be executed against the reference application and checked against the result produced by an agent's implementation.
The benchmark's mine-craft-patch pipeline discovers behaviors through interaction with fully functional applications, then constructs repair tasks without human intervention. The resulting tasks are intended to provide executable tests of whether an agent has reproduced the target behavior rather than merely produced code that appears plausible.
This design changes the starting point for software-engineering evaluation. Instead of receiving a description of a bug or a requested feature, an agent must study a working product, infer the behavior that matters, and reproduce it in another application state.
More Than 4,000 Tasks
The paper reports that the pipeline discovered 1,975 replay-verified behaviors across 26 applications. From those behaviors, it constructed 4,063 tasks.
The benchmark includes tasks with different levels of difficulty. Some require restoration of individual features, while others test cumulative workflows in which several behaviors must work together. This allows the evaluation to measure not only whether an agent can implement a feature in isolation, but also whether it can preserve the dependencies needed for a longer sequence of actions.
The benchmark is evaluated across nine frontier coding agents. Its results show a clear gap between individual repair and full-application reconstruction. In the full-application setting, GPT-6 Astra achieved a 49.2% success rate on cumulative workflows, while Claude Opus 5 achieved 28.8%.
Success Falls as Workflows Become Deeper
ProgramDistill also measures performance in partial-application reconstruction as restoration depth increases. At depth one, GPT-6 Astra reached a 100% success rate, but that figure fell to 64.0% at depth eight. Claude Opus 5 declined from 96% at depth one to 32% at depth eight.
These results indicate that agents perform better when they must restore a single behavior than when they must recover a sequence of dependent behaviors. A workflow may require an agent to infer not just what an individual action does, but how that action changes the state needed by later actions.
By tying each task to behavior replayed through a working reference application, ProgramDistill provides a controlled way to vary task difficulty while retaining an executable verification process. The authors describe the benchmark as a basis for evaluating and diagnosing coding agents, as well as for future curriculum-based training.