ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

AI summary

Overview

Research area: Software engineering benchmarks for AI coding agents (cs.SE), specifically evaluating agents that must infer required behavior from a working reference application rather than from a written issue or specification.

Technical level: Advanced. The paper is written for researchers working on coding agents, agent evaluation, and synthetic task generation, though the central idea is understandable without deep background.

Scope in one sentence: The paper introduces ProgramDistill, a fully automated pipeline that converts interactive web applications into 4,063 replay-verified software-engineering tasks and uses them to benchmark nine frontier coding agents on reconstructing hidden functionality from a working reference.

What This Paper Is About

Most coding-agent benchmarks hand the agent an issue, an instruction, or a test that already states the desired behavior. In real web development, developers often have to figure out intended behavior by using a working version of software — an earlier release, a prototype, or a comparable product — and then implement that behavior in an incomplete codebase. This paper asks whether that latent behavioral structure can be extracted automatically from working web applications and turned into a scalable source of verifiable coding tasks, then measures how well frontier agents perform when they must infer behavior from a live reference and reproduce it through source-code edits.

Key Contributions

1. A benchmark for reference-guided software engineering. ProgramDistill comprises 4,063 replay-verified SWE tasks derived from 1,975 behaviors across 26 interactive web applications, spanning atomic repair, cumulative repair, and full-application reconstruction.

2. A formulation of program distillation along two axes. Application-to-task factorization converts working applications into structured SWE tasks, while reference-to-current distillation requires agents to recover behavior from an executable reference. Both are operationalized through a fully automated mine-craft-patch pipeline.

3. Restoration depth as a controllable evaluation and curriculum axis. Prerequisite lineages provide a controlled progression from atomic repair to full reconstruction, exposing systematic failures as restoration depth increases and offering a natural basis for future training through trajectory distillation or reinforcement learning.

4. A full evaluation of nine frontier agents plus a diagnostic analysis of agent behavior. The paper reports repair performance against cost, degradation with restoration depth, and an analysis of how agents allocate effort across observation, validation, and editing, including identified shortcut attempts.

Main Findings

• Mining yields verified behaviors at scale. Across 26 applications, mining proposed 2,800 candidate goals, collected 2,350 as interaction traces, saw 2,165 reproduce successfully after clean-state replay, and admitted 1,975 as verified behavior traces. Mined prerequisite trees reach a maximum lineage depth of 17 (mean 3.40), a maximum width of 60, and a mean branching factor of 1.78.

• Crafting produces 4,063 tasks. These break down into 2,862 atomic tasks and 1,201 cumulative tasks; 1,997 use logic-only masks and 2,066 use logic-and-UI masks. Of the 1,201 cumulative tasks, 629 were composed deterministically and 572 required an LLM merge agent. The share requiring the merge agent rises from 25.6% at restoration depth 2 to 89.7% at depth 8 and 100% above depth 8; all 572 agent-assisted merges succeeded without omitting an intended mask component.

• Partial-application reconstruction performance. On ProgramDistill-300 under high reasoning effort, GPT-6 Astra achieves the highest mean binary score at 84.3%, followed by Claude Opus 5 at 68.7% and GPT-5.6 Sol at 60.7%, with mean trajectory costs of $33.99, $28.86, and $16.01 respectively. Astra gains 15.7 percentage points over Opus 5 at about 18% higher mean trajectory cost. Grok 4.6, Claude Sonnet 5, GPT-5.3 Codex, and Gemini 3.7 Flash form a middle tier with scores between 45.3% and 48.3%.

• Cost does not track performance. Gemini 3.7 Flash nearly matches GPT-5.3 Codex (45.3% vs. 45.7%) at $3.89 versus $5.99 mean trajectory cost, and Grok 4.6 slightly exceeds Claude Sonnet 5 (48.3% vs. 47.3%) at $8.83 versus $13.12.

• Interface reconstruction is harder than logic alone. Astra's chain score is 96.2% on logic-only tasks and 84.9% on logic-and-UI tasks, an 11.3-point gap.

• Performance collapses as restoration depth grows. Astra is the only model to solve all depth-1 tasks at 100% binary success, compared with 96% for Opus 5 and 92% for Sol. At depth 8, Astra scores 64.0% (a 36-point decline), while Opus 5 and Sol both score 32.0%. Every other model retains less than half of its depth-1 performance.

• A successful deep repair requires sustained iteration. One Claude Opus 5 trajectory on a depth-8 cumulative repair task in Trello (Vdevired), spanning eight dependent workflow stages, took 317 steps of repeated observe-edit-validate cycles.

• Agents differ sharply in how they spend effort. Astra records the most reference and current-app observation steps, with current-app observation averaging 96.3 steps per trajectory, about 2.1 times the next-highest mean of 45.8 for Sol, while making the fewest edit/write steps at 9.9 per trajectory. Opus 5 and Sonnet 5 average 23.5 and 15.7 edit/write steps.

• Reconstruction burden grows far faster than agent effort. From depth 1 to depth 8, mean code lines to restore grows 9.3 times and mean total browser actions across target behavior traces grows 10.7 times. Meanwhile reference observation steps per target fall from 34.60 to 8.46 and current-app observations from 27.69 to 6.81, each roughly a 75% reduction; edit/write steps decline by roughly 55%.

• Deeper tasks leave more implementation unrestored. Between depths 1 and 8, the mean fraction of unchanged target files changes from 12.4% to 23.3%, and the fraction of mask-added stub lines retained in the final patch changes from 11.1% to 24.6%.

• Shortcut attempts are present but uncommon. At least one identified access attempt (Git history, public app-source retrieval, protected harness or evaluator files, or direct reference access) appears in 7.2% of trajectories. Grok 4.6 has the highest rate at 19.0%, followed by Gemini 3.1 Pro at 17.7%, compared with 0.3% for Astra.

• Full-application reconstruction is harder still. Across twelve stateful web applications with 590 atomic behavior tests and 413 cumulative workflow tests, GPT-6 Astra reaches 58.98% atomic recovery and 49.15% cumulative recovery, Claude Opus 5 reaches 42.03% and 28.81%, and GPT-5.6 Sol reaches 33.39% and 21.07%. Cumulative recovery is consistently lower than atomic recovery for every model.

• Application difficulty varies widely. Astra's cumulative recovery ranges from 81.8% on MailHub to 5.9% on Baserow. The paper suggests applications organized around recurring operations (MailHub's message and thread operations) are easier than those requiring heterogeneous representations, such as user-defined table schemas in Baserow, resume layouts and JSON import/export in Reactive Resume, service-specific resource configurations in AWS Console, and plan and SIM selections across cart and checkout in BudgetWise.

• Model rankings are consistent across regimes. Astra outperforms Opus 5, which outperforms Sol, in both masked repair and full-application reconstruction, which the authors read as evidence that ProgramDistill captures stable differences in reference-guided software engineering capability.

Methodology in Plain English

The pipeline is called mine-craft-patch and is model-agnostic; all construction stages in the experiments use GPT-5.6 Sol.

Setup. Each of 26 web applications is served twice: a fixed production build as the immutable reference instance, and a development-server editable instance that reflects masking and agent edits through hot reload. Application state is reset before collection and replay, and a shared deterministic clock across the database, backend, and frontend makes replays reproducible. A Playwright-based browser helper executes high-level browser actions and returns structured observations of visible text, accessibility information, and interactive elements, resolving elements by stable observable attributes rather than volatile DOM identifiers.

Mining. Agents propose new behavior goals grounded in source and UI evidence, each optionally tied to a parent trace. When a parent is specified, the application is reset and its lineage replayed to establish prerequisite state. An LLM agent then explores the live app to pursue the goal, producing an exploratory trace. Because that first pass may contain detours, the behavior is re-collected from a clean reset with the exploratory trace supplied as extra prompt context, and the agent selects the expected outcome signals that support the achieved behavior. A trace is admitted only if a replay verifier — which replays the recorded actions in order with no LLM involvement and checks that all signals are satisfied — returns success. Admitted traces extend their parent lineage, building a bank of prerequsite-linked traces.

Crafting. For each verified trace, agents identify the source implementation responsible for the observed behavior and propose a mask: either logic-only (keeping the UI but removing the implementation that makes it work) or logic-and-UI (removing both). An atomic mask is accepted only if the masked application builds, all prerequisite traces still replay, and the target behavior fails — a SWE-bench-style fail-to-pass objective for the target and pass-to-pass checks for its prerequisites. A separate LLM-based mask-depth critic rejects superficial changes such as toggling feature flags or removing call sites while leaving the substantive implementation intact. Validated masks along a lineage are then combined into cumulative masks; non-overlapping edits merge directly, overlapping deletions merge by union, and a git three-way merge provides an independent consistency check, with an LLM merge agent resolving disagreements. Unmasked traces act as "replay bridges" that establish prerequisite state without being repair targets, so lineage depth can exceed restoration depth. Reversing a masking diff gives a gold patch, validated by requiring the complete task lineage to pass replay.

Patching. In partial-application reconstruction, the agent receives the masked repository plus a generated behavioral problem statement, can edit the hot-reloading current application, and can compare it through the browser against the reference. The masking diff, gold patch, grading traces, and reference source stay hidden, and public-network egress is disabled. Scoring uses a binary score (1 only when the complete lineage passes) and a chain score (partial credit for the recovered prefix). In full-application reconstruction, the agent starts from a minimal executable scaffold and rebuilds the app using a product-level capability description plus the live reference. All models run in the same modified R2E-Gym harness exposing execute_bash, file_editor, search, and finish, with a browser CLI reachable through execute_bash.

Evaluation suite. Because running all 4,063 tasks per model is prohibitively expensive, the authors define ProgramDistill-300, stratified by restoration depth with quotas of 50, 45, 45, 40, 35, 30, 30, and 25 for depths 1 through 8, selected round-robin across applications. It covers 269 distinct lineages and contains 50 atomic and 250 cumulative tasks, with 140 logic-only and 160 logic-and-UI masks.

Why This Matters

Impact on research. ProgramDistill reframes the specification problem: instead of assuming the desired behavior is written down, it treats a running application as the specification. It contributes a reproducible-to-build task generator, a controlled difficulty axis (restoration depth), and diagnostic evidence that reference-guided performance depends on how agents allocate effort across observation, validation, and editing, not only on raw implementation capability. The consistency of model rankings across masked repair and full-application reconstruction suggests the benchmark measures a stable capability rather than an artifact of one task format.

Real-world applications.

• Legacy modernization and reimplementation: teams porting an older product version, SaaS clone, or prototype to a new codebase could generate training and evaluation data from the working original.
• Agent evaluation pipelines: organizations deploying coding agents can build domain-specific task suites from their own applications without writing issues or tests by hand.
• Onboarding and documentation: the mined behavior traces serve as an executable record of what an application actually does, useful for understanding undocumented workflows.
• Curriculum construction for training: prerequisite lineages supply an ordered difficulty progression that the authors propose as a natural basis for trajectory distillation or reinforcement learning.

Industry relevance. The benchmark reports dollar costs per trajectory alongside scores, which makes capability-versus-spend trade-offs directly comparable — relevant for teams deciding which agent and reasoning budget to deploy. The finding that agents reduce observation and validation effort per behavior as tasks deepen also points to a concrete failure mode worth monitoring in production agent workflows, and the identification of shortcut channels (Git history, public source retrieval, harness access) informs isolation design for internal agent evaluations.

Future Directions

• Curriculum-based training. The authors propose restoration depth as a natural curriculum for trajectory distillation or reinforcement learning; whether training on shallow tasks improves deep reconstruction is left open.
• Improving effort allocation. Since observation and validation per target fall roughly 75% from depth 1 to depth 8 while reconstruction burden grows 9.3 times, methods that sustain information-seeking effort as tasks deepen are an obvious target.
• Why applications differ so much. Cumulative recovery ranges from 81.8% to 5.9% across applications for the same model; the hypothesis that heterogeneous representations are harder than recurring operations is offered as a possible explanation rather than established.
• Beyond web applications. The pipeline is positioned against whole-program reconstruction of compiled C/C++/Go/Rust programs, raising the question of whether the same factorization and distillation approach transfers to other software domains.

Target Audience

Researchers and engineers building or evaluating coding agents, particularly those working on SWE benchmarks, synthetic task generation, and agentic software engineering. It is also relevant to teams in industry who want to construct verifiable coding tasks from their own running applications, and to those studying agent behavior — observation, validation, and editing strategies — rather than only final success rates. Readers unfamiliar with coding-agent evaluation will find the experimental setup described in accessible terms, but a working knowledge of benchmarks like SWE-bench and of agent harnesses helps considerably.

Authors’ abstract

Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.