EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

AI summary

Overview

Research area: Benchmarks and evaluation for AI coding agents, specifically web application development (AI/software engineering).

Technical level: Intermediate. The abstract assumes familiarity with coding agents, benchmarks, and web development workflows, but the core ideas are explained conceptually.

Scope: This paper introduces EvoWeb, a benchmark of 320 cumulative requests across 32 web applications designed to test whether coding agents can correctly evolve an existing codebase through successive, dependency-linked feature requests.

What This Paper Is About

Most web-generation benchmarks ask an agent to build a site from a fixed specification, but real development happens through a stream of follow-up requests that extend and modify existing code. This mismatch means current benchmarks may not capture whether an agent can keep a working application healthy while adding features on top of it. The paper's goal is to measure agent reliability across a long sequence of dependent requests rather than on a single isolated task.

Key Contributions

1. EvoWeb, a cumulative-request benchmark: 320 requests spread over 32 web applications, grounded in real-world product features rather than synthetic specifications.
2. Dependency-structured task design: Requests are organized by functional dependency graphs, so later requests build on earlier ones instead of standing alone.
3. Automated multimodal evaluation in a live browser: A multimodal web agent tests each intermediate version of the application, checking three things — the newly requested functionality, integration across requests, and previously requested functionality that should still work.
4. A trajectory-based reliability metric: Instead of a single fixed stopping criterion, the benchmark tracks performance across a range of user-tolerance thresholds, characterizing how reliability holds up throughout development.

Main Findings

• Models are distinguishable: Experiments across open- and closed-source frontier models show that EvoWeb surfaces clear performance differences between them, indicating the benchmark has enough difficulty to separate systems.
• Cross-request integration is the hardest part: The abstract identifies integration across successive requests — making new features work together with earlier ones — as a key challenge, distinct from simply producing the requested code.
• Dependency depth predicts difficulty: Later requests that have deeper dependency chains or a larger number of dependencies are solved less accurately.
• Reliability degrades over time: Runtime errors become more frequent as the sequence of requests progresses, suggesting agents accumulate problems in the persistent codebase rather than holding it stable.
• No numeric results reported in the abstract: The abstract does not state accuracy figures, per-model scores, or threshold values; it only describes the qualitative patterns above.

Methodology in Plain English

The researchers built a benchmark where an agent works on one persistent codebase at a time. Each of the 32 applications receives a series of cumulative requests, so the agent must modify what already exists rather than start fresh. The requests are linked by dependency graphs, meaning some features logically require earlier ones to be in place first. To judge the result of each step, the team uses a multimodal agent that opens the running application in a real browser and interacts with it, rather than relying only on static code inspection. That evaluation checks three things each round: whether the new feature works, whether it integrates with what came before, and whether earlier features still function. Instead of deciding on a single pass/fail bar, they measure performance across a range of how tolerant a user would be, producing a trajectory of reliability over the whole development sequence. They then run this setup against several frontier models from both open- and closed-source families.

Why This Matters

Impact on research: Existing web-generation benchmarks largely measure one-shot generation against fixed specs. EvoWeb reframes evaluation around long-horizon, multi-request development, which is closer to how software is actually built, and introduces reliability-over-time and cross-request integration as measurable axes.

Real-world applications:

• Evaluating coding assistants that operate inside an existing repository over days or weeks of iterative feature work.
• Regression testing for AI-generated code, since the benchmark explicitly checks that previously working functionality survives new changes.
• Prioritizing product features by complexity, using dependency graphs to flag which requests are likely to be harder for automated tools.
• Comparing candidate coding agents or models before deployment in development pipelines.

Industry relevance: Teams adopting AI coding agents need evidence that those agents do not silently break the application as they add features. A benchmark that measures integration and error accumulation over successive requests speaks directly to the reliability questions that block adoption in production settings.

Future Directions

• Improving agents specifically on cross-request integration, which the abstract names as the key unsolved challenge.
• Addressing the degradation pattern: understanding why runtime errors increase over successive requests and how to prevent codebase drift or accumulated faults.
• Handling deep and highly connected dependency chains, since accuracy falls as dependency depth and count grow.
• Using EvoWeb as a testbed for developing and comparing methods, as the abstract frames the benchmark as a tool for both evaluation and improvement of long-horizon coding agents.

Target Audience

Researchers and engineers working on coding agents, LLM evaluation, and automated software engineering; benchmark designers interested in long-horizon or multi-step task evaluation; and product or platform teams assessing whether AI coding tools can be trusted to maintain a live application across a series of changes.

Authors’ abstract

Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily evaluate development toward fixed specifications. We introduce EvoWeb, a novel benchmark comprising 320 cumulative requests over 32 web applications, grounded in real-world product features and structured by functional dependency graphs. Agents iteratively modify a persistent codebase, while a multimodal web agent tests each intermediate application in a live browser, assessing newly requested functionality, cross-request integration, and previously requested functionality. To characterize reliability throughout development, we integrate trajectory performance across user-tolerance thresholds rather than impose a fixed stopping criterion. Experiments across open- and closed-source frontier models show that EvoWeb reveals clear performance differences between models, with cross-request integration emerging as a key challenge. Further analysis shows that later requests with deeper or more numerous dependencies are solved less accurately, while runtime errors become more frequent over successive requests. EvoWeb provides a testbed for evaluating and improving coding agents in long-horizon, multi-request development.