EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants

AI summary

Overview

Research area: Evaluation of large language models as generators and maintainers of interactive web interfaces (generative UI), spanning frontend generation, multi-turn instruction following, stateful interaction, and tool-grounded external state.

Technical level: Intermediate. The concepts are accessible, but the metric definitions (Turn Pass, TP@5, CPT, APR) and the browser-execution evaluation pipeline assume some familiarity with LLM benchmarking.

Scope: This paper introduces EvoGenUI-Bench, a 150-task, 750-turn benchmark that scores whether eight LLMs can repeatedly revise a single executable web interface across five cumulative turns while preserving prior behavior, local state, external state, and consistency with their own text responses.

What This Paper Is About

LLMs can already produce dashboards, forms, comparison views, and mini-applications as a direct response modality, but real assistant use means the user keeps changing requirements. The core problem is that a model must update one persistent executable interface, not generate a fresh page each turn, and each update must implement the new request without breaking requirements that remain valid. The paper's goal is to build and run a benchmark that measures reliability across the whole revision sequence, not just whether a single generated page looks correct in isolation.

Key Contributions

1. EvoGenUI-Bench, a multi-turn benchmark of 150 human-authored tasks and 750 requested turns across three 50-task suites (Presentation, Interaction, Tool-grounded), each task a five-turn cumulative revision of the same interface.
2. A browser-based, evidence-grounded evaluation protocol that builds and executes each generated artifact, then scores Presentation, Execution, and Alignment using screenshots, DOM and source evidence, actor interaction traces, tool and resource logs, runtime snapshots, and assistant text.
3. Reliability metrics at three levels: Turn Pass (TP) for individual turns, TP@5 and Consecutive Passed Turns (CPT) for episodes, and Adjacent Pass Retention (APR) for whether success persists across adjacent revisions, plus an independence baseline for TP@5.
4. A human-validated diagnostic taxonomy of six interface-maintenance failure mechanisms, validated for exact-label agreement and inter-annotator reliability, applied to all 2,750 executed non-passing calls.

Main Findings

• Turn-level success overstates episode reliability. Claude-Opus-4.7 reaches the highest overall TP (74.9%) and APR (83.6%) but completes only 37.3% of five-turn episodes. Model-averaged TP is 42.7% while model-averaged TP@5 is only 11.8%.

• Tool-grounded tasks are the hardest after conditioning on a prior pass. Model-averaged TP falls from 55.5% (Presentation) and 47.6% (Interaction) to 25.0% (Tool-grounded), with TP@5 at 5.0%. APR shows the same ordering: 71.1%, 68.7%, and 52.4%. These suites also have much denser validation contracts, averaging 11.9 private requirements per turn versus 3.0 for Presentation and 5.9 for Interaction, and 6–22 tools per task.

• Passing turns cluster within episodes rather than behaving independently. Observed TP@5 exceeds the independence estimate for all eight models: 21.3% versus 8.7% for GPT-5.5, 7.3% versus 1.3% for Qwen3.6-Plus, 4.7% versus 1.2% for Gemini-3-Flash, and 37.3% versus 23.4% for Claude-Opus-4.7.

• Reliability degrades after the second turn. Aggregate pass rate falls to 39.4% at turn 3 and 35.1% at turn 4, with the largest drop between turns 2 and 3. For Tool-grounded tasks the rate falls from 39.5% at turn 2 to 18.5% at turn 3 and 14.0% at turn 4.

• Models have distinct suite profiles. GPT-5.5 and Claude-Opus-4.7 both reach 40.0% TP@5 on Interaction. Qwen3.6-Plus drops from 60.0% TP on Presentation to 25.2% on Tool-grounded. Gemini-3.1-Pro has low overall TP (23.6%) yet comparatively high APR once a passing state is reached (70.3% Presentation, 79.6% Interaction).

• Failures have different mechanisms per suite. Across 2,750 executed non-passing calls, the taxonomy assigns 859 information-architecture, 586 derived-state-propagation, 460 affordance-binding, 410 requirement-decomposition, 289 external-state-grounding, and 146 domain-representation failures. Presentation failures center on information architecture; Interaction failures on derived-state propagation and affordance binding; Tool-grounded failures additionally involve external-state grounding and requirement decomposition.

• Failed transitions are split between regression and new-requirement failure. A fixed-seed post hoc audit identifies 110 failed APR transitions with sufficient attribution evidence: 58 (52.7%) involve regression on prior behavior (sometimes alongside failure on the new requirement), and 52 (47.3%) preserve prior behavior but fail the new requirement.

• Evidence sources are complementary, and actor traces matter most. On a balanced set of 240 cases, full-evidence evaluation reaches 87.5% accuracy; removing the actor trace drops it to 55.0%, removing the private reference to 63.8%, the source code to 67.5%, and the screenshot to 78.3%.

• Suite-level difficulty ordering is stable across regenerations. Three independent runs of Qwen3.6-Plus give overall TP between 39.7% and 46.4%, with Presentation > Interaction > Tool-grounded holding in all three runs.

• Evaluator–actor choice shifts agreement only modestly. Across 120 stratified cases, agreement with human annotations ranges from 86.1% to 91.4% across tested actor–evaluator pairings; the default MiMo–MiMo configuration achieves 89.7%.

Methodology in Plain English

The benchmark reframes generative UI as multi-turn artifact maintenance. Each task is a five-turn episode where the user progressively revises the same web interface. At every turn the model receives the current request, public task context, optional public tool contracts, compact evidence from earlier turns, and the most recently generated source code, and must return both a user-facing reply and complete source for the updated interface.

The harness builds the returned code in a fixed React/Vite web environment and serves it in a browser. An interaction actor operates the running interface, and the system records build and browser evidence, screenshots, DOM state, actor traces, tool and resource logs, and runtime snapshots. Execution is sequential: later turns reuse earlier generated source and snapshots and, for tool-grounded tasks, restored runtime state. Missing artifacts block dependent turns rather than restarting from a clean state, and blocked requested slots are not omitted from scoring.

A model-based evaluator then assigns three turn-level scores on a 1–5 scale: Presentation (coherence, readability, domain appropriateness of the rendered interface), Execution (whether the current request and still-valid prior requirements work as interface behavior or evidence-supported visible state), and Alignment (whether the assistant response, source, rendered UI, interaction observations, and runtime logs agree). A turn passes only if generation succeeds, the source yields an evaluable artifact, and all three scores are at least 4.

The metrics build on that pass indicator. Turn Pass is the mean pass indicator over all requested model–turn slots. TP@5 is 1 only when all five turns in an episode pass, CPT is the length of the initial uninterrupted run of passing turns (0 to 5), and APR is the probability a turn passes given that the previous turn passed and the current generation call returned a response. TP, TP@5, and CPT are macro-averaged across tasks at suite level; APR pools all eligible adjacent transitions. The paper also reports an independence baseline that multiplies each model's empirical position-specific pass rates.

Validation used a blinded sample of 240 turn-level cases covering all eight models and all three suites, with 10 cases per model–suite pair stratified by automatic outcome, labeled by three non-author annotators. The diagnostic taxonomy was frozen as a six-label codebook before large-scale labeling and checked on a category-balanced sample of 120 classified failures.

Setup details: eight models were evaluated on the same 150 tasks — GPT-5.5, Qwen3.6-Plus, Qwen3-Coder, GLM-4.5-Air, Gemini-3-Flash, Gemini-3.1-Pro, Claude-4.5-Haiku, and Claude-Opus-4.7 — with MiMo-V2.5 used as both interaction actor and evaluator for the main results.

Why This Matters

Impact on research. The results separate two questions that prior benchmarks tended to conflate: whether a model can produce a correct artifact, and whether it can preserve correctness across revisions. The paper also shows that no single evidence surface is sufficient for judging generated interfaces, and that diagnosis requires matching evidence to mechanism — screenshots for layout defects, actor traces and source diffs for broken interactions and stale derived state, runtime logs for interface-to-external-state mismatch. The benchmark, code, and data are released as version v1.0 under the MIT License at the project repository, with generator-visible material separated from evaluator-only material such as hidden validation contracts and backend fixtures.

Real-world applications.

• AI assistants that build and edit dashboards or reports on request, where users expect earlier filters, views, and computed metrics to survive subsequent edits.
• Stateful mini-applications and simulators such as control or tuning labs, where dependent readouts must update when a user changes an input.
• Tool-backed operations consoles where an interface reads and writes authoritative external state, for example scheduling, case management, or approval workflows.
• Quality gates for generated interfaces, where the six-mechanism taxonomy and the evidence ablation can be reused as an internal checklist for auditing generated UI before shipping.

Industry relevance. The 2,750-call diagnostic breakdown gives teams a concrete failure profile to test against: mismatched backend tokens that make controls unreachable, stale derived metrics after local state changes, and buttons that render but are not wired. The evidence ablation is directly actionable for anyone building an LLM-as-judge pipeline, since dropping interaction traces cut accuracy to 55.0%.

Future Directions

• Extending beyond the current environment. The benchmark uses human-authored five-turn episodes in a fixed React/Vite browser environment; the authors state that naturally occurring interaction logs, other UI frameworks, accessibility requirements, and device conditions are out of scope.
• Full causal attribution rather than outcome-level retention. APR is described as an outcome-level measure, not causal attribution; comprehensive attribution would require replaying all prior-turn validation contracts against each later artifact.
• Moving from deterministic mock runtimes to live services. Tool-grounded tasks use deterministic mock runtimes for reproducibility, so latency, authentication and permission failures, outages, rate limits, and API changes are not captured.
• Closing the turn-versus-episode gap. Observed TP@5 exceeds the independence estimate for every model, suggesting episodes differ in difficulty; the paper leaves open what would make sustained five-turn success more achievable and how models could be helped to preserve still-valid requirements.

Target Audience

Researchers and engineers working on LLM evaluation, generative UI, frontend code generation, and multi-turn agents will get the most from this paper, particularly those building agent or assistant products where generated interfaces persist and change over time. Benchmark and evaluation practitioners will find the evidence-ablation results and the reliability metric definitions directly reusable. Product teams evaluating model choices for interface-generating features also benefit, since the per-model, per-suite table shows that a model strong on one suite can be weak on another.

Authors’ abstract

Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve. We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state. We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs. Beyond turn-level and episode-level success, we measure cross-turn retention with Adjacent Pass Retention. Across eight models, even the strongest achieves 74.9% Turn Pass while completing only 37.3% of five-turn episodes; APR further falls to 52.4% on tool-grounded tasks. Diagnostic analysis shows that presentation failures center on information architecture, interaction failures on derived-state propagation and affordance binding, and tool-grounded failures additionally involve external-state grounding and requirement decomposition. These results reframe generative UI evaluation from judging isolated outputs to testing whether interface behavior, derived state, external state, and assistant claims remain synchronized as the artifact evolves.