ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents

AI summary

Overview

Research area: AI agent evaluation — specifically computer-use agents (CUAs) that operate graphical user interfaces via screenshots, mouse, and keyboard.

Technical level: Intermediate. The paper is readable without deep machine-learning background, but it assumes familiarity with agent benchmarks, accessibility trees, and GUI grounding concepts.

Scope: The paper introduces a benchmark and diagnostic pipeline that isolates and measures how well computer-use agents handle individual web UI components, rather than whole multi-step workflows.

What This Paper Is About

Evaluation of computer-use agents currently sits at two extremes: long-horizon workflow benchmarks (like WebArena or OSWorld) that measure end-to-end success but make it hard to attribute why an agent failed, and atomic grounding tests (like ScreenSpot) that check whether an agent can click the right element but stop before the short multi-step interactions real widgets demand. This paper fills the under-instrumented middle layer by building a benchmark of realistic, component-centered tasks — toggle a button set, set a date range, drag a splitter — that are short enough to diagnose and rich enough to capture the burdens of modern interfaces.

Key Contributions

1. A library-agnostic component ontology. 97 canonical UI component types organized into 14 interaction families and 24 reusable task templates, derived by combining the WAI-ARIA Authoring Practices Guide with the production component inventories of Ant Design, MUI, Mantine, Fluent UI, Chakra UI, and Headless UI.

2. A programmatically verified benchmark of 2,910 tasks. Each task targets one primary component, is implemented as a real interactive Next.js page (not a static mockup), is scored by a deterministic end-state verifier rather than LLM judgment, and is paired with a cleaned human reference trajectory. A distilled 912-task hard-only companion suite, ComponentBench-Core, concentrates benchmark mass on unresolved interaction families.

3. A four-regime observation and action space design. The same underlying tasks run under AX-tree (screenshot + accessibility tree, element-ID actions), Set-of-Marks (screenshot with numbered overlays, element-ID actions), Pixel (screenshot only, coordinate actions), and Browser-Use (a separate tool-rich reference regime with serialized DOM access), allowing direct measurement of how interface design affects the same model.

4. A three-stage diagnostic pipeline and trace-grounded failure taxonomy. A deterministic layer assembles per-task evidence packets, an LLM layer produces structured task-level observations, and an aggregation layer produces component-level reports. Realized difficulty is audited by replaying human traces in Playwright to extract a 24-feature structural vector, distinguishing what the designers intended to build from what the rendered page actually presents.

Main Findings

• Observation and action space can shift pass rates by more than 30% within a single model. GPT-5 mini falls from 83.1% under accessibility-tree observations to 48.9% under coordinate-only Pixel control in the same harness — a 34.2-point swing driven purely by what the agent sees and how it refers to targets.

• Structured visual aids are not universally helpful. Set-of-Marks gives GPT-5 mini a 29.6-point boost over Pixel and Gemini 3.1 Flash-Lite a 10.2-point boost, but for GPT-5.4 mini and GPT-5.4 the ordering reverses, with Pixel outperforming SoM by 2.4 and 6.8 points. Overlays appear to help weaker models and mildly hurt models already strong at direct visual interaction.

• Efficiency is a major deployment bottleneck. Even the fastest configuration (GPT-5.4 mini under SoM) takes 3.7× as long as the matched human reference; the slowest (GPT-5 mini Pixel) takes 21.5× as long. The strongest model solves many more tasks eventually than within the human step budget.

• A difficulty inversion exists for spatial manipulation. Nine canonical component types require two or fewer human steps yet achieve below 60% agent pass rates: resizable columns (24.4%), window splitters (38.3%), range sliders (39.9%), and six other drag- or precision-heavy tasks. No component type shows the reverse pattern.

• Visual context disproportionately burdens screenshot-only agents. Medium clutter drops Pixel performance by 14.3 points but AX-tree by only 0.2; compact spacing drops Pixel by 13.6 versus 4.5 for AX-tree. For GPT-5.4, the preferred observation space actually flips on a single scene factor: Pixel leads by 4.5 points with no clutter, but AX-tree leads by 9.7 points under medium clutter.

• A trace-grounded failure taxonomy identifies specific mechanisms. Across 8,864 failed traces, the dominant categories are continuous calibration errors (20.2%) — the agent finds the right slider but cannot map pointer motion to the required value — transient state loss (19.9%), missing commit or confirmation actions (11.6%), target acquisition or wrong-instance errors (11.2%), and repetition loops (11.2%). Repeated-action loops appear in 55.8% of failures but are distributed across every category, making them a symptom rather than a root cause.

• The tool-rich Browser-Use regime helps unevenly. It delivers a 7.8- to 18.2-point aggregate advantage over the mean of the other modes, largest for weaker models, but this is strongly family-dependent: +29.0 points on Advanced Editors and −24.0 points on Drag/Drop, because DOM-level actions cannot replicate genuine spatial drags.

• The Core suite converts broad coverage into a stress test. Pass rates drop 10 to 39 points from Full to Core, with the largest drops for already-borderline combinations (GPT-5.4 mini Pixel falls from 77.1% to 37.7%). Even a frontier model, Opus 4.6, reaches only 65.4% on Core Pixel, with 34.1% solved within the human step budget.

Methodology in Plain English

The researchers built the benchmark in stages. First, they assembled an inventory of modern web UI patterns by cross-referencing the WAI-ARIA accessibility guidance with what major React component libraries actually ship, producing 97 canonical component types. Then GPT-5.2 Pro generated 30 task specifications per type under a strict YAML schema with balanced difficulty (10 easy, 10 medium, 10 hard) and enforced variation across eight controlled scene factors such as theme, spacing, clutter, and number of instances. Claude Code implemented each specification as a real interactive Next.js page.

Every task was then executed by a human operator twice, recorded as low-level actions with timestamps, and cleaned by merging consecutive keystrokes and dropping the longer pass. These traces serve double duty: they validate during construction that each task is solvable and faithful to its specification, and they later act as reference trajectories for efficiency analysis and as a "traversal oracle" for difficulty auditing — reaching hidden popovers and nested panels that a static screenshot would miss.

Success is judged by a programmatic verifier that checks the committed end state, not whether the agent briefly opened the right menu. Verifier state is deliberately isolated: target values live inside React component closures and never appear in DOM attributes, accessibility labels, or page text, and a MutationObserver strips test-only attributes in real time. The auditing pipeline then replays human traces in Playwright to extract 24 structural features and aggregates them into seven difficulty axes, separating intended from realized difficulty.

Why This Matters

Impact on research: The paper argues that component-level reliability is a hidden ceiling on workflow-level success — under a simple independence approximation, five critical interactions each at 80% reliability imply only about a 33% end-to-end ceiling. ComponentBench provides the instrument to measure that ceiling directly, and its failure taxonomy converts opaque pass rates into actionable mechanism categories that long-horizon benchmarks and single-step grounding tests cannot surface.

Real-world applications:

• Web automation and RPA vendors can use the benchmark to identify which widget types their agents systematically fail on before deploying to customers, particularly drag-and-drop, splitters, and range inputs.
• UI component library maintainers (Ant Design, MUI, Mantine, and others) gain evidence about which component patterns are hostile to automated interaction, informing accessibility work and agent-friendly design.
• QA and accessibility testing teams can adopt the programmatic end-state verification approach as a model for automated regression testing of component behavior.
• Enterprise deployment teams evaluating whether to adopt computer-use agents can use the efficiency metrics — the 3.7× to 21.5× time-to-human ratios — to estimate realistic latency and cost budgets.

Industry relevance: The findings directly challenge a common assumption that better vision-language models will automatically close the agent reliability gap. The paper shows that the interface regime — what the agent observes and how it acts — can matter more than model capability, with the same model swinging over 30 points between regimes. This has direct implications for product decisions about whether to expose accessibility trees and DOM structure to agents or rely on screenshot-only control.

Future Directions

• Validating whether component-level scores predict workflow-level success. The paper explicitly names this as future work; establishing the mapping from component competence to end-to-end task reliability would make the benchmark a predictive tool rather than a diagnostic one.

• Extending the ontology beyond the web. The benchmark is currently web-first (Next.js on Chromium), but the authors note the ontology is designed to port to desktop and mobile interfaces, where component behavior and interaction affordances differ substantially.

• Closing the spatial manipulation gap. Since sliders, splitters, and drag targets are trivial for humans in one or two steps yet resistant to added reasoning, the question is whether this requires new action primitives, calibration-aware training, or hybrid observation spaces — the paper offers no solution, only evidence that the problem exists.

• Moving beyond single-run evaluation and LLM-generated tasks. The authors report a repeated-run study with at most 1.4% deviation and acknowledge that task specifications were generated by GPT-5.2 Pro and implemented by Claude Code, which may introduce generator-specific regularities. Broader contamination auditing and native-interface evaluation (via Operator, the Computer Use API, or Anthropic's full computer-use environment rather than chat completions) would strengthen confidence in the results.

Target Audience

This paper is most valuable to AI researchers and engineers building or evaluating computer-use agents, particularly those working on GUI grounding, visual web navigation, and agent benchmarking methodology. It also serves product and infrastructure teams deciding how much structured interface information to expose to agents, and UI/accessibility researchers interested in which component patterns remain hostile to automated interaction. Readers seeking a general introduction to agent evaluation will find the framing useful, but some familiarity with accessibility trees, coordinate-based control, and benchmark design will help.

Authors’ abstract

Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.