Skip to content
AI.info

Research

WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents

Overview Research area: AI agents for web browsing and computer use — specifically benchmarking LLM- and vision-language-model-driven browser agents. Technical level: Intermediate.

arXiv
2611.17955
Published
2026-11-13
Authors
Imene Kerboua, Fatemeh Pesaran Zadeh, Xing Han Lù, Weijian Qi, Alexander Miller, Junyi Song, Yunjia Tian, Dongjin Kang, Seyeon Choi, Marzia Nouri, Ewen Gueguen, Matteo Boglioni, Fengyuan Liu, Zeyi Liao, Mengqi Yuan, Yue Li, Alexandre Lacoste, Alexandre Drouin, Spandana Gella, Huan Sun, Gunhee Kim, Siva Reddy

AI summary

Overview

  • Research area: AI agents for web browsing and computer use — specifically benchmarking LLM- and vision-language-model-driven browser agents.
  • Technical level: Intermediate. Readers should know roughly what an agent harness, an accessibility tree, and a success-rate metric are, but no specialist web-agent background is required.
  • Scope (one sentence): The paper introduces WebArena-Pro, a reproducible benchmark of 300 human-validated tasks across 20 self-hosted web applications in six domains, which adds audio and video to the agent's observation space and evaluates 12 model backbones across 4 agent harnesses.

What This Paper Is About

Existing reproducible web-agent benchmarks cover only a handful of self-hosted applications from a few software categories, and they restrict what the agent can perceive to text and screenshots. Live benchmarks built on real production websites cover many more sites but lose reproducibility, because pages, data and policies drift between runs. The authors build a benchmark that keeps reproducibility (self-hosted apps, per-task container resets, seeded data) while widening heterogeneity to 20 applications and extending the modality range to audio and video embedded in web pages.

Key Contributions

  1. Heterogeneous web coverage. 20 self-hosted applications across six domain categories (productivity, communication, analytics, business operations, design and media, travel), each modeled on a widely used reference service such as Notion, Slack, Spotify, Figma or Airbnb, with distinct interface conventions, authentication flows, workflows and backend data models.
  2. Audio and video as core web observations. Audio and video sources are surfaced as nodes in the accessibility tree, and the action space gains transcribe() (local Whisper model) and extract_frames() for tool-based media access, plus listen() and watch() for omni models that process raw media directly.
  3. Reproducible, parallel evaluation. Each task runs in isolation against per-task seeded container state, with a fresh website container, UI login and navigation to the task's start URL at the beginning of every episode, so an agent's backend changes never leak into later tasks.
  4. Human-validated task construction. A task creation pipeline combining a dedicated annotator interface, LLM-assisted triage that discards tasks solved correctly on the first attempt, and final human review on the live application before release.

Main Findings

  • Top scores are only 62.3%. Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tie for the best overall success rate at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7% and OpenCode with Qwen 3.8 Max Prime at 55.7%.
  • The best GenericAgent configurations reach 30.3%. Gemini 3.1 Pro as an omni model and Gemma4 31B tie there, while the Qwen backbones lag at 5.0% (Qwen 3.5 9B) and 15.7% (Qwen 3.8 27B).
  • Substantial headroom remains. Even the strongest configuration leaves 38% of tasks without a recorded success.
  • Cross-site tasks are consistently harder. Gemma4 31B under GenericAgent shows the largest gap, falling from 38.2% on single-site tasks to 13.5% cross-site, a 24.7 percentage-point drop. Claude Code has the smallest gap among the five coding-agent configurations (65.7% single-site vs 55.2% cross-site, 10.5 points), and Codex reaches the highest single-site score at 67.2% but drops to 52.1% cross-site.
  • Domain rankings differ by agent. Communication yields the highest success rate for each of the five coding-agent configurations (Codex 81.8%, OpenCode with Kimi 79.5%, Claude Code 72.7%, OpenCode with Qwen 70.5%, OpenCode with Gemini 65.9%). Codex and Claude Code tie for the lead in Productivity at 60.4%; Claude Code leads Design & Media at 65.3% and Business Operations at 57.1%; OpenCode with Kimi leads Analytics at 68.8%; Travel is hardest with a best of 57.7% (Codex). No configuration leads outright in every domain.
  • The modality picture is mixed. Within GenericAgent, audio/video tasks score higher than text/vision tasks for all models except Gemma4 26B A4B — Gemini 3.1 Pro omni, for example, gets 26.2% on text/vision versus 41.8% on audio/video. Among coding agents the advantage reverses for OpenCode with Qwen 3.8 Max Prime and Kimi 3.1 Pro, Codex and Claude Code; only OpenCode with Gemini 3.1 Pro shows a small audio/video advantage of 1.9 percentage points.
  • Media access matters a great deal. On the conservative subset of 43 media-content-dependent tasks, removing media tools drops Gemini 3.1 Pro to 2.3% and Qwen 3.6 27B to 0.0%; with transcribe() and extract_frames() they reach 30.2% and 16.3%, and Gemini 3.1 Pro with native watch() and listen() reaches 34.9%. Claude Code with Opus 5.5 goes from 14.0% without media tools to 48.8% with them.
  • Failures on minimal tooling can be near-total. On the audio subset, Gemini 3.1 Pro and Qwen 3.6 27B fall to 0% without media tools; Gemini 3.1 Pro solves a single video task without them, which the authors attribute to a correct guess.
  • Evaluation is not free. GenericAgent costs approximately USD 800 for a 300-task run with Claude Opus 4.7; recorded coding-agent costs are USD 386 (Claude Code with Opus 5.5), USD 166 (Codex with GPT 6.1 Sol), USD 689 (OpenCode with Qwen 3.8 Max Prime) and USD 340 (OpenCode with Kimi 3.1 Pro). Judging with Qwen 3.6 27B costs less than USD 4 per 300-task evaluation in those runs.
  • Tool use varies sharply. Average tool calls per task: 31.1 for Claude Code, 33.3 for Codex, 36.5 for OpenCode with Gemini, 28.1 for OpenCode with Qwen 3.8 Max Prime, 29.8 for OpenCode with Kimi.

Methodology in Plain English

The authors stood up 20 real web applications locally, each in its own Docker container, each modeled on a familiar product but stocked with their own seeded data — files, messages, calendar events, dashboards, tickets, listings — and pre-made user accounts so tasks start already logged in. Before each task the container is reset to that seeded baseline, which is what makes runs repeatable and lets many tasks run in parallel without interfering with each other.

Tasks are written by the paper's own annotators through a purpose-built interface: they write a natural-language instruction, pick which sites it involves, and draft the grading rubric. A candidate task is then sanity-checked automatically by running a model on it and throwing it away if the model solves it on the first try, and finally a second human verifies it on the live app. Each of the 300 tasks is its own dedicated template rather than one of many instances of a shared template.

Grading is rubric-based. Each task comes with a small set of human-written criteria tagged as either action criteria (did the agent search, filter, submit?) or outcome criteria (is the booking there, does the file say the right thing?). These criteria form a directed acyclic graph, and a task counts as solved only if every node passes. Each node is checked by one or more graders in order: a deterministic grader that queries the app's backend state or looks for required strings in the answer, then a heuristic grader that matches URL and action patterns, then an LLM judge (Qwen 3.6 27B) that reads the whole trajectory and makes the final call.

For media, the agent sees audio and video nodes in the accessibility tree and can call transcribe() to get a transcript of a time range or extract_frames() to get evenly spaced frames from a video range; omni models instead call listen() and watch() to receive the raw segment. Agents run in a BrowserGym-compatible environment with a 50-action budget per task, reasoning effort set to high, temperature 1.0 and, for GenericAgent, a 40k-token-per-step cap. Coding agents (OpenCode, Codex, Claude Code) run through the Harbor harness with a Playwright MCP for browsing and an augmented media MCP.

Why This Matters

Benchmarks shape what agent research optimizes for, and this one argues that the field's reproducible benchmarks have been measuring agents on a narrow slice of the web: few apps, few software categories, and only text and images. By widening applications, adding temporal media as a first-class observation, and preserving per-task isolation, WebArena-Pro makes it possible to ask questions the older benchmarks could not — does an agent transfer across two apps with different conventions, and can it actually extract information from a podcast or a video?

Real-world applications:

  • Enterprise workflow automation, where assistants must move data between tools that look similar but behave differently, such as a Drive-like store and a Dropbox-like store with distinct permission models.
  • Customer support and ticketing, where an agent must read a ticket in one system and summarize it in another.
  • Media-heavy sites, where the relevant fact is spoken in a video or shown on screen rather than written in the page's text.
  • Sales and analytics operations, where agents query dashboards or update CRM records and must coordinate across systems.

Industry relevance: the reported cost figures and the large spread between agent harnesses using similar models suggest that harness and tool design matter as much as raw model capability — a directly actionable signal for teams building browser agents. The self-hosted, containerized setup also means the same infrastructure can be reused as a training environment with a per-task reward signal.

Future Directions

  • Richer temporal interaction. The current action space covers only prerecorded audio segments and discrete video frames; live voice dialogue and streaming video are explicitly left to future work.
  • Training on the benchmark. The authors propose reusing the infrastructure as a reinforcement-learning or rejection-sampling environment, since the rubric verifier already supplies a per-task reward.
  • Robustness studies. The controlled environment allows perturbing pages to test resilience to UI drift and prompt injection.
  • Cleaner attribution of performance. The authors note that current comparisons conflate model capability with harness design, that a shared 50-step budget does not equalize token consumption, native tool use or latency, and that controlled cross-harness comparisons with repeated runs are needed.

Target Audience

Researchers and engineers building or evaluating browser and computer-use agents, benchmark designers interested in reproducible multi-application and multimodal evaluation, and applied teams choosing agent harnesses and models for web automation. Readers focused on pure model architecture will find the paper less relevant than those working on agent scaffolding, tool design and evaluation methodology.

Authors’ abstract

Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between…

The full abstract on arXiv
Read the original paper