WebAnything: From the Wild Web to Scalable Agent Environments

AI summary

Overview

Research area: AI agents / web-agent training environments — automated website reconstruction, environment verification, and data generation for agent post-training.

Technical level: Advanced. The paper assumes familiarity with web-agent benchmarks, browser automation, and supervised fine-tuning, and it reports a multi-stage system (explorer, builder, independent verifier, task pipeline).

Scope (1 sentence): The paper describes WebAnything, a fully automated framework that reconstructs source-grounded, executable, resettable local web environments from real URLs, verifies them independently, and uses their interaction data to fine-tune a web agent.

What This Paper Is About

Training and evaluating web agents requires websites that are diverse, executable, and resettable, but live websites are hard to reproduce and isolate safely. Existing alternatives each trade off automation, fidelity to the original source, and interaction coverage: human-assisted mirrors need per-site engineering, generative construction can drift from the specified site, and capture-and-replay is limited by what was captured. The paper's goal is to convert real websites into reusable local environments at scale, grounding reconstruction in browser-observable evidence and verifying it independently, without manual per-site implementation or repair after initial setup and authorization.

Key Contributions

1. Fully automated, source-grounded reconstruction. A unified Explorer–Builder reconstructs executable web environments from real URLs using browser-observable page structures, resources, and before/after interaction observations, with no per-site manual implementation or debugging in the audited construction runs.

2. Independent verification with evidence-aware refinement. An Independent Verifier examines frozen candidates in a separate context against source evidence and site contracts, and routes each discrepancy either to Local Repair (when evidence establishes the expected outcome) or Targeted Recapture (when evidence is insufficient), followed by automated revision and reverification.

3. A scalable environment and task supply. The resource pool contains 244 website artifacts across 14 application domains and 53 categories, supporting 12,984 executed and accepted tasks drawn from two disjoint collections.

4. Downstream utility evidence. Supervised fine-tuning on 5,856 WebAnything interaction trajectories (5,167 unique training task IDs, converted into 60,517 step-level examples) improves Qwen3.5-9B on REAL, Odysseys-Easy, and WebArena-Verified.

Main Findings

• Source-state fidelity (five matched WebHarbor sites, 14 historical states): WebAnything scores higher on all three aggregate metrics — Pixel Similarity 0.9240 vs. 0.8083, SSIM 0.7904 vs. 0.6260, and Text F1 0.8541 vs. 0.2617.

• Blind human audit: 12 wins, two ties, and no losses (85.7% win rate) on overall fidelity; 10 wins, 3 ties, 1 loss (71.4%) on information fidelity; discrepancy severity mean 0.143 vs. 1.429 on the 0/1/2 scale.

• LinkedIn comparison (four historical states): Pixel 0.9574 vs. 0.9436 and SSIM 0.8190 vs. 0.7637 in favor of WebAnything; Text F1 is N/A.

• Fidelity does not imply interactivity. On the WebHarbor subset, WebAnything reaches 75% capability coverage and 63.3% all-test pass (19/30), versus WebHarbor's 80% and 76.7% (23/30). On the separate VeriEnv subset, selected pool artifacts pass 29/33 tests versus 21/33 for VeriEnv, with VeriEnv ahead on LinkedIn.

• Self-checks miss real defects. Of 229 historical Builder–Verifier report pairs across 69 sites, 219 have passing Builder self-checks, yet independent verification found evidence-supported discrepancies in 39 report pairs across 22 sites.

• Repairs are traceable and reverified. A repair-trace cohort over 46 sites yields 18 independently reverified resolution records across six sites: 14 local implementation repairs, three runtime repairs, and one request-lifecycle issue.

• Cost and automation record. For 84 of the 85 historical sites, recorded model-call cost totals $220.62 (mean $2.63, median $1.60 per site), split roughly $142.52 for Builder and $78.11 for the Independent Verifier; the figure is a lower bound excluding unpriced calls and system overhead.

• Construction outcomes (85-site historical subset): 59 passed sites, 15 source-blocked, 6 failed, 5 with missing reports — statuses reflecting differing versions of acceptance criteria.

• Task diversity and depth: Observed tool-interaction depth averages 10.4 steps (median 8, 90th percentile 21, maximum 89), and 34.7% of tasks exceed 10 steps; capability labels mark 26.8% state operations and 37.7% cross-site attributes.

• SFT gains on REAL: On 91 jointly valid tasks, success rises from 22/91 to 31/91 (+9.9 percentage points); retaining infrastructure failures across the full 102-task set gives 21.6% vs. 30.4% (+8.8pp).

• SFT gains on Odysseys-Easy (44 jointly valid tasks): Rubric Avg 19.71% → 25.96% (+6.25pp), Task Avg 18.86% → 25.18% (+6.32pp), perfect tasks 2/44 → 4/44.

• SFT gains on WebArena-Verified (fixed harness): Overall 16.41% → 29.74% (+13.33pp), with Navigate +18.49pp (22.22 → 40.71) and Mutate +31.50pp (10.51 → 42.01), while Retrieve drops 8.75pp (20.75 → 12.00) under strict scoring.

• Scoring-protocol sensitivity: Rescoring the same SFT outputs while ignoring JSON-format judgment raises Retrieve passes from 39/325 (12.00%) to 105/325 (32.31%), i.e. 66 additional passes (+20.31pp).

• Data-source comparison (same backbone, config, and harness; 661/617/663 trajectories): On 189 jointly valid Odysseys-All tasks, WebAnything leads with Rubric Avg 10.44 and Task Avg 11.70 (1/189 perfect), versus WebHarbor at 6.93/7.79 and VeriEnv at 6.42/7.53, both with 0/189 perfect.

Methodology in Plain English

The system starts from a real URL, an entry point, a permitted access scope, and a construction budget, and aims to produce a local environment that behaves like the site within a declared "site contract." Rather than reading private backend code, it works only from what a browser can see.

• Collect evidence. An Explorer–Builder visits page states and records three kinds of evidence: page structure and appearance, resources and dependencies (fonts, images, media), and behavioral observations — what the page looks like before and after an action. Importantly, two different states at the same URL (a closed menu versus an expanded one) are kept as distinct evidence.

• Build locally. Those observations are turned into local components, routes, shared object relationships, and state transitions. Screenshots complement structural and behavioral evidence rather than standing alone. When observations cannot establish a required behavior, the system seeks more evidence instead of guessing.

• Two exploration policies. Full Coverage expands page types, interaction states, and neighboring branches from the entry URL. Just-in-Time Coverage prioritizes dependencies of a target task set. The reported construction experiments use Full Coverage.

• Verify independently. The Builder self-checks a candidate, then freezes it. A separate Independent Verifier — which cannot edit the code — re-renders and re-operates the frozen candidate and compares it against source evidence and the contract, restoring initial state between stateful tests.

• Route discrepancies. If existing evidence shows what should happen and the candidate fails to do it, the system issues a Local Repair request with the affected page, action, expected and observed outcomes, and supporting evidence. If the evidence is insufficient, it issues a Targeted Recapture request to revisit that part of the source. The candidate is frozen and rechecked again after any change.

• Package and use. Released environments bundle local implementations, resources, initial data, runtime configuration, and reset interfaces. A task pipeline generates instructions with matching success conditions for implemented capabilities; agents run them through a fixed browser-based computer-use harness, evaluators check final answers or terminal states, and accepted trajectories are exported as step-level training examples.

• Evaluate. Quality is measured with Pixel Similarity, SSIM, and Text F1 (aggregated within sites, then averaged equally across sites), plus a blind human audit and offline interaction tests with external network access disabled. Downstream utility is measured by fine-tuning Qwen3.5-9B for 473 steps with global batch size 256 and learning rate 5 × 10⁻⁶, then testing on external benchmarks.

Why This Matters

Impact on research. The paper separates two things that are often conflated — how much an environment looks like the source and what an agent can actually do in it — and shows they can diverge in both directions across baselines. Its verification design also distinguishes implementation bugs from missing evidence, which is a reusable idea for any system that builds artifacts from partial observations, not just websites.

Real-world applications:

• Agent training data at scale: Converting authorized sites into resettable environments that generate thousands of accepted tasks and interaction trajectories for post-training.
• Reproducible benchmarking: Fixed initial states, reset mechanisms, and task evaluators let researchers compare agents without live-web instability.
• Regression testing of web workflows: Checkable site contracts and state restoration support repeated multi-step workflow tests (search, booking drafts, favorites, login handling) in isolation.
• Safe sandboxing: Local mirrors with external network disabled let developers explore risky or stateful interactions without touching production sites.

Industry relevance. The recorded cost profile (mean $2.63 and median $1.60 per site across 84 sites, a stated lower bound) and the absence of per-site manual implementation, debugging, or repair in the 85-site historical subset indicate that environment construction can be treated as an automated pipeline rather than an engineering project. The 9.9pp, 6.25pp/6.32pp, and 13.33pp downstream gains give a concrete signal that such data transfers to external benchmarks.

Future Directions

• Scaling to larger models and other post-training methods. The paper states its training experiments primarily concern the 9B parameter scale and that it has not systematically evaluated the data with larger foundation models, different post-training methods, or larger training budgets.

• Handling continuous real-web change. WebAnything reconstructs and verifies source states at particular times; the authors note no systematic evaluation of mirror timeliness or long-term maintenance efficiency, and propose source-change detection, incremental evidence capture, and local environment updates.

• Copyright, authorization, and distribution. High-fidelity reconstruction involves content that may be protected by copyright, trademarks, and terms of service; future releases may prioritize the general framework, evaluation protocols, and de-identified research data, with mirror distribution scoped by license.

• Open questions the results themselves raise. Why does strict-scoring Retrieve fall by 8.75pp while Navigate and Mutate rise sharply, and why does WebHarbor still show broader tested interaction coverage (80% vs. 75% capability coverage, 76.7% vs. 63.3% all-test pass) despite lower source fidelity? The authors note that no unified construction success rate is computed because neither the 244-site pool nor the 85-site subset has a complete attempt cohort under uniform acceptance criteria.

Target Audience

Researchers and engineers building web or computer-use agents who need controllable, resettable environments; teams working on agent post-training data pipelines; benchmark designers interested in verification methodology and in separating visual fidelity from functional coverage; and practitioners evaluating whether automated site reconstruction is a practical alternative to human-assisted mirroring. Readers should be comfortable with agent evaluation harnesses and supervised fine-tuning to get the most from the results tables.

Authors’ abstract

Web agents require diverse, executable, and resettable website environments for reproducible training and evaluation. Human-assisted mirrors demand per-site engineering, while generative construction can diverge from specified real websites. Capture-and-replay preserves observations, with interaction coverage constrained by captured content and replay capabilities. We present WebAnything, a fully automated framework that reconstructs source-grounded, reusable Web environments from real URLs without per-site manual implementation or repair after initial setup and authorization. An Explorer-Builder collects browser-observable structures, resources, and interaction outcomes; an independent verifier checks frozen candidates and guides local repair or targeted source recapture. We construct 244 website artifacts across 14 application domains and 53 categories, supporting 12,984 executed and accepted tasks. Compared with human-assisted WebHarbor, WebAnything achieves higher aggregate scores on three fidelity metrics, with 12 wins, two ties, and no losses in a blind audit. Fine-tuning Qwen3.5-9B on 5,856 interaction trajectories improves REAL success by 9.9 percentage points, Odysseys-Easy Rubric Avg and Task Avg by 6.25 and 6.32 percentage points, and WebArena-Verified Overall success by 13.33 percentage points under the fixed harness. Code: https://github.com/china-qijizhifeng/Web-Anything