Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation

AI summary

Overview

• Research area: Web agents (LLM-driven browser automation), synthetic training-data generation, and automatic evaluation of agent trajectories.
• Technical level: Intermediate. The paper is readable without deep math, but assumes familiarity with LLM agents, supervised fine-tuning, LoRA, and vision-language-model judges.
• Scope: The paper proposes an end-to-end pipeline for generating, evaluating, filtering, and distilling web-agent trajectories, and evaluates the resulting 24B-parameter student model on a new booking benchmark (BookingArena) and on WebVoyager.

What This Paper Is About

Training capable web agents requires very large amounts of high-quality trajectory data, and because each web page can contain hundreds of interactive elements and tasks span many sequential decisions, most of that data must be synthesized. Synthesizing it creates a new problem: how to verify which generated trajectories are actually useful, since binary success/failure labels throw away partially correct attempts. The paper's goal is to build a scalable pipeline that generates trajectories automatically, scores them at the level of individual task constraints, and uses that fine-grained signal to train a small model that competes with much larger commercial systems.

Key Contributions

1. A scalable automatic data-generation pipeline that uses few-shot prompted publicly available language models to synthesize large numbers of web-agent trajectories for complex, multi-constraint tasks such as travel booking.
2. A constraint-based evaluation framework that decomposes a task into individual constraints, verifies each one against the final observation, and reports a Constraint Satisfaction Rate (CSR) alongside the binary task success rate (SR). This enables fine-grained filtering and lets partially successful trajectories be reused for training.
3. BookingArena, a new benchmark of 120 complex, structured booking tasks spanning 20 popular real-world websites, with difficulty defined by the number of constraints (6 tasks per site: 2 easy, 2 medium, 2 hard) and scripts that automatically refresh past dates so tasks stay valid.
4. A distilled 24B-parameter student model trained on the curated data that outperforms open-source alternatives and matches or exceeds commercial systems — notably, it beats the much larger 405B-parameter teacher model used to generate its trajectories.

Main Findings

• BookingArena average results (SR / CSR): the authors' model reaches 29.9 / 60.2, compared with Browser Use at 18.8 / 45.3, Claude Computer Use at 16.8 / 48.7, UI-TARS at 26.7 / 53.6, and Operator at 33.0 / 68.7. The authors' model leads on task success rate except against Operator.
• Success rate is harsh across the board. On several BookingArena websites performance is zero for multiple systems, which the authors attribute to the genuinely difficult nature of the tasks; they argue CSR is the more informative comparison signal.
• Partial data beats success-only data. Training only on successful trajectories gives 25.0 SR / 55.0 CSR; training on all trajectories gives 29.4 / 57.2; training on partially successful trajectories gives the best result at 29.9 / 60.2.
• LoRA outperforms full fine-tuning. Partial-success data with full fine-tuning scores 28.4 / 58.7, below the LoRA result of 29.9 / 60.2. The authors attribute this to overfitting in full fine-tuning given their data regime.
• The student beats its teacher. Few-shot teacher models score 18.0 / 46.0 (Qwen 2.5 72B), 22.5 / 49.8 (Llama 3.3 70B), and 22.5 / 52.4 (Llama 3.1 405B), all below the fine-tuned 24B Mistral 3 Small student.
• URL and page content are the cleanest evaluation signals; action history is not. Adding the URL improves evaluation for every model tested (e.g., the authors' model goes from 25.0 / 50.3 with screenshot only to 29.9 / 60.2 with URL). Adding action history appears to inflate scores through judge hallucination — the authors' model reaches 30.0 SR but CSR drops to 58.8 — and the authors confirm this setting correlates less with human evaluation.
• Constraint success depends on when in the task the constraint appears. Among the ten most frequent constraints, location succeeds 79% of the time, departure location 73%, destination location 67%, return date 64%, departure date 56%, start date 55%, pet-friendly 50%, end date 47%, rental type 41%, and number of guests 25%. Later-stage constraints that depend on earlier actions fare worst.
• WebVoyager overall: the authors' model scores 64.5% versus 52.6% for Wilbur (Claude), 52.8% for WebVoyager with Claude, 55.5% with GPT-4o, and 57.1% with GPT-4V. The authors report the largest gains on hard domains, improving 11.1% on Booking (55.0 vs. the best baseline's 43.9) and 15.1% on Google Flights (66.7 vs. the best baseline's 51.6). They state that all standard deviations were under 0.05 due to high GPT-4o self-agreement.
• Dataset scale achieved: 150k trajectories corresponding to roughly 1M individual actions/steps were collected; about 16k were judged completely successful, while about 65k had usable prefixes, yielding a curated training set of 300k actions/steps.

Methodology in Plain English

The pipeline has four stages.

Task generation. The authors start from a list of the 1,000 most popular safe websites and remove inaccessible ones. They then prompt GPT-4 with hand-written seed tasks to produce diverse, realistic target tasks across difficulty levels. Their prompt enforces criteria including precision, diversity, realism, complexity scaling, no spending money, no personal credentials, no reporting back of results, high-level goals rather than GUI-specific instructions, and booking dates one to four months in the future.

Trajectory generation. A few-shot prompted LLM agent (prompt adapted from Shi et al., 2024) interacts with real websites. At each step it sees the task description, current URL, the page's accessibility tree, prior actions and reasoning, and open browser tabs. The accessibility tree lists each element as [id] [tagType] [text content] [properties]. The agent emits an action — clicks, text input, hover, page navigation — plus chain-of-thought reasoning. Each observation logs the URL, accessibility tree, and a screenshot. Teachers were LLAMA 3 70B and LLAMA 3.1 405B Instruct.

Constraint-based evaluation and curation. For each task, an LLM derives the set of constraints that would need to hold for completion (for example, location, start date, end date). A judge then predicts, for each constraint, whether the observed value matches — and CSR is the fraction of constraints satisfied. Two judge models (LLAMA 3.3 70B and Gemma 3 27B) score the data, and only instances where both agree are kept, to reduce evaluation noise.

The key trick is prefix extraction: the authors compute the maximum CSR reached anywhere in a trajectory, then keep the shortest prefix that attains it, discarding later actions that made things worse. This converts a failed trajectory into a valid training example for the part of the task the agent did handle. Stop actions are kept only when CSR equals 1. Hindsight re-labeling handles premature stops with partial progress: the task description is rewritten to match the constraints that were actually satisfied. The paper's example rewrites "On hilton.com, find a hotel in New Orleans on February 08 for 3 people" into "On hilton.com, search for hotels in New Orleans on February 8", and the agent's reasoning for the stop action is revised to match.

Distillation. The student (Mistral 3 Small 24B) is LoRA fine-tuned on the query and value projections (q_proj, v_proj) with learning rate 1e-4, a cosine scheduler, and batch size 16, learning to predict both the next action and its reasoning. All experiments ran on two machines, each with 16 A100 40GB GPUs.

For final BookingArena evaluation, GPT-4o is used as the judge with the final screenshot and URL — deliberately different from the LLAMA and Gemma models used for curation, to avoid curation bias.

Why This Matters

The paper's central claim is that the bottleneck for web agents is not model size but the availability of verified training data, and that a fine-grained constraint metric unlocks much more of that data than binary success labels. Using their constraint framework, they expand usable data from roughly 16k fully successful trajectories to roughly 65k trajectories with usable prefixes, a several-fold increase that translates directly into a better model.

Research impact: the constraint framework offers an alternative to LLM/VLM-as-judge evaluation for structured, multi-criteria tasks, and the prefix-extraction plus hindsight-relabeling recipe gives a concrete method for salvaging failed trajectories — an issue that affects essentially every synthetic agent-data pipeline.

Real-world applications:

• Travel and hospitality booking — flights, hotels, rental cars, and bus tickets across sites like Booking.com, Kayak, Google Flights, Hilton, and Airbnb.
• Consumer shopping agents that navigate product search and cart flows without needing payment credentials.
• Enterprise customer-service automation that must satisfy several stated customer requirements in one session.
• Benchmark and evaluation engineering for any team that needs to score multi-constraint agent behavior rather than ask a judge for a yes/no verdict.

Industry relevance: the result that a 24B fine-tuned model beats a 405B prompted teacher and is competitive with commercial computer-use systems matters economically, since inference cost and latency scale with model size. The finding that LoRA beats full fine-tuning at this data scale is also a practical recipe for teams without massive compute budgets.

Future Directions

• Extending to multimodal observations. The conclusion explicitly names extending support to multimodal observations as future work.
• Broadening beyond the web. The authors also call for expanding scope to other platforms and environments beyond web browsing.
• Improving late-stage constraints. The constraint-level breakdown shows number of guests at 25%, rental type at 41%, and pet-friendly at 50%, far below location at 79%, suggesting room to improve constraint handling that depends on earlier actions.
• Making evaluation less dependent on indirect signals. The history-ablation showed judge hallucination inflates scores when action history is included, while URL adds genuine signal — how best to give evaluators context without introducing optimism remains open.

Target Audience

Researchers and engineers working on LLM-based agents, browser automation, and synthetic data generation will get the most from this paper, particularly those building or evaluating long-horizon, multi-constraint web tasks. It is also useful for practitioners interested in data curation and distillation recipes — what to keep from failed trajectories, which fine-tuning method to choose, and which judge inputs to trust — and for teams that need a realistic benchmark of booking-style web tasks. Readers should be comfortable with concepts like accessibility trees, LoRA, and LLM-as-judge evaluation; no specialized mathematical background is required.

Authors’ abstract

We present a scalable pipeline for automatically generating high-quality training data for web agents. In particular, a major challenge in identifying high-quality training instances is trajectory evaluation - quantifying how much progress was made towards task completion. We introduce a novel constraint-based evaluation framework that provides fine-grained assessment of progress towards task completion. This enables us to leverage partially successful trajectories, which significantly expands the amount of usable training data. We evaluate our method on a new benchmark we propose called BookingArena, which consists of complex booking tasks across 20 popular websites, and demonstrate that our distilled student model outperforms open-source approaches and matches or exceeds commercial systems, while being a significantly smaller model. Our work addresses the challenge of efficiently creating diverse, realistic web interaction datasets and provides a systematic evaluation methodology for complex structured web tasks.