LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training

AI summary

Overview

Research area: Natural Language Processing / LLM-based digital agents / synthetic training-data generation for UI agents.

Technical level: Advanced. The paper assumes familiarity with LLM agent training, accessibility trees, trajectory synthesis, and continual learning, though the writing is fairly readable.

Scope: The paper introduces UI-Simulator, an LLM-driven "digital world model" that synthesizes structured UI states and transitions so that training trajectories for web and mobile agents can be produced at scale, plus UI-Simulator-Grow, a targeted scaling strategy that selects the most valuable tasks to synthesize next.

What This Paper Is About

Training digital agents requires large amounts of high-quality UI trajectory data, but collecting it is prohibitively expensive: the paper cites Xie et al. (2024), who report that designing 360+ realistic computer-use tasks took more than 1,800 human hours. The authors argue that environment diversity is a chief driver of agent robustness, yet running many parallel real UI environments is bottlenecked by resource demands, network instability, and a lack of native distributed support. Their goal is to replace real-environment exploration with an LLM-based simulator that can generate diverse, plausible, and previously unseen UI states on demand, and then to scale that synthesis strategically rather than blindly.

Key Contributions

1. UI-Simulator, a scalable trajectory-synthesis paradigm built on an LLM-based digital world simulator that generates structured UI states and state transitions, combined with a step-wise guided rollout process and a trajectory wrapper that produces instructions, actions, and step-wise reasoning.
2. A hybrid transition design: multi-step LLM simulation (predict an overview of the next state, generate a rich natural-language draft, then convert the draft into a structured accessibility-tree format with coordinates) alongside rule-based transitions for deterministic actions such as scroll, plus an optional retrieval-augmented mode that conditions generation on a small amount of prior test-environment experience.
3. UI-Simulator-Grow, a targeted scaling paradigm that ranks candidate tasks by teacher-forcing loss on dynamically updated validation sets, selects the middle 25%–75% percentile band, synthesizes lightweight instruction variants of those tasks, and uses replay-based continual learning to avoid forgetting.
4. Empirical validation on WebArena (812 complex web navigation tasks) and AndroidWorld (116 daily mobile usage tasks), showing competitive or superior success rates against open-source agent baselines despite using the weaker GPT-4o-mini teacher model rather than GPT-4o.

Main Findings

• Simulation alone beats the base model: UI-Simulator-F, trained with no exposure to the real test environment, raises AndroidWorld success rate from 0% (Qwen-2.5-7B-Instruct base) to 9%, and reaches 6.28% on WebArena (Llama-3-8B-Instruct base, up from 2.34%). It also outperforms OS-Genesis on WebArena (6.16%), which is trained on trajectories synthesized directly from the WebArena test environments using GPT-4o.

• Retrieval-augmented simulation closes the gap to much larger models: UI-Simulator-R reaches 6.40% on WebArena (on par with Gemini Pro at 7.12% being the comparison cited, and close to Qwen-1.5-72B-Instruct at 7.14%) and 12.9% on AndroidWorld, which the paper notes is on par with GPT-4o's 11.7%, despite being built on an 8B-scale LLM.

• Better than real-environment synthesis with the same pipeline: Training on an equivalent number of trajectories collected directly on the real test environment yields only 4.31% on WebArena (both -F and -R settings) versus 6.28% / 6.40%. The authors attribute this to real environments failing to provide consistent useful state transitions, returning "Search not found", and restricting access to account/settings pages.

• Much better use of limited environment experience: When test-environment experience is held equal, UI-Simulator-R achieves roughly 4 times (WebArena) and 2.5 times (AndroidWorld) the performance of OS-Genesis (1.48% and 5.2% respectively).

• Improved robustness to UI perturbation: Under randomly shuffled layout structures that preserve content and solutions, UI-Simulator-F changes from 6.28% to 5.54% on WebArena and 8.6% to 8.7% on AndroidWorld, while OS-Genesis drops from 6.16% to 4.43% on WebArena.

• Guided rollout and multi-step simulation both matter: Removing step-wise task controls drops performance from 6.28% to 1.72% on WebArena and 8.6% to 5.2% on AndroidWorld. Task diversity, measured via PCA effective dimension at ≥90% explained variance on RoBERTa-large embeddings, falls from 153 with task control to 118 without. Replacing multi-step simulation with single-step simulation drops performance by approximately 2.4% and 3.8% on WebArena and AndroidWorld.

• Targeted scaling is more data-efficient than standard scaling: UI-Simulator-Grow shows a steeper improvement curve, matches Qwen-1.5-72B-Instruct by the third iteration, surpasses Llama-3-70B-Instruct, and does so using only 66% of the original UI-Simulator-R trajectories. Its headline claim is matching Llama-3-70B-Instruct performance using only Llama-3-8B-Instruct as the base model, with 7.14% WebArena and 13.4% AndroidWorld.

• Gains spread across task categories: Figure 4 reports a consistent upward trend across most major WebArena task categories, with code repository operations reaching tasks that neither standard scaling nor earlier iterations could solve.

Methodology in Plain English

The authors treat the agent's environment as something an LLM can imagine rather than something they must actually run. Most digital UIs can be described as a structured accessibility tree: elements with text content, spatial coordinates, and dynamic attributes like focus status. Given a current state and an action, the authors prompt GPT-4o-mini in a three-step chain: first predict a short overview of the resulting page (for example, "a search results page for the keyword sneakers"), then generate an unconstrained natural-language draft of the page's contents, then convert that draft into a structured tree with automatically assigned coordinates. They use 4 in-context examples for the overview step and 1 example each for the other two steps. For deterministic actions like scrolling, they fall back on rule-based transitions.

On top of this simulator, a teacher agent (GPT-4o-mini) explores without any preset user instruction, in a step-wise guided rollout. At each step, the teacher proposes a short-horizon "task control" (for example, "navigate to my account page"); once the teacher judges that control done, it proposes the next one. This prevents the teacher from repeatedly clicking the same one or two elements, which the paper identifies as a bias that homogenizes trajectories. The teacher also emits a reasoning thought, an action, and a step summary. A final trajectory wrapper summarizes what was accomplished and turns that into a user instruction, then rewrites the reasoning thoughts so they are conditioned on that instruction.

For retrieval-augmented simulation, the authors build a small offline corpus of real transitions from the test environment and retrieve the most relevant one using a three-stage pipeline: BM25 over action histories, then GPT-4o as a semantic retriever, then BM25 again over a composite key of state plus action history. That retrieved state is passed to the simulator as an extra conditioning signal.

UI-Simulator-Grow adds an outer loop. Each iteration computes teacher-forcing cross-entropy loss between teacher and student predictions, ranks tasks by that loss, and selects tasks in the 25%–75% percentile band as targets, excluding the too-easy and too-hard. It then synthesizes variants of those tasks by rewriting instructions while preserving action types and flow (for example, "search running shoes" becomes "search slippers"). To avoid forgetting across iterations, it uses a replay strategy following Dynosaur, selecting representative prior tasks via cosine similarity over Sentence Transformer (RoBERTa-large) embeddings of task instructions.

Evaluation uses WebArena and AndroidWorld, reporting success rate at temperature 0.6, with Llama-3-8B-Instruct as the WebArena base model and Qwen-2.5-7B-Instruct for AndroidWorld because of its context length support beyond 8192 tokens. The authors note they reproduce baselines under the original WebArena evaluation settings rather than BrowserGym or lite versions.

Why This Matters

Impact on research. The paper reframes data collection for digital agents as an environment-simulation problem rather than a scraping or human-annotation problem. Its finding that simulated environments can outperform trajectories collected on the real test environment is a substantive challenge to the assumption that fidelity to the deployment environment is what matters most; the authors argue breadth of interaction scenarios matters more. The targeted scaling results also connect agent training to curriculum-style, loss-driven data selection.

Real-world applications:

• Training web and mobile automation agents in domains where running parallel real environments is expensive, unstable, or rate-limited.
• Building agents for internal or gated applications (account pages, admin settings) where real-environment trajectories cannot be collected due to access restrictions or a small number of available user accounts.
• Bootstrapping agents for new or rapidly changing products where no historical interaction data exists and only a small amount of target-environment experience can be gathered.
• Adapting agents to a specific deployment target quickly with a small retrieval corpus rather than a full data-collection campaign.

Industry relevance. The economics are the headline: replacing more than 1,800 human hours of task design with LLM synthesis, improving data efficiency by using only 66% of the original trajectories, and reaching 70B-class performance from an 8B base model all directly reduce the cost of building digital agents. The reliance on GPT-4o-mini as the teacher (with GPT-4o used only as a retriever in the retrieval-augmented variant) shows competitive results are achievable without the most expensive teacher model.

Future Directions

• Pairing UI-Simulator with stronger teacher agents. The authors repeatedly note that their results use the weaker GPT-4o-mini, and explicitly state that results "highlight the potential of UI-Simulator when paired with stronger teacher agents" — the upper bound of the approach under a GPT-4o-class teacher is untested in the reported numbers.
• Extending beyond web and mobile. The paper claims the simulator construction process "can be applied to a variety of digital and even non-digital agent tasks," but only WebArena and AndroidWorld are evaluated.
• Scaling the targeted paradigm further. UI-Simulator-Grow is demonstrated over three iterations; whether the loss-ranked selection and replay strategy continue to pay off over longer horizons, and how the dynamic validation set should evolve, remains open.
• Richer robustness characterization. The paper's perturbation experiment only shuffles layout structure while preserving content and solutions. Whether simulated training transfers to other classes of distribution shift — different platforms, visual rendering, or genuine functional changes — is not reported.

Target Audience

Researchers and engineers working on LLM-based digital agents, agentic training-data synthesis, and world models, particularly those who need to train web or mobile UI agents without access to large-scale real-environment trajectory data. It is also relevant to practitioners interested in data-efficient training and curriculum/targeted data selection, and to readers following the broader debate over whether simulated environments can substitute for real ones in agent training.

Authors’ abstract

Digital agents require diverse, large-scale UI trajectories to generalize across real-world tasks, yet collecting such data is prohibitively expensive in both human annotation, infra and engineering perspectives. To this end, we introduce $\textbf{UI-Simulator}$, a scalable paradigm that generates structured UI states and transitions to synthesize training trajectories at scale. Our paradigm integrates a digital world simulator for diverse UI states, a guided rollout process for coherent exploration, and a trajectory wrapper that produces high-quality and diverse trajectories for agent training. We further propose $\textbf{UI-Simulator-Grow}$, a targeted scaling strategy that enables more rapid and data-efficient scaling by prioritizing high-impact tasks and synthesizes informative trajectory variants. Experiments on WebArena and AndroidWorld show that UI-Simulator rivals or surpasses open-source agents trained on real UIs with significantly better robustness, despite using weaker teacher models. Moreover, UI-Simulator-Grow matches the performance of Llama-3-70B-Instruct using only Llama-3-8B-Instruct as the base model, highlighting the potential of targeted synthesis scaling paradigm to continuously and efficiently enhance the digital agents.