Generative Visual Code Mobile World Models

AI summary

Overview

Research area: Mobile Graphical User Interface (GUI) agents and generative world models — specifically, training Vision-Language Models (VLMs) to predict the next screen state of a mobile app given the current screen and a user action.

Technical level: Advanced. The paper assumes familiarity with Vision-Language Models, supervised fine-tuning, world models in Markov Decision Processes, diffusion-based image generation, and VLM-as-a-Judge evaluation.

Scope: The paper proposes a new paradigm for mobile GUI world modeling — predicting next states as executable web code rather than pixels — and introduces the models (gWorld 8B/32B), a data generation framework, and the MWMBench benchmark.

What This Paper Is About

Mobile GUI world models try to predict what a phone screen will look like after an action, which can help train and run mobile agents. Existing approaches face a trade-off: text-based world models lose visual detail, while visual (pixel-generating) world models cannot render text precisely and require slow, complex pipelines built from many external models (OCR, masking, diffusion, extra VLM calls).

The goal of this work is to build a single self-contained model that predicts the next GUI state as renderable web code that a browser turns into pixels — retaining the language model's text-rendering strength while exploiting its pre-training on structured web code for visual fidelity.

Key Contributions

1. A new world-modeling paradigm — "visual world modeling via renderable code generation." A single VLM predicts the next GUI state as executable web code instead of pixels. The paper states this is the first demonstration that a code-based representation can be used for mobile GUI world models.

2. gWorld (8B and 32B) — described as the first open-weight, single self-contained visual mobile GUI world models operating via renderable code generation. They are post-trained from Qwen3 VL 8B and 32B.

3. A data generation framework that automatically synthesizes code-based world-model training data from existing mobile-agent policy trajectories, comprising three steps: (1) repurposing offline policy trajectories into transition triplets, (2) cross-modal re-labeling of the next state from pixels to renderable web code, and (3) synthesizing reasoning traces using look-ahead access to the ground-truth next state. A dataset of 260K samples was generated from AitW, GUIOdyssey, AndroidControl, and AMEX, using Gemini 3 Flash as the frontier data-generation model.

4. MWMBench, a benchmark curating four in-distribution and two out-of-distribution mobile GUI world modeling evaluations, which preserves the native visual modality, keeps actions in original coordinate space (rather than converting them to text), and is described as the first to assess zero-shot generalization on held-out OOD sets.

Main Findings

• New accuracy-versus-size Pareto frontier. gWorld 32B and 8B achieve the best and second-best Instruction Accuracy (IAcc.) respectively across all six benchmarks. Average IAcc. is 79.6% for gWorld 32B and 74.9% for gWorld 8B. The paper reports gains of +45.7% and +27.1% in average IAcc. over the base models Qwen3 VL 8B and 32B respectively (Qwen3 VL 8B averages 29.2%, Qwen3 VL 32B averages 52.5%).

• Outperforms much larger models. The paper states gWorld outperforms 8 frontier open-weight models up to 50.25× larger. GLM-4.6V 106B and Llama 4 402B-A17B are the next best baselines, exceeding gWorld 8B's parameter count by factors of 13.25× and 50.25×. Per-benchmark, gWorld 8B scores higher average IAcc. (74.9%) than Qwen3 VL 235B-A22B (67.4%) and GLM-4.6V 106B (67.4%).

• Image-generation baselines fail at action-conditioned dynamics. Image-generation models (Qwen-Image-Edit 20B at 13.4% average IAcc., Emu3.5 34B at 25.8%) score high on visual similarity but low on IAcc. The Pearson correlation between ground-truth transition similarity Sim(S_t, S_{t+1}) and output similarity is ρ = 0.92 for Emu3.5 34B and ρ = 0.74 for Qwen-Image-Edit 20B, versus ρ ≈ 0.4 for gWorld — indicating image models rely on near-identity copying of the input rather than modeling the action's effect.

• Low structural failure. gWorld averages 1.4% (8B) and 0.6% (32B) render failure across the six benchmarks, which the paper describes as virtually eliminating structural errors. Baselines such as Qwen3 VL 235B-A22B average 40.1% render failure.

• Data scaling follows a power law. Training set sizes of 37K, 77K, 129K, and 240K produced monotonic gains; the fit follows a power law (y = ax^b) with an average R² of 0.948 and R² ≥ 0.94 for most splits. The paper computes a maximum attainable dataset size of 3.7 million transitions from the four existing offline trajectory datasets, and projects further gains from the remaining trajectories.

• Each pipeline component helps (ablations). For next-state code generation, the paper's method yielded 100% renderable code versus 97% for a naïve alternative, with IAcc. of 100% vs 94.60% (judge: Gemini 3 Pro) and 86.70% vs 84.80% (judge: Claude 4.5 Opus) on 100 sampled instances (25 from each dataset). Look-ahead reasoning traces also outperformed non-look-ahead traces across all five benchmarks tested, with both models trained on 37K samples on Qwen3 VL 8B.

• Human evaluation supports the automatic metric. With 12 annotators, 100 samples, three annotations each (300 total), gWorld 32B and 8B ranked first and second with average ranks of 1.68 and 2.16. Pairwise win rates were 77.5% and 61.4%. Spearman ρ = 0.806 and Kendall τ = 0.600.

• Robust on photo-realistic states. Photo-realistic transitions are 17.4% of in-distribution test transitions, text/structure-heavy transitions 81.5%, and 1.1% ambiguous. gWorld 8B scored 75.26% IAcc. on photo-realistic states versus 75.92% on others (a 0.66% drop); gWorld 32B scored 74.98% versus 79.68% (a 4.70% drop), still outperforming all baselines in both categories.

• Better world models improve downstream policies. Using the M3A agent with K=3 action candidates and two backbone policies (Gemini 2.5 Flash, GPT-5 Mini), gWorld 8B gave the largest gains over the M3A baseline, averaging 69.9 and 67.2 step-wise accuracy. The delta versus Qwen3 VL 8B was +22.4 and +21.8 points on average. On average, a 1.0 percentage point increase in world modeling performance translated to a 0.49 percentage point improvement in downstream policy.

• Much lower latency. Using vLLM on 4× H200 GPUs, gWorld 32B and 8B reach approximately 5,000 and 20,000 tokens/sec — generation latencies of 1.0s and 0.25s per state. Rendering adds 0.3s per state after a one-time 1s browser launch, for end-to-end latencies of 1.30s and 0.55s. VIMO reports 160s per state; even subtracting VIMO's reported 38s local model-execution time leaves 122s, still 94× slower than gWorld 32B.

Methodology in Plain English

Instead of teaching a model to draw the next screenshot pixel by pixel, the researchers teach it to write the web code that would produce that screen. A browser then renders the code into an image. Because the underlying models are language models, they can write precise text (a known weakness of image generators) and have seen a lot of structured web code during pre-training.

To create training data, the authors took existing collections of mobile-agent interactions (recorded screens plus the actions taken). From each episode they formed examples of the form "current screenshot + action → next screenshot." A strong frontier model (Gemini 3 Flash) then converted each next screenshot into web code, and separately wrote a short natural-language reasoning trace describing what changes between the two screens — written with knowledge of the true next screen, which the authors call "look-ahead." The model is fine-tuned to output the reasoning trace first, then the code.

For evaluation, they built MWMBench from six data sources. Four are in-distribution (AitW, GUIOdyssey, AndroidControl, AMEX — 500 sampled instances each), and two are new out-of-distribution sets: AndroidWorld (automatically collected offline trajectories) and KApps (manually curated Korean-language trajectories). The primary metric, Instruction Accuracy, asks three frontier VLM judges (GPT-5 Mini, Claude 4.5 Haiku, Gemini 3 Flash) to give a binary pass/fail verdict on whether the generated next state is consistent with the state-action pair; the paper reports high inter-judge agreement. A secondary similarity metric uses averaged DINO v1 and v2 embeddings. Un-renderable code is classified as an automatic failure.

Why This Matters

Impact on research: The paper reframes GUI world modeling as a code-generation problem rather than an image-generation problem, which sidesteps the multi-model pipelines (OCR, masking, diffusion, extra VLM calls) of prior work. It also releases open weights, which the authors note prior systems such as VIMO did not do, and it shows a measurable link between world-model quality and downstream agent performance. On directly comparable Similarity v1 scores, gWorld 8B and 32B average 81% and 81.9% versus VIMO's 74%.

Real-world applications:

• Safer agent training — simulating consequential actions such as financial transactions without executing them on a real device.
• Cheap data generation for hard-to-reach states — recursively generating trajectories from already-collected states to cover deep application states.
• Scalable reinforcement learning — replacing the 1:1 device-policy coupling (one Android emulator per rollout, with GPUs idle during >2s action execution) with parallel, compute-bound rollout generation.
• Accessibility and interface prototyping — the impact statement cites enhancing digital accessibility for users with impairments and lowering compute requirements for GUI automation research.

Industry relevance: Mobile computing is ubiquitous, with an estimated 8.9 billion mobile subscriptions worldwide. The latency result (0.55s/1.30s versus VIMO's 160s) and the plug-in improvement to existing agent scaffolds (M3A, with Gemini 2.5 Flash and GPT-5 Mini backbones) make the approach practical for production GUI agents. The paper was fully funded by Trillion Labs.

Future Directions

• Scaling to the remaining trajectory budget: The authors compute a maximum attainable dataset of 3.7 million transitions from the four existing datasets and project significant further gains from the unused portion, based on the power-law fit.
• World models for synthetic data scaling and scalable RL: Using WMs to avoid risky actions, expand deep-state coverage, and break the device-policy coupling bottleneck in online GUI RL.
• Direct comparison with existing visual world models: The authors could not compare directly against VIMO because its diffusion model weights were not released at the time of writing; a direct comparison remains open.
• Limitations and future work: The paper states that limitations and future directions are discussed in Appendix E, which is not included in the provided content, so those specifics are not reported here. The impact statement also raises dual-use risks (e.g., automated fraud) and argues that open research is needed to develop robust safety measures.

Target Audience

Researchers and engineers working on mobile GUI agents, world models, and vision-language models, particularly those interested in agent data generation, simulation, or reinforcement learning for GUI tasks. It is also relevant to practitioners who need scalable, low-latency environment simulation for agent training pipelines. Because the paper is heavy on experimental detail and modeling terminology, readers without a background in VLMs or reinforcement learning will need to consult the cited prior work first.

Authors’ abstract

Mobile Graphical User Interface (GUI) World Models (WMs) offer a promising path for improving mobile GUI agent performance at train- and inference-time. However, current approaches face a critical trade-off: text-based WMs sacrifice visual fidelity, while the inability of visual WMs in precise text rendering led to their reliance on slow, complex pipelines dependent on numerous external models. We propose a novel paradigm: visual world modeling via renderable code generation, where a single Vision-Language Model (VLM) predicts the next GUI state as executable web code that renders to pixels, rather than generating pixels directly. This combines the strengths of both approaches: VLMs retain their linguistic priors for precise text rendering while their pre-training on structured web code enables high-fidelity visual generation. We introduce gWorld (8B, 32B), the first open-weight visual mobile GUI WMs built on this paradigm, along with a data generation framework (gWorld) that automatically synthesizes code-based training data. In extensive evaluation across 4 in- and 2 out-of-distribution benchmarks, gWorld sets a new pareto frontier in accuracy versus model size, outperforming 8 frontier open-weight models over 50.25x larger. Further analyses show that (1) scaling training data via gWorld yields meaningful gains, (2) each component of our pipeline improves data quality, and (3) stronger world modeling improves downstream mobile GUI policy performance.