LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents

AI summary

Overview

Research area: Multimodal vision-language models applied to GUI agents (computer-use / screen-control agents), specifically masked discrete diffusion language models as an alternative to autoregressive decoding.

Technical level: Advanced. The paper assumes familiarity with autoregressive versus diffusion decoding, mixture-of-experts (MoE) architectures, vision-language projectors, GUI grounding, and interactive agent benchmarks.

Scope: The paper introduces LLaDA-UI, a 16.7B-parameter MoE block-wise diffusion vision-language GUI agent, and evaluates it on general multimodal benchmarks plus six GUI grounding and navigation benchmarks.

What This Paper Is About

Most vision-language GUI agents generate actions one token at a time, left to right, which limits parallelism and adds inference latency. The authors ask whether a masked diffusion language model — one that refines a whole block of output tokens in parallel — can be turned into a capable GUI agent that perceives screens, reasons, and emits executable actions in real time. LLaDA-UI is their answer: they pair a native-resolution vision encoder with the LLaDA2.0-mini-base diffusion language backbone, pre-train on general multimodal data, then fine-tune on large-scale GUI trajectories.

Key Contributions

1. A strong LLaDA-based vision-language foundation. Large-scale multimodal pre-training aligns a native-resolution SigLIP-initialized ViT with the LLaDA2.0-mini-base diffusion backbone, producing competitive results across representative general multimodal benchmarks while using only 145B pre-training tokens.

2. The first practical open-source diffusion vision-language GUI agent. GUI-agent supervised fine-tuning on large-scale GUI data yields a model that substantially outperforms Qwen2.5-VL-7B and surpasses Qwen3-VL-8B on four of six reported grounding and navigation benchmarks.

3. Detailed empirical analysis of diffusion GUI inference. The paper reports diffusion-specific inference settings (block size, denoising steps, EOS handling), a paired non-cached API latency study against Qwen3-VL-8B, and structured-action validity statistics.

4. Failure-mode characterization. The authors document where LLaDA-UI breaks down — repeated actions and unproductive loops on long-horizon tasks, and degraded grounding on very large screenshots with small targets.

Main Findings

• General multimodal performance before GUI fine-tuning: The foundation checkpoint reaches MMMU (val) 56.2, MMMU-Pro (standard) 36.0, MathVista (mini) 70.5, WeMath 34.6, MathVision (mini) 27.5, MathVerse (mini) 48.0, and MathVerse (vision only) 39.7. On general tasks it reports SimpleVQA 44.3, HallusionBench 50.7, MMBench (en) 80.5, MMBench (cn) 77.7, MMStar 61.3, and RealWorldQA 63.5.

• OCR, chart, and other tasks: ChartVQA 84.8, DocVQA 91.5, InfoVQA 71.5, CharXiv (DQ) 72.2, CharXiv (RQ) 35.6, OCRBench 855, AI2D (w mask) 80.2, CountBench 86.2, and VLRewardBench 50.8.

• Token efficiency claim: The foundation is trained on only 145B tokens, compared with Qwen2.5-VL's 4.1T and Qwen3-VL's 2.2T, yet the authors state it establishes a new state of the art among diffusion-based MLLM baselines. Relative to SDAR-VL-8B, they report leads of 12.8%/13.6%/31.1% on MathVista, MathVision, and MathVerse, and 2.5%/8.6%/17.8% on ChartQA, CharXiv-DQ, and OCRBench.

• GUI grounding and navigation (Table 3): LLaDA-UI scores 90.7 on ScreenSpot-V2, 52.9 on ScreenSpot-Pro, 53.5 on AndroidWorld, 25.6 on MobileWorld, 29.4 on OSWorld-Verified, and 56.9 on WebVoyager.

• Comparisons: For reference, the paper reports Qwen2.5-VL-7B at 86.0 / 26.8 / 25.5 / 7.0 / 3.0 / 11.0 and Qwen3-VL-8B at 93.0 / 52.7 / 47.9 / 9.4 / 33.9 / 45.2 on the same six benchmarks. LLaDA-UI exceeds Qwen2.5-VL-7B on every reported benchmark and beats Qwen3-VL-8B on ScreenSpot-Pro, AndroidWorld, MobileWorld, and WebVoyager; ScreenSpot-V2 is close and OSWorld-Verified remains the main gap.

• Specialized agent baselines: UI-TARS-7B reports 91.6 (SS-V2) and 35.7 (SS-Pro); GUI-G2-7B reports 93.8 and 47.5; GUI-Owl-7B reports 92.8, 54.9, 66.4, and 7.7; UI-TARS-1.5-7B reports 91.6, 35.7, and 30.0; UI-Venus-1.0-7B reports 94.1, 50.8, 49.1, and 8.5; UI-Venus-1.5-2B reports 92.8, 57.7, 55.6, and 56.4; Holo2-8B reports 93.2, 58.9, 60.4, and 80.2; Step-GUI-4B reports 93.6, 60.0, 63.9, and 47.8; MAI-UI-2B reports 92.5, 57.4, and 49.1. GPT-4o reports 20.1 (SS-V2), 0.8 (SS-Pro), and 30.6 (AndroidWorld).

• Latency advantage (non-cached paired API study): On Web inputs, LLaDA-UI averages 4.764 s (median 4.801, range 4.474–5.156, 144 tokens) versus Qwen3-VL-8B at 17.050 s (median 16.458, range 15.873–20.044, 55.2 tokens), a 3.579x mean speedup. On OSWorld, 6.379 s (median 6.313, range 5.911–6.741, 129 tokens) versus 43.545 s (median 42.787, 70.0 tokens), a 6.826x mean speedup. On MobileWorld, 5.921 s (median 5.944, range 5.434–6.299, 67 tokens) versus 53.000 s (median 52.394, 65.2 tokens), an 8.951x mean speedup. Median speedups are 3.428x, 6.778x, and 8.814x respectively.

• Cache caveat: Qwen3-VL-8B has a separate exact-replay cache-hit mode measured at 0.890–1.284 s on Web, 1.444 s on OSWorld, and 1.746–1.794 s on MobileWorld, whereas the evaluated LLaDA-UI SGLang stack exposes no comparable cache-hit path. The authors report these separately because normal GUI interaction produces a new screenshot at every step.

• Decoding ablation (AndroidWorld, intermediate checkpoint): The baseline (early-stop EOS, block 32, 32 steps) succeeds at 42.7% (50/116) with 14.9 average actions. Two-stage decoding reaches 50.9% (58/115) with 13.7 average actions. Disabling EOS gives 52.6% (61/116) with 15.8 average actions — the paper states this improves success by 9.9 percentage points in this controlled comparison. Using block 64 with 16 steps drops success sharply to 33.0% (38/115) with 15.1 average actions.

• Structured-action validity (116-task AndroidWorld run, 1,849 steps): Parse rate 98.65% (1,824/1,849), schema-valid rate 98.59% (1,823/1,849), coordinate-valid rate 100.00% (1,457/1,457), exact-repetition rate 18.82% (348/1,849), and multiple-action rate 1.30% (24/1,849). Task success for that run is reported as 51.72% (60/116 tasks), and the authors note these diagnostic values are not the final benchmark result.

• Long-horizon degradation: Exact repetition rises from 5.41% of steps in successful trajectories to 26.70% in failed ones. Success falls from 62.96% on tasks with at most five optimal actions to 14.29% beyond 20 actions. Stratified by annotated optimal task length: 1–5 actions, 54 tasks, 34 successes, 62.96%; 6–10 actions, 35 tasks, 17 successes, 48.57%; 11–20 actions, 20 tasks, 8 successes, 40.00%; more than 20 actions, 7 tasks, 1 success, 14.29%.

• Qualitative traces: The appendix-adjacent figures show complete trajectories of 20 steps (WebVoyager, constrained flight search), 10 steps (OSWorld, transferring a formatted Calc table to Writer), and 12 steps (MobileWorld, reading an event time from email and setting an alarm one hour earlier).

Methodology in Plain English

LLaDA-UI is built from three parts. The language brain is LLaDA2.0-mini-base, a 16B-parameter mixture-of-experts model that generates text by a block-wise masked diffusion process: instead of emitting one token at a time, it starts from masked tokens in a block and denoises several of them in parallel, refining the block over a number of steps. The eyes are a vision transformer initialized from pretrained SigLIP that accepts images at native resolution and uses 2D rotary positional embeddings to capture spatial layout — important because GUI screenshots are spatially structured. A projector then compresses every four adjacent image features and maps them into the language model's embedding space with a two-layer perceptron, following the Qwen2.5-VL design.

Training happens in two macro-stages. The first is general multimodal pre-training, split into three curriculum phases: Stage 0 aligns vision and language by training only the projector on image-caption and visual knowledge data (5B tokens); Stage 1 unfreezes everything and adds interleaved image-text, OCR, counting, and grounding data plus pure text (70B tokens); Stage 2 adds reasoning-intensive data such as multimodal VQA, multimodal math, and agent tasks (70B tokens). All stages use 16K sequence length, and the training stack (dFactory on VeOmni) supports sequence packing so short samples are concatenated without crossing sample boundaries.

The training objective is a block diffusion cross-entropy loss that only scores masked tokens, weighted by a diffusion-derived time factor. Three practical tricks support it: auxiliary-loss-free MoE load balancing with a bias update normalized RMSNorm-style and gate activations scaled by 2.5; mask-token reweighting with an inverse square-root weight per sample to avoid long targets dominating gradients; and complementary masking, where two corrupted views per target are logical inverses so every token is observed once and predicted once.

The second macro-stage is GUI-agent supervised fine-tuning on more than 6M samples covering mobile navigation (more than 100 Chinese apps and more than 70 English apps), desktop and web workflows, and grounding data. Tasks were posed, decomposed into atomic GUI capabilities, recombined into harder tasks, executed in instrumented environments, and validated into training trajectories. Responses use a tagged <think>...</think><action>...</action> format for navigation, while grounding returns a target point directly. Fine-tuning runs three epochs at a peak learning rate of 1e-5 with cosine decay, a 3% warmup ratio, 16K maximum sequence length, mixed precision, micro-batch size one per GPU, 32–64 H100 GPUs, and a maximum image-pixel budget of approximately 12.8M pixels. Spatial outputs are normalized to the range [0, 1000], parsed into platform actions, and executed by each benchmark's runner.

For evaluation, agents receive the same screenshot observation, interaction history, action space, environment feedback, and action budget whenever supported. Both a standalone Hugging Face inference path and an OpenAI-compatible SGLang serving path (two data-parallel workers on a two-GPU host) are validated end to end.

Why This Matters

Impact on research: This is the first work the authors identify as making a diffusion language model operate effectively as a vision-language GUI agent. It suggests masked diffusion decoding, previously explored mainly in text-only modeling and static GUI grounding, can support long-horizon interactive behavior — and it provides a latency argument for doing so. It also reports a foundation trained on 145B tokens that is competitive on a broad multimodal suite, which is relevant to arguments about pre-training data efficiency.

Real-world applications:

• Mobile assistants that operate apps directly from screenshots, including the more than 100 Chinese apps and more than 70 English apps covered by the training data.
• Desktop automation, such as transferring a formatted spreadsheet table into a word processor and saving the document.
• Web agents that carry out multi-step constrained tasks, such as flight search with multiple constraints.
• Screen grounding utilities that convert a natural-language instruction into a clickable coordinate, useful for accessibility and UI testing.

Industry relevance: Latency is a first-order constraint for interactive agents, and the reported non-cached mean speedups of 3.579x to 8.951x over Qwen3-VL-8B address a real deployment bottleneck. The larger MoE model with a diffusion decoder also completes inference faster than the denser autoregressive Qwen3-VL-8B in this comparison, which is an unusual efficiency profile. The model is open-sourced on GitHub and HuggingFace, and the paper documents serving paths (SGLang, Hugging Face), which lowers the barrier for teams building on it.

Future Directions

• Long-horizon recovery. Repeated actions and unproductive loops are the dominant failure mode on long tasks, with repetition rising from 5.41% of steps in successful trajectories to 26.70% in failed ones and success dropping to 14.29% beyond 20 optimal actions. The authors frame improving recovery as a main direction.

• High-resolution perception and pointing. OSWorld exposes a different bottleneck: screenshots are much larger and contain smaller targets, revealing insufficient high-resolution perception and inaccurate pointing.

• Decoding robustness. Interactive performance is sensitive to block size and denoising step count — the study shows success falling from 42.7% to 33.0% when moving from block 32/32 steps to block 64/16 steps — so the authors identify robustness to these inference hyperparameters as an open problem. The EOS early-stopping finding is also flagged as something that should be validated for the released model.

• Compression of the reported benchmarks. The paper does not report results on every GUI benchmark it discusses for every baseline (several entries in Table 3 are blank, including GPT-4o's navigation results and several specialized agents' MobileWorld scores), so broader and more complete cross-benchmark evaluation remains open.

Target Audience

This paper is most useful to researchers and engineers working on GUI agents, multimodal foundation models, and efficient inference for vision-language systems. It will particularly interest teams evaluating diffusion decoding as an alternative to autoregressive generation, practitioners concerned with interactive latency budgets, and researchers studying agent failure modes such as action repetition and long-horizon degradation. Readers focused on general multimodal capabilities alone will find the pre-training comparison of secondary interest relative to the GUI-agent results and the diffusion-specific analysis.

Authors’ abstract

Diffusion large language models (dLLMs) achieve high decoding efficiency through block-parallel, arbitrary-order generation, making them attractive for latency-sensitive applications. GUI agents represent a natural testbed for this paradigm, as they must repeatedly perceive screen states and emit structured, spatially grounded actions in real time. However, whether dLLMs can be extended into capable multimodal GUI agents while preserving their parallel decoding advantage remains an open question. We present LLaDA-UI, a 16.7B-parameter MoE-based, block-wise diffusion vision-language GUI agent. LLaDA-UI follows a two-stage training pipeline: general multimodal pre-training aligns a native-resolution vision encoder with the LLaDA2.0-mini-base diffusion language backbone, followed by GUI-agent supervised fine-tuning on diverse mobile, desktop, web, and grounding data. Across widely adopted grounding benchmarks and navigation benchmarks spanning multiple platforms, LLaDA-UI substantially outperforms Qwen2.5-VL-7B and surpasses Qwen3-VL-8B on four of six reported GUI benchmarks. These results establish block-wise diffusion as a practical generative paradigm for multimodal GUI agents.