Learn from Move: the Next Step for GUI Agents

AI summary

Overview

Research area: Machine learning for multimodal GUI (graphical user interface) agents — specifically benchmark design, interaction-interface design, and post-training (supervised fine-tuning and reinforcement learning) for computer-use agents.

Technical level: Advanced. The paper combines a formal identifiability taxonomy with a new interactive benchmark, a fine-grained action schema, and SFT/RL training pipelines.

Scope: The paper introduces LatentGUIWorld, a six-task benchmark that measures whether GUI agents can identify hidden, episode-specific environment properties from their own actions, and proposes "Move," a separated move/click/release interface plus training data that teaches agents to explore and adapt.

What This Paper Is About

Existing GUI agent benchmarks mostly make tasks harder by making them longer — more steps, more applications, more sustained execution. This paper argues that a distinct and under-tested difficulty is whether an agent can act on an unfamiliar interface, observe what its action caused, and infer a hidden rule (for example, how sensitive a drag or rotation is) before committing to a goal-directed action. The authors build a benchmark that isolates this ability, plus a training interface and dataset that let a 9B model learn to do it.

Key Contributions

1. LatentGUIWorld, a benchmark of six short GUI tasks arranged in three matched pairs, organized by the minimum evidence needed for control: I0 (prior knowledge and observed context suffice), I1 (a probe outcome directly reveals hidden semantics), and I2 (the action–outcome relationship is required to identify hidden dynamics).
2. The Move interface, which separates cursor movement from mouse-down and mouse-up events so that each primitive returns a new observation and a new decision point before the agent commits through a click or release.
3. A feedback-rich training formulation producing multi-step exploration demonstrations (rule-based action sequences plus teacher-generated Think rationales), with ablations showing that both action granularity and trajectory structure matter.
4. Empirical demonstration that feedback-conditioned control can be learned jointly across the matched interaction regimes, and that the same interaction format also improves standard GUI grounding after RL.

Main Findings

• Frontier models fail most on dynamics tasks: The paper reports that frontier models struggle especially on I2 tasks, where success requires interpreting a visual transition together with the action that produced it. The per-model benchmark table (Table 2) is not included in the provided text, so individual frontier-model scores are not reported here.

• SFT with the Move format is competitive: After supervised fine-tuning, LatentLearner-9B (built on Qwen3.5-9B) achieves 50.0% overall success on LatentGUIWorld, described as comparable to several frontier models.

• Action granularity matters: In ablations, overall success is 50.0% with the full interface and data, versus 19.6% for bundled click/drag and 27.6% when each movement phase contains a single supervised move.

• Joint RL substantially improves egocentric control: Egocentric Drag and Ten-Choice success rises from 41.3% and 40.7% after SFT to 92.0% and 75.3% after joint RL, while exocentric success remains 96.7% and 100.0%.

• Rotation training generalizes across geometries: Paired Rotation training reaches 27.3% and 11.3% on the inner and outer rings, improving on the inner-ring specialist and matching the outer-ring specialist.

• Models learn to compensate for episode-specific dynamics: In egocentric Drag, the Pearson correlation between error-normalized commands and ideal gain compensation (1/g) increases from 0.610 after SFT to 0.803 after paired RL. Behavioral analysis reveals multi-step exploration.

• The Move format transfers to standard grounding: Applying the same interaction format to grounding yields 67.37% mean accuracy across five benchmarks after RL, compared with 66.72% for conventional grounding RL.

• Benchmark scale: The fixed test suite contains 150 episodes per variant, totaling 900 episodes, with a separate validation set of the same size (900 episodes) used for checkpoint selection. Drag uses 75 IID cases with training shapes and 75 OOD cases with held-out shapes per variant.

• Training scale: LatentLearner-9B is trained jointly on 24,000 successful demonstration trajectories (4,000 per environment), yielding 154,713 supervised examples.

Methodology in Plain English

The authors first define what it means for an environment property to be "hidden": it stays fixed within an episode, cannot be read off the current screenshot or prior knowledge, changes what the correct next action is, and can be identified by acting and watching what happens. They then classify tasks by the minimum evidence needed to resolve that ambiguity, giving three levels — I0, I1, and I2.

Next they build matched task pairs in which the goal stays the same but the interaction mechanism changes. Drag and Ten-Choice each have an exocentric variant (a movable cursor over a fixed scene) and an egocentric variant (the interaction point stays at screen center and the viewpoint moves). Rotation has an inner-ring and an outer-ring control variant, both I2. In the hidden-dynamics variants, an episode-specific gain (for example, how far a held object moves per unit of cursor displacement, or how much rotation follows a horizontal move) is sampled per episode and never shown in the first frame.

To give agents a chance to gather evidence, the authors reorganize the action space around Move: MOVE(p) is separate from coordinate-free CLICK, MOUSE_DOWN, and MOUSE_UP, and the environment returns a fresh observation after every primitive. Conventional CLICK(p) and DRAG(p,q) macros are shown to be equivalent to sequences of Move primitives, but only the Move form exposes intermediate observations as decision points.

Training data is generated with rule-based scripts that act, record the screenshot after each action, inspect revealed content or estimate movement responses, and then choose the next move before clicking or releasing. A teacher model adds a short Think rationale per prescribed action without changing the sequence. The policy is then trained by supervised fine-tuning on these trajectories, followed by reinforcement learning with Group Relative Policy Optimization (GRPO) using trajectory rewards based on task success and action-format compliance. Evaluation reports episode success per environment, a macro-average over the four I2 tasks, and a macro-average over all six; Rotation uses strict first-release scoring.

Why This Matters

Impact on research. The paper reframes GUI benchmark difficulty along a second axis — evidence required for environment identification — rather than only task horizon. Its findings suggest that failures of long-horizon GUI agents may partly reflect an inability to learn from interaction feedback, and that exposing intermediate observations to the model is a lever as important as model scale or workflow length.

Real-world applications.

• Software that adapts to a user's specific display scaling, sensitivity, or pointer-acceleration settings by probing them rather than assuming defaults.
• Agents operating unfamiliar or vendor-specific interfaces where the response rules are not documented in the visible screen.
• Interactive data-entry or form-filling tools that must verify field behavior before committing irreversible actions.
• Assistive and automation tooling on devices with unusual or non-standard input mappings.

Industry relevance. The paper connects directly to commercial computer-use agent development: it provides an evaluation suite, a public repository (https://github.com/ti-mm/LearnFromMove), and a public benchmark dataset (https://huggingface.co/datasets/OpenMOSS-Team/LatentGUIWorld). Its grounding result — 67.37% versus 66.72% for conventional grounding RL — indicates the interaction format can be folded into existing grounding training rather than replacing it.

Future Directions

• Deriving Move data from existing demonstrations. The authors state that multi-step Move data can be obtained from existing GUI and computer-use demonstrations by decomposing actions and supplying intermediate observations through environment replay or intermediate-frame construction "where feasible," and they explore this only for grounding data. Scaling this to broader datasets is open.
• Closing the I2 gap on Rotation. Paired Rotation training reaches 27.3% on the inner ring and 11.3% on the outer ring, far below the egocentric Drag and Ten-Choice results, leaving room to understand why continuous control latents are harder to learn than discrete ones.
• Whether frontier models can acquire the same behaviors. Since frontier models struggle especially on I2 tasks, an open question is whether the trajectories and interface generalize as training data for other model families or scales.
• Reducing reliance on task horizon as a difficulty proxy. The paper argues that increasing workflow length should be reconsidered when alternative interfaces can substantially simplify execution; how evidence-level difficulty and horizon interact in combined settings is not settled.

Target Audience

Researchers and engineers working on multimodal agents, computer-use agents, and GUI automation; benchmark designers interested in interaction-based evaluation rather than workflow-length scaling; and practitioners doing post-training (SFT and RL) who want a concrete recipe for teaching exploration and feedback-conditioned control in a 9B-scale model. Readers need some familiarity with reinforcement learning terminology (GRPO, trajectory rewards) and with GUI agent action spaces to get full value from the methodology sections.

Authors’ abstract

Existing benchmarks for GUI agents primarily focus on increasing task difficulty by extending task horizons. Besides, enabling agents to explore unfamiliar environments and successfully complete tasks within them is also a critical yet challenging problem. We introduce LatentGUIWorld, a benchmark of six tasks in three matched pairs organized by the evidence needed for control: $I_0$ uses prior knowledge and available observations, $I_1$ requires interaction that directly reveals hidden semantics, and $I_2$ identifies dynamics from action-response relationships. Frontier models struggle especially on $I_2$ tasks. We introduce MOVE, which separates movement from button events and exposes intermediate visual feedback, together with demonstrations of iterative exploration and control. Ablations show that both action granularity and multi-step demonstrations improve performance. After SFT, LatentLearner-9B achieves 50.0% overall success on LatentGUIWorld, comparable to several frontier models. Joint RL further raises egocentric Drag and Ten-Choice success to 92.0% and 75.3%. Behavioral analysis reveals multi-step exploration. In egocentric Drag, RL increases Pearson correlation between error-normalized commands and ideal gain compensation from 0.610 to 0.803. Together, these behaviors show that models learn to explore through interaction and adapt their control to environment dynamics. The same format improves GUI grounding after RL, reaching 67.37% mean accuracy across five benchmarks versus 66.72% for conventional RL.