Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

AI summary

Overview

Research area: GUI visual grounding and post-deployment test-time adaptation for multimodal large language models (MLLMs), with connections to reinforcement learning and on-policy distillation.

Technical level: Advanced. The paper assumes familiarity with policy-gradient methods (GRPO), token-level advantage estimation, LoRA adapters, and auto-regressive coordinate generation.

Scope: The paper proposes a test-time self-evolving loop (Exploration → Evaluation → Reflection → Internalization) that lets a frozen-after-deployment GUI grounding model improve on unseen interfaces without human-annotated ground truth, using an MLLM Reflector plus Reflection-Guided On-Policy Self-Distillation (R-OPSD) and a Contrastive Calibration (CC) method.

What This Paper Is About

GUI grounding models typically have their parameters frozen after training, so they cannot get better when deployed on unfamiliar applications or layouts. Prior test-time adaptation work, such as GUI-RCPO, adapts online using sparse scalar rewards that only say whether a click succeeded, not why it failed or how to fix it. This paper closes that gap by having the model generate a textual reflection on its own failed or successful explorations and then converting that reflection into dense, token-level learning signals that update the model's weights — all without ground-truth annotations.

Key Contributions

1. A Test-Time Self-Evolving framework for GUI visual grounding that gives a deployed agent four capabilities in a closed loop: Exploration of unseen interfaces, Evaluation of predicted coordinates, Reflection on the evaluation, and Internalization of those reflections into parameters.
2. Reflection-Guided On-Policy Self-Distillation (R-OPSD), which builds a self-teacher from the grounding model itself conditioned on the Reflector's evaluation score S and reasoning R as privileged information, translating free-form natural language into dense token-level supervision over generated coordinate tokens.
3. A Contrastive Calibration (CC) method that addresses corrupted supervision from incorrect auto-regressive prefixes: an inverse-prompted student is told the failed prediction was correct, producing a negative advantage at the initial error token while the advantage decays toward zero as tokens drift.
4. Direction-based advantage clamping and integration with query-level GRPO advantage, plus extensive experiments and ablations across six benchmarks showing the first successful use of on-policy self-distillation for test-time adaptation in GUI visual grounding.

Main Findings

• Average improvement over the base model: across six benchmarks, the framework achieves an average accuracy improvement of 7.4% over the base model, with no ground-truth annotations used.
• Improvement over the prior test-time RL method: compared with GUI-RCPO, the method achieves an average performance gain of up to 7.7%.
• Qwen2.5-VL-3B: base average accuracy 50.2%. Adapting on ScreenSpot-v2 (SSv2) raises the average to 57.4% (+7.2); adapting on MMBench-GUI (MMG) raises it to 57.6% (+7.4). GUI-RCPO reaches 53.4% (+3.2) and 49.9% (−0.3) respectively.
• Qwen3-VL-2B: base average 65.7%. The framework raises it to 69.4% (+3.7) when adapting on SSv2 and 70.3% (+4.6) when adapting on MMG. GUI-RCPO reaches 67.6% (+1.9) and 65.6% (−0.1).
• Negative transfer avoided on complex datasets: GUI-RCPO's relative rewards fail when all rollouts in a group fail, causing negative transfer on MMG; R-OPSD still improves by using reflections as token-level feedback.
• Full-model per-benchmark numbers (Qwen2.5-VL-3B, adapting on SSv2): 88.8 on SSv2, 30.5 on ScreenSpot-Pro (SSP), 66.6 on MMG. Adapting on MMG: 87.5 on SSv2, 30.1 on SSP, 68.2 on MMG. Base model values are 80.4, 20.3 and 57.5.
• Reflection matters: incorporating Reflection raises SSP accuracy from 24.6% to 28.5% when adapting on SSv2.
• CC prevents collapse: the ablation variant using reflection but no CC suffers policy collapse, while adding CC turns negative transfer into gains reaching 64.3% on MMG. The baseline that uses only evaluation results as privileged information also collapses.
• R-OPSD beats scalar-reward GRPO: GRPO with averaged bounding-box pseudo-labels suffers negative transfer on MMG; GRPO with the Reflector's binary score S remains limited by reward sparsity; GUI-SD, which draws the bounding box in the image as privileged information, degrades sharply (78.5 on SSv2, 16.7 on SSP, 52.3 on MMG, and collapse on MMG adaptation).
• Contrastive Calibration quantitatively: without CC, R-OPSD produces corrupted token-level advantages with magnitude |ā_t| = 1.34 on drifted tokens; with CC, the update magnitude decays toward zero as tokens drift.
• Reflector reliability: the trained Reflector reaches 89.5% accuracy, 86.9% precision and 93.3% recall on Qwen2.5-VL-3B (versus 76.5 / 72.9 / 85.4 zero-shot), and 91.7% accuracy, 88.8% precision and 95.7% recall on Qwen3-VL-2B (versus 80.5 / 78.9 / 86.0 zero-shot), evaluated on a held-out set of 1,000 balanced instruction-prediction pairs.
• Integration strength: λ = 0.2 gives the best performance regardless of the adaptation and evaluation datasets.
• Computational cost: with 4 NVIDIA A100 40GB GPUs and Qwen2.5-VL-3B, GRPO + GUI-RCPO takes 449 minutes and GRPO + Binary takes 447 minutes; R-OPSD without CC takes 129 minutes (≈0.29×), R-OPSD with CC takes 151 minutes (≈0.34×), and the full framework with CC and query-level advantage (QA) takes 778 minutes (≈1.73×).
• Memory footprint: the shared-base-model, LoRA-adapter design restricts GPU memory to approximately 10GB for 3B/2B models and 30GB for 7B/8B models.
• Scalability to larger models: Qwen2.5-VL-7B goes from 86.8 / 19.9 / 68.2 (SSv2 / SSP / MMG) to 92.7 / 31.4 / 77.9 when adapting on SSv2 and 92.5 / 32.3 / 79.2 when adapting on MMG; GUI-RCPO reaches 89.2 / 25.9 / 70.3 and 88.3 / 25.5 / 70.9. The paper states adapting the 7B model on MMG boosts accuracy from 68.2% to 79.2%. Qwen3-VL-8B baseline is reported as 92.9 / 53.5 / 82.7, with the adapted results truncated in the provided content.
• Cold-start supervised fine-tuning provided no benefit: synthesizing 30K high-quality reasoning trajectories with Qwen3.7-plus gave no noticeable performance gain over applying RL directly to the base model for the Reflector.

Methodology in Plain English

The method runs a repeating four-stage loop every time the deployed model meets a new interface.

1. Exploration. The grounding model π_G receives a screenshot I and an instruction L and predicts either a bounding box [x1, y1, x2, y2] or a center point (x, y). There is no ground truth available.
2. Evaluation and Reflection. A separate MLLM Reflector π_R takes the screenshot, instruction and predicted coordinates and returns two things: a binary score S ∈ {0, 1} saying whether the prediction succeeds, and a step-by-step reasoning trace R extracted from <think> tags. The Reflector follows a fixed prompt template that requires it to analyze the instruction's intent, identify what sits at the clicked coordinate, verify the match, and state a conclusion in <answer>Yes|No</answer> tags.
3. Reflector training. The Reflector is trained once, offline, with GRPO using a format reward plus a binary reward, on 10K samples from the Functional Split of GroundCUA, for 1 epoch with learning rate 1×10⁻⁴, group size 8 and batch size 64. Its training data is built by sampling 8 coordinate predictions per instruction at temperature 1.0, labeling each against the ground-truth box, keeping only instructions with mixed results, then selecting one correct and one incorrect prediction per instruction — yielding approximately 10,160 pairs at a 1:1 positive-to-negative ratio. After training, the Reflector stays frozen during adaptation.
4. Internalization via R-OPSD. Instead of learning from a scalar reward, the method reuses the grounding model itself as a "self-teacher," but feeds it a prompt conditioned on S and R as privileged information. The token-level advantage is the log ratio between the conditioned teacher's probability and the unconditioned student's probability for each generated token, and the objective maximizes the sum of these advantages. To save memory, π_G and π_R share one base model and alternate roles by switching active LoRA adapters.
5. Contrastive Calibration. On failed explorations, an incorrect early token corrupts everything after it, making the teacher's later probabilities unreliable. CC replaces the denominator with an "inverse-prompted student" that is told the failed prediction was actually correct. At the first wrong token, this student is confident about the wrong token while the reflection-guided teacher is not, giving a negative advantage that suppresses the error. As errors accumulate, both models condition on similar prefixes and their distributions align, decaying the advantage to about zero. CC is applied only to failures.
6. Advantage clamping and query-level mixing. Token-level advantages are clamped by direction — max(0, a_i) when S = 1 and min(0, a_i) when S = 0 — so updates never contradict the evaluation. The token-level advantage is then added to the standardized GRPO query-level advantage with a coefficient λ, set to 0.2. Rollouts use temperature 1.0 and top-p 0.95; the grounding model trains for 2 epochs with learning rate 1×10⁻⁴ and batch size 64.

Why This Matters

Impact on research. This is described as the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. It provides an alternative to sparse-reward test-time reinforcement learning, showing that free-form textual reflection can be turned into dense token-level supervision. The Contrastive Calibration analysis also offers a general treatment of the incorrect-prefix problem that affects on-policy distillation more broadly.

Real-world applications (as implied by the benchmarks and framing).

• GUI agents that navigate and interact with desktop, web and mobile interfaces on behalf of users.
• Assistive and accessibility tools that must operate on interfaces the developer never saw during training.
• Robotic process automation and enterprise workflow tools that encounter custom or frequently updated internal applications.
• Cross-platform deployment where a single agent must handle many unseen layouts, operating systems and languages.

Industry relevance. The memory-efficient design — one shared base model, alternating LoRA adapters, roughly 10GB for 3B/2B models and 30GB for 7B/8B models — targets realistic deployment budgets. The cost analysis shows R-OPSD with CC runs at about 0.34× the time of a standard GRPO baseline, making continual on-device-style adaptation plausible for resource-constrained settings, while the full framework with query-level advantages trades 1.73× cost for the best accuracy.

Future Directions

• A fully self-contained loop. The current design still relies on an offline-trained Reflector built from GroundCUA with ground-truth labels; whether the Reflector itself could be improved during deployment is not reported.
• Component-level credit assignment. The paper reports rewards based only on S and R, without decomposing success or failure into which coordinate tokens were wrong; finer reward decomposition is a natural extension.
• Larger-model behavior. Results for Qwen3-VL-8B are truncated in the provided content, leaving the behavior of the framework at that scale an open question relative to the completed 7B results.
• Tuning and scope of the calibration. λ = 0.2 was best over the tested range of 0 to 0.5 on the tested adaptation and evaluation datasets; whether this transfers to other backbones, interface domains or benchmarks is not reported.

Target Audience

Researchers and engineers working on GUI agents, MLLM-based visual grounding, and test-time adaptation or continual learning. It will be most useful to readers already comfortable with reinforcement learning from verifiable rewards, policy-gradient objectives such as GRPO, and on-policy distillation, as well as practitioners who need a deployment-time adaptation method that avoids ground-truth annotation and fits within modest GPU memory.

Authors’ abstract

GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework's effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.