PreView: Enhancing GUI Grounding with Preprocessed Views and Cues

AI summary

Overview

Research area: Computer vision and GUI agents — specifically GUI grounding, the task of mapping a natural-language instruction ("click Get involved") to a precise pixel location in a screenshot.

Technical level: Intermediate. Readers should be comfortable with vision-language models, LoRA fine-tuning, and standard grounding benchmarks, but the central idea (preprocess the input rather than retrain the model) is conceptually simple.

Scope: The paper proposes PreView, a trained auxiliary preprocessor that produces task-relevant cropped/zoomed views and candidate-specific visual and textual cues for a frozen GUI grounding model, and evaluates it across five benchmarks and three grounding models.

What This Paper Is About

GUI grounding is hard for two distinct reasons: localization (finding the exact pixel of a small control in a dense, high-resolution screenshot) and understanding (deciding which of several visually similar controls the instruction actually refers to). Existing fixes — training on more GUI data, iterative zooming, repeated prediction — either modify the model or spend extra inference calls. PreView asks a different question: how should the screenshot and instruction be presented so an existing, unmodified grounding model makes a better decision? The goal is to improve a frozen grounder through input preparation alone, in a single downstream call.

Key Contributions

1. A training-free-for-the-grounder preprocessing framework. PreView is an auxiliary model that prepares inputs for an existing grounding model without training, modifying, or accessing the internal states (logits, hidden states) of that model. It preserves the grounder's input interface and decoding procedure, adding only preprocessing and coordinate mapping.
2. Two complementary input feature types. Task-relevant views support localization: the auxiliary model predicts a center, refines it, crops, and enlarges. Candidate-specific cues support understanding: a visual cue (a translucent red circle plus a fixed location hint) and a textual cue (a generated description of the candidate's identity, function, or local relationships).
3. Joint training of center prediction and candidate verification/description in a single auxiliary model, using positive and negative examples so the model can either describe a matching candidate or output REJECT for a mismatch.
4. Center-consistency routing that compares the initial and refined centers (δ ≤ τ, with τ = 0.02) and falls back to the unmarked view with the original instruction when the two disagree, limiting how often misleading cues reach the grounder.

Main Findings

• Average accuracy improves for all three grounders, by very different amounts. Reported average accuracy rises by 7.4 points for Qwen3-VL-2B (57.2 → 64.6), 13.9 points for OS-Atlas-4B (21.6 → 35.5), and 1.5 points for Qwen3-VL-4B (66.8 → 68.3), with no additional training of these models. Averages use OSW-G Refined.
• A small model can partly close the size gap. With PreView, Qwen3-VL-2B matches the Qwen3-VL-4B Full score on SSPro (59.3%) and exceeds it on UI-V (33.7% vs 25.3%). The 2B and 4B models are reported as outperforming all listed baselines of the corresponding larger size (3B and 7B) on SSPro and OSW-G Refined.
• Gains are uneven across benchmarks, and some benchmarks regress. Qwen3-VL-2B reaches 43.1 → 59.3 (SSPro), 46.1 → 55.0 (OSW-G O), 62.1 → 66.8 (OSW-G R), 72.9 → 77.0 (MMB-GUI), 16.5 → 33.7 (UI-V), but falls 91.4 → 86.0 on SSv2. Qwen3-VL-4B gains 15.3 points on UI-V yet loses 5.0 on MMB-GUI. The paper states the first two models improve on four benchmark families: SSPro, OSW-G, MMB-GUI, and UI-V.
• Icon targets benefit far more than text targets. On SSPro, Qwen3-VL-4B icon accuracy rises from 32.5% to 42.5%, while its text accuracy moves only from 75.8% to 76.8%.
• Spatial instructions remain the weak spot. Within UI-V, spatial accuracy is lowest at 17.0% for the 2B model and 26.0% for the 4B model after preprocessing.
• Views supply much of the gain before any cue is added. For Qwen3-VL-2B on SSPro, the view alone lifts accuracy from 43.1% to 56.9%, close to the 59.3% reached with full cues. On UI-V, cues add 7.7 points beyond the view for that model, versus 2.5 points on SSPro.
• A location marker alone can hurt. For Qwen3-VL-2B on SSPro, adding the red-circle marker and hint drops accuracy from 56.9% to 55.6%; the description then recovers it to 59.3%. For Qwen3-VL-4B on SSPro, accuracy goes 65.0% (View) → 64.0% (View + visual cue) → 63.7% (both cues), so the loss starts with the marker and is not recovered by the description.
• One refinement step is enough. On SSPro, Refine 1 keeps the target center inside the 12.5%-area view in 80.14% of cases while directly hitting the target in only 53.83%. Mean direct hit rises from 60.83% to 64.02% at the refined center c₂; the analysis-only third prediction c₃ gives 63.93%, so the pipeline stops at c₂.
• Rejected candidates can still be correctly located. Sample-weighted direct hit is 26.1% for rejected candidates versus 66.5% for accepted ones, and rejected candidates reach 63.4% direct hit on SSv2. Among 1,163 rejected candidates that pass consistency filtering, retaining visual cues yields 28 (2B) and 12 (4B) more correct clicks than view-only.
• How the visual budget is spent matters more than how much is spent. At the same 25% pixel budget, enlarging a 12.5%-area crop beats a native 25%-area crop on all three tested benchmarks with identical processed visual-token counts. For a fixed 12.5% crop, doubling both dimensions adds 6.5–10.8 points. At a 50% budget, smaller crops do better on SSPro and UI-V, while the wider crop improves OSW-G R by 0.71 points.
• Cue accuracy determines whether cues help or harm. Across 8,288 target-present Qwen3-VL-2B examples passing the consistency filter, cues located inside the annotated target correct 71.0% of view errors and disrupt only 1.9% of correct predictions; outside the target, these rates invert to 3.2% and 57.8%.
• Gains persist even for a grounder trained on the same source data. GroundNext-7B, whose supervised fine-tuning used 700,000 GroundCUA examples (PreView's 148,595 training examples are 21% of that), improves from 52.9% to 59.0% on SSPro and from 67.7% to 68.3% on OSW-G R when given the view. It regresses on MMB-GUI (81.1 → 76.9), SSv2 (90.4 → 81.8), and UI-V (60.3 → 54.5).
• One call versus multi-step search. On SSPro, PreView's gains of 16.2, 25.4, and 4.4 points for the 2B, OS-Atlas-4B, and 4B models require one grounding-model call, compared with training-free methods such as ScreenSeekeR (OS-Atlas-7B, 18.9 → 48.1, +29.2), AutoFocus (UI-Venus-7B, 50.3 → 67.8, +17.5), and ZoomClick (UI-Venus-7B, 50.3 → 65.7, +15.4).
• Main failure modes are identified explicitly. Cropping can exclude the target entirely (one case study excludes cell U42), and misleading descriptions can override a correct prediction (one case study redirects a correct click to a neighboring notification icon). The paper notes that most MMB-GUI regression for the 4B model occurs before cue injection (84.7% → 79.9% with the view alone).

Methodology in Plain English

The setup. Take an off-the-shelf grounding model and freeze it completely. Train a separate, small auxiliary model (Qwen3.5-0.8B) whose only job is to decide what to show the grounder and what to tell it.

Training the auxiliary model on two tasks at once. Using 50,000 desktop-application screenshots from GroundCUA, the authors build three example types: 50,000 center-prediction examples (predict the target box's center in image-relative coordinates), 49,311 positive description examples (describe a sampled candidate inside the target box), and 49,284 rejection examples (respond REJECT to annotated non-target elements). Both tasks share one set of parameters and are trained with standard next-token loss computed only over the response tokens. Training runs two epochs in BF16 with LoRA on the language-model attention and feed-forward layers, a trainable multimodal merger, and a frozen vision encoder.

Inference: build the view. The auxiliary model predicts an initial center c₁ from the whole screenshot, then refines it to c₂ using a local view plus the original instruction. The final view is a crop of roughly the target area around c₂, shifted inside the image boundary if needed, then enlarged. Refinement uses a 50%-area crop enlarged by √2 per dimension; description generation and grounding use a 12.5%-area crop enlarged 2×.

Inference: build the cues. The visual cue is a translucent red circle at c₂ paired with a fixed textual hint explaining that clicking the circle's center is a safe fallback. The textual cue is a description generated from the marked view, covering the candidate's identity, function, or relationships. If the model outputs REJECT, the description is withheld.

Inference: route and map back. If the initial and refined centers agree within τ = 0.02 in normalized coordinates, the grounder receives the marked view plus available cues; otherwise it receives the same unmarked view with the original instruction. The grounder's raw click is normalized within the input view and then mapped back to full-screenshot coordinates using the crop rectangle's corners — so the frozen model keeps its native coordinate convention and decoding procedure.

Why This Matters

Research impact. The paper reframes part of the grounding problem as an input presentation problem rather than a model capacity problem, and shows this can be studied without touching the grounder. It also provides an unusually honest accounting of when the approach backfires: benchmark-level regressions on SSv2 and MMB-GUI, cue-induced regressions quantified at 57.8% when the cue sits outside the target, and cropped-out targets. That kind of negative-result reporting is useful for anyone building zoom-based pipelines.

Real-world applications:

• GUI automation and RPA agents that must click small controls in dense professional interfaces (CAD, development tools, office suites), where a wrong click executes an unwanted action.
• Deployment of small, cheap grounding models on-device, where a 2B model with preprocessing approaches a 4B model's accuracy on some benchmarks — relevant when memory or latency budgets rule out larger models.
• Desktop accessibility tooling, letting a user describe a target in natural language and having the system locate and click it, with human confirmation for consequential operations as the paper's ethics section recommends.
• Software test automation and QA, where test scripts need to click the right widget among visually similar siblings such as a button and its dropdown arrow.

Industry relevance. Because PreView adds no parameters to the deployed grounder and requires no access to logits or hidden states, it can be bolted onto an existing grounding API as a preprocessing layer. The cost is quantified: auxiliary inference adds computation and latency, and the pipeline deliberately stops at c₂ rather than adding a second refinement call.

Future Directions

• Adaptive view selection and cue gating. The authors explicitly call for adapting view selection and cue use based on estimated target coverage and cue reliability — a direct response to the finding that 57.8% of correct predictions are disrupted when the cue lies outside the target.
• Extending beyond desktop. Training used only desktop applications; the 2B model improves across Windows, macOS, and Linux but regresses on both iOS subsets, and the authors list mobile generalization as an open limitation.
• Cheaper auxiliary inference. The auxiliary model adds computation and latency before the single grounding call; reducing that overhead, or deciding when preprocessing is not worth it, is an open engineering question.
• Evaluation beyond single-step clicks. The suite deliberately isolates single-step grounding; whether these gains survive planning, multi-step recovery, and interactive environments (VisualWebArena, OSWorld, AndroidWorld are cited) is untested. The paper also notes single-run-per-configuration reporting, so initialization noise is unquantified.

Target Audience

Researchers and engineers working on GUI agents and vision-language grounding who already have a working grounder and want to improve it without retraining. It is most useful for practitioners building zoom- or crop-based inference pipelines, since the paper's ablation tables (view vs. visual cue vs. textual cue, cropping area vs. enlargement at matched token budgets) give directly actionable guidance about which of those components earns its cost on which benchmark type. Readers seeking architectural innovations or new grounding-model training recipes will find little here — the contribution is deliberately confined to the input side.

Authors’ abstract

GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropping and enlargement can expose local detail, but the resulting image may still leave ambiguity about the intended target. We propose PreView, an auxiliary preprocessor that improves the inputs to an existing grounding model without further training that model. PreView prepares two complementary types of input features: task-relevant views for localization and candidate-specific cues for target understanding. A trained lightweight auxiliary model predicts and refines a center, crops the screenshot into a local view, and enlarges that view for zoomed-in grounding. It also prepares a visual cue (i.e., a red-circle marker with a location hint) and a textual cue describing the candidate's identity or function. The frozen grounding model uses the prepared inputs to predict the final click. Across five benchmarks, PreView improves the average grounding accuracy of Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B without further training these grounding models. Extensive analyses examine the contributions of views and cues, center refinement, and visual input budgets.