MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements

AI summary

Overview

• Research area: GUI grounding for autonomous computer-use agents — mapping natural-language instructions to screen coordinates using Vision-Language Models (VLMs).
• Technical level: Intermediate. The high-level design is accessible, but the paper assumes familiarity with VLM baselines, ROI (Region-of-Interest) search, benchmarking protocols and agentic pipelines.
• Scope (one sentence): The paper proposes MEGA-GUI, a modular multi-agent framework that decomposes GUI grounding into coarse ROI selection and fine-grained element localization, and reports state-of-the-art accuracy on ScreenSpot-Pro and OSWorld-G without training or fine-tuning any model.

What This Paper Is About

GUI grounding — the task of turning a command like "click the small wrench icon" into exact screen coordinates — remains unreliable because high-resolution displays dilute the target widget (the paper cites targets occupying less than 0.1% of total screen area) and because user instructions are often ambiguous. Prior work relies either on large monolithic VLMs trained on GUI data or on one-shot, unidirectional zoom-in pipelines that cannot recover once an early prediction is wrong. MEGA-GUI's goal is to reframe grounding as a structured, multi-stage control problem in which different VLMs are orchestrated as specialized agents, so that errors can be corrected and ambiguity resolved before the final click.

Key Contributions

1. A principled, decoupled framework. MEGA-GUI separates coarse spatial localization (Region-of-Interest deduction) from fine-grained semantic grounding. Its centerpiece is a bidirectional adaptive zoom algorithm — a recoverable search mechanism that uses geometric feedback to refine visual focus and recover from initial prediction errors via zoom-out and re-centering.

2. Empirical validation of a "No Free Lunch" principle for VLM specialization. The authors report the first empirical validation that models have distinct, scale-dependent strengths: some excel at broad contextual search and others at fine-grained identification, providing a data-driven basis for routing different VLMs to different stages. MEGA-GUI reports new state-of-the-art results on ScreenSpot-Pro and OSWorld-G.

3. An open-source toolkit for principled design. The Grounding Benchmark Toolkit (GBT) supports systematic evaluation of GUI grounding components, including resolution-based sweeps and the test-set ROIs extracted from ScreenSpot-Pro and OSWorld-G. Code and toolkit are released at https://github.com/samsungsds-research-papers/mega-gui.

4. No training or fine-tuning required. The framework orchestrates publicly available VLMs as specialized agents rather than training a new model.

Main Findings

• State-of-the-art accuracy. The optimal MEGA-GUI configuration achieves 73.18% on ScreenSpot-Pro and 68.63% on OSWorld-G. In the paper's Table 1, the strongest agentic baseline reported is RegionFocus at 61.60% on ScreenSpot-Pro (its OSWorld-G figure is listed as N/A), and the strongest reproduced single-shot baseline on ScreenSpot-Pro is GTA1-7B at 50.03%; on OSWorld-G, GTA1-7B is the strongest single-shot baseline at 56.22%.

• The winning configuration. Gemini 2.5 Pro powers the adaptive ROI zoom toward a target ROI size of 1000 pixels, GPT-4o performs context-aware instruction rewriting, and UI-TARS-72B carries out final precision grounding on a 3× bicubic-upscaled image crop.

• Component gains depend on the benchmark. Ablated sequentially over a UI-TARS-72B baseline (37.12% on ScreenSpot-Pro, 55.67% on OSWorld-G): ROI Zoom (1000 pixel) adds +28.47 pp on ScreenSpot-Pro but only +2.17 pp on OSWorld-G; Image Scaling (3×) adds +2.40 pp and +1.18 pp; Instruction Rewrite adds +5.19 pp and +9.61 pp. Total gain over baseline: +36.06 pp and +12.96 pp. On the visually dense benchmark, search dominates; on the semantically complex benchmark, disambiguation dominates.

• A lower-resolution check. On ScreenSpot v2 reported in the paper's footnote, starting from a 90.30% baseline, ROI Zoom changes performance by −0.91 pp (to 89.39%), Image Scaling by +0.58 pp (to 89.97%) and Instruction Rewrite by +3.35 pp (to 93.32%), for a net gain of +3.02 pp.

• Bidirectional zoom beats unidirectional zoom. Across twelve model and hyperparameter configurations on ScreenSpot-Pro, the bidirectional policy yields consistent absolute accuracy gains of +1.14 to +4.36 pp (mean +2.22 pp) over a "zoom-in only" variant.

• Asymmetric zoom-in improves containment. The asymmetric crop (pruning the region farthest from the predicted point) achieves an ROI containment rate of 89.69%, versus 81.65% for a symmetric baseline — a margin of 8.04 pp.

• ROI containment varies sharply by model and size. On ScreenSpot-Pro, Gemini 2.5 Pro scales from 72.6% containment at a 400-pixel ROI to 93.2% at 1800 pixels. The paper notes that commercial models such as Gemini 2.5 Pro and CUA score worse in containment rate than some baselines and are more sensitive to ROI size.

• Best grounding model and scale. UI-TARS-72B shows the strongest conditional accuracy at an ROI size of 1000 pixels; among scaling factors from 1× to 4×, a 3× scale gave the best improvement. The combined Scale and Rewrite agents lift final conditional accuracy to 81.4% on ScreenSpot-Pro (measured only on ROIs that contain the target).

• Refusal of infeasible instructions. The Refusal Agent (Gemini 2.5 Pro) reaches 68.5% refusal accuracy with a 3.3% false positive rate on OSWorld-G (N=54 infeasible tasks). Baseline models (JEDI, UI-TARS, etc.) span 0.0–18.5%; Gemini-2.5-Pro alone reaches 38.9%. One alternative configuration reached a maximal refusal accuracy of 78.0% but at a 5.3% FPR. The Refusal Agent is enabled for OSWorld-G (which mixes feasible and infeasible tasks) and disabled for ScreenSpot-Pro (all tasks have valid targets).

• Latency is reported, not solved. The research prototype averages 21.54 seconds (0.359 minutes) per task, which the authors attribute to sequential client-server API calls in a non-optimized environment rather than to the algorithm.

• Failure archetypes. Qualitative analysis identifies three failure modes: attentional fixation (Stage 1 repeatedly returns the same wrong coordinates with high confidence until the error budget E_max is depleted), deterministic bias (Stage 2 selects familiar icons, e.g., a three-dots menu instead of a gear icon), and instructional over-correction — a "curse of specificity" where rewriting "Save the document" into an explicit textual directive makes grounding harder for the executing model.

Methodology in Plain English

The authors treat grounding as a pipeline with independent, separately measurable stages rather than one end-to-end prediction:

1. Feasibility check (optional). An agent decides whether an instruction is executable on the current screen, and refuses otherwise.
2. ROI deduction. A VLM picks a fixed-size window (1000 pixels in the final experiments) that should contain the target. Success is measured by containment rate.
3. Fine-grained grounding. Given that window, a grounding VLM outputs the exact coordinate. Success is measured by accuracy, computed only on windows that actually contain the target.

The core search mechanism is a closed loop. Starting from the full screen, the model predicts a point. If the point is inside the current window, the window is cropped aggressively inward from the boundary farthest from that point (asymmetric zoom-in). If the prediction falls outside the window, the window is expanded again (zoom-out) to restore context. An error counter penalizes oscillation: once it exceeds a threshold, the agent is forced to zoom in rather than keep expanding. The loop stops either when the window shrinks below a minimum area or when several consecutive in-bounds predictions cluster tightly together (convergence).

Two additional agents refine the input: a Context-Aware Rewrite Agent first infers the active application from the crop, then rewrites the user's raw instruction into an explicit, context-rich command; and a Conservative Scale Agent upscales the cropped ROI back to high resolution (3× was best) before grounding.

Evaluation used GPT-4o (version gpt-4o-1120) and Gemini-2.5-Pro via official APIs during July–August 2025; open-source comparison models were run locally on a server with eight NVIDIA V100 GPUs. The CUA baseline used the computer-use-preview model (v. 2025-03-11) via Azure OpenAI Service. Benchmarks: ScreenSpot-Pro (1,581 tasks on 4K-resolution professional software interfaces) and OSWorld-G (564 tasks on standard 1080p operating system interfaces). The primary metric is Top-1 Accuracy against a ground-truth bounding box (ScreenSpot-Pro) or bounding polygon (OSWorld-G). Agentic baselines are reported as published; single-shot baselines were reproduced in a unified environment.

Why This Matters

Impact on research. The paper argues that monolithic, end-to-end grounding models are not the only path forward: system-level orchestration of existing VLMs, with per-stage metrics and error recovery, can outperform them. It also introduces the "No Free Lunch" framing for VLM specialization — different models are better at coarse search versus fine localization — which motivates adaptive, data-driven model routing rather than hard-wiring a single VLM into a pipeline. The released GBT and code are positioned to make component-level comparisons reproducible.

Real-world applications (as discussed in the paper):

• Assistive technologies that let users with disabilities control software by natural-language instruction.
• Robotic process automation (RPA) for enterprise software workflows.
• Autonomous computer-control agents that operate on a user's full-screen interface rather than a cropped demo view.
• Safety screening of commands, via the Refusal Agent, which blocks infeasible instructions and could be extended to flag unsafe ones.

Industry relevance. The work comes from Samsung SDS and is explicitly framed around practical deployment: proprietary API models, a full-screen interface setting, and an efficiency analysis acknowledging that the 21.54-second-per-task prototype latency is an artifact of a non-optimized, sequential client-server setup that co-location and batching could reduce. The authors also note the framework currently assumes access to high-performing proprietary models, which limits deployment in resource-constrained settings.

Future Directions

• Latency and cost reduction. Move from sequential API calls to co-located agents and batched requests, and explore pruning strategies so the multi-query pipeline becomes viable for interactive use.
• Reducing dependence on proprietary models. Explore open-source alternatives or model distillation to produce smaller, highly efficient specialists for each stage.
• Safer and more capable refusal behavior. Extend the Refuser Agent to ask the user for clarification on overly ambiguous instructions and to detect manipulative or harmful instructions — the authors call advanced ethical reasoning an important open challenge.
• Mitigating the remaining failure archetypes. Attentional fixation, training-inherited icon biases in the grounding model, and the "curse of specificity" in instruction rewriting all indicate that the interface between agents needs further study — the best rewritten instruction is not always the most explicit one.
• Adaptive model routing. The "No Free Lunch" finding suggests building data-driven routing that selects which VLM handles which stage, rather than fixing the assignment in advance.

Target Audience

Researchers and engineers working on GUI agents, computer-use automation, and multimodal grounding; practitioners building RPA or accessibility tooling that must click the right pixel on a real screen; and benchmark/evaluation researchers interested in the Grounding Benchmark Toolkit and in component-level, per-stage metrics for agentic pipelines. Readers seeking a training recipe or a new model architecture will not find one here — this is a systems and orchestration paper.

Authors’ abstract

Graphical User Interface (GUI) grounding - the task of mapping natural language instructions to screen coordinates - is essential for autonomous agents and accessibility technologies. Existing systems rely on monolithic models or one-shot pipelines that lack modularity and fail under visual clutter and ambiguous instructions. We introduce MEGA-GUI, a multi-stage framework that separates grounding into coarse Region-of-Interest (ROI) selection and fine-grained element grounding, orchestrated by specialized vision-language agents. MEGA-GUI features a bidirectional ROI zoom algorithm that mitigates spatial dilution and a context-aware rewriting agent that reduces semantic ambiguity. Our analysis reveals complementary strengths and weaknesses across vision-language models at different visual scales, and we show that leveraging this modular structure achieves consistently higher accuracy than monolithic approaches. On the visually dense ScreenSpot-Pro benchmark, MEGA-GUI attains 73.18% accuracy, and on the semantically complex OSWorld-G benchmark it reaches 68.63%, surpassing previously reported results. Code and the Grounding Benchmark Toolkit (GBT) are available at https://github.com/samsungsds-research-papers/mega-gui.