Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learning

AI summary

Overview

Research area: Multimodal AI — search-augmented visual question answering (VQA) with Large Multimodal Models (LMMs), combining visual grounding, web retrieval, and reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with LMMs, supervised fine-tuning, LoRA, reinforcement learning (specifically Group Relative Policy Optimization), visual grounding, and retrieval-augmented pipelines.

Scope: This paper introduces Glance-or-Gaze (GoG), a two-stage trained framework in which a multimodal model decides for itself whether to look at an entire image ("Glance") or zoom into a specific region ("Gaze"), and whether to search the web, before answering knowledge-intensive visual questions.

What This Paper Is About

LMMs answer questions from fixed, pre-trained knowledge, so they fail on queries about recent events, obscure entities, or details outside their training data — producing hallucinations or generic answers. Existing fixes bolt search onto the model in a blunt way, retrieving whole images indiscriminately (adding noise) or converting visual detail into text (losing information), and rarely letting the model check or correct its own retrieval. GoG's goal is to train an LMM to plan its own visual search adaptively: filter visual noise by selecting high-value image regions, and iterate through multi-step search and reflection based on how hard the question is.

Key Contributions

1. Glance-or-Gaze (GoG) framework: The authors describe it as the first fully autonomous framework that shifts from passive image perception to dynamic, complexity-adaptive visual planning, with a Selective Gaze mechanism that evaluates and prioritizes relevant image patches before retrieval.
2. Dual-stage learning architecture: Reflective GoG Behavior Alignment (supervised fine-tuning on curated trajectories) installs the basic Glance/Gaze paradigm, and Complexity-Adaptive Reinforcement Learning (GRPO on difficulty-stratified data) improves planning and reasoning on hard queries.
3. A curated training pipeline and dataset: GoG-Instruct, built from FVQA and InfoSeek via uncertainty-aware filtering, synthesized trajectories, and human verification, containing 5,750 samples (43.5% search-free, 56.5% requiring search).
4. Empirical validation: State-of-the-art results across six benchmarks, plus ablations showing both Selective Gaze and complexity-adaptive RL are necessary.

Main Findings

• State-of-the-art aggregate performance: GoG-3-8B-Think-RL reaches an average of 48.02 across the six benchmarks, outperforming the strongest Full-Search Workflow by +9.89 and the strongest Prompt-based GoG agent by +15.18 on average. Per-benchmark scores are FVQA 56.88, InfoSeek 68.44, SimpleVQA 49.05, MMSearch 66.44, LiveVQA 65.50, and DynVQA 43.85.
• Large gain over a comparable search-equipped model: Against the reproduced MMSearch-R1 baseline (average 34.84), GoG reports a +19.97 improvement. The abstract summarizes the overall margins over strong baselines as ranging from 5 to 20.
• SFT teaches basic tool use; RL shifts behavior toward multi-step search: After SFT, single-type searches dominate (62.3% for Qwen3-VL-Think, 50.2% for Qwen2.5-VL) and some samples need no search at all (9.0% and 30.4% respectively). After RL, "mix search" rises from 28.7% to 76.7% (Qwen3-VL-Think) and from 19.4% to 74.0% (Qwen2.5-VL), while the no-search ratio falls below 3% for both models.
• RL sharpens where the model looks: Crop Selection Accuracy — the proportion of selected crops containing answer-relevant information — rises from 42.1% to 48.9% (+6.8%) for Qwen3-VL-Think and from 46.7% to 51.3% (+4.7%) for Qwen2.5-VL.
• RL improves error awareness, not just frequency of tool use: In a manual analysis of 100 GoG-3-8B-Think samples, Gaze correctness improves from 59% to 75%, and the reflection rate on incorrect Gaze selections rises from 30% to 70%.
• Selective Gaze is essential: Removing it (the "w/o SG" variant) drops average scores by 1.52 points for Qwen2.5-VL-7B (43.28 → 41.76) and 1.83 points for Qwen3-VL-8B-Thinking (50.17 → 48.34). Gains are largest where target information is small or obscured — DynVQA (+4.25) and MMSearch (+7.60) for the Qwen3 backbone.
• Hard training data beats easy data: Training RL on the harder Level 2 pool beats the easier Level 1 pool by +5.94 average for Qwen2.5-VL-7B (53.22 vs 47.28) and by +3.49 for Qwen3-VL-8B-Thinking (52.38 vs 48.89).
• A stated limitation: Roughly 1–5% of search attempts fail because of network instability, API timeouts, or malformed webpage content, which can degrade answer quality. Experiments also focus on English-language benchmarks; multilingual generalization is not tested.

Methodology in Plain English

The authors build two training stages on top of existing multimodal models (Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Think).

Stage 1 — Behavior Alignment (supervised fine-tuning). They first assemble training data from FVQA and InfoSeek. To avoid wasting effort on questions the model already answers from memory, they query Qwen3-VL-235B-Instruct four times per question and discard any question the model gets right all four times, keeping only the harder ones that need external evidence. For the retained questions, they synthesize reasoning trajectories with an explicit structure: Glance (analyze the whole image), Decision (call a tool or crop a region), and Gaze (run a targeted search on the chosen region). Human annotators then check every trajectory for answer accuracy and whether cropped regions are logically relevant to the question, discarding failures. The resulting GoG-Instruct set has 5,750 samples. The model is fine-tuned on these multi-turn conversations with LoRA adapters (rank 8) across all transformer blocks, using a standard causal language modeling objective.

Stage 2 — Complexity-Adaptive Reinforcement Learning. After SFT, the authors test the model on raw queries and split them by difficulty. Level 1 contains queries the SFT model passes about 50% of the time — the decision boundary where learning signal is richest. Level 2 adds queries the GoG paradigm consistently fails, lowering the overall pass rate. They train on Level 2 using GRPO: for each input, the model samples a group of candidate trajectories, each trajectory gets a reward, and advantages are computed by normalizing rewards within the group. The reward mixes an accuracy score judged by gpt-oss-120b (0 or 1) with a format-compliance score (0 to 1).

Tools. Image search and text search use SerpAPI; text search results are processed by Jina Reader and summarized by Qwen3-32B. A grounding tool based on Grounding DINO returns the top-5 bounding boxes for a query, and a Gaze Selection step picks the 1 to 5 most question-relevant regions, which are searched in parallel.

Evaluation. Six benchmarks are used: InfoSeek and FVQA-test (in-distribution), plus SimpleVQA, MMSearch, DynVQA, and LiveVQA-New (out-of-distribution). The authors randomly sample 2,000 instances from InfoSeek and LiveVQA and filter all datasets to English only. Answers are scored by an LLM-as-a-Judge setup using gpt-oss-120b. Baselines cover Direct Answer, Full-Search Workflow, Prompt-based GoG agents, and search-equipped LMMs (MMSearch-R1, reproduced).

Training setup. Training ran on a single node with 8 NVIDIA H800 GPUs. SFT used LoRA rank 8, 3 epochs, bf16 precision, DeepSpeed ZeRO-3, and a maximum sequence length of 32,768 tokens. RL used GRPO in the veRL framework for 15 epochs with a global batch size of 256, an actor KL loss coefficient of 0.001, up to 5 rounds of multi-turn tool interaction, and limits of 3 image searches and 3 text searches. (Note: the main text and Appendix D list different values for a few hyperparameters, such as the SFT learning rate, SFT batch size, RL rollout sample size, and actor learning rate.)

Why This Matters

The work targets a real weakness of multimodal models — confidently answering questions about things they cannot know — and shows that a small model (7B to 8B parameters) can be trained to search selectively and verify itself, rather than retrieving everything and hoping. If the approach generalizes, it changes how multimodal agents are trained: not just to call tools, but to decide whether, where, and how often to look.

Potential real-world applications:

• Product and landmark identification: recognizing newly released items, niche festival imagery, or unfamiliar locations that postdate a model's training data.
• News and misinformation checking: verifying claims against up-to-date visual information published after model training.
• E-commerce and catalog work: grounding fine details in product photos before querying inventory or knowledge sources.
• Accessibility and assistive tools: helping users identify objects or scenes with region-level focus rather than whole-image guessing.

Industry relevance: The method builds on open-weight models (Qwen2.5-VL and Qwen3-VL, both Apache-2.0) and standard tooling (SerpAPI, Jina Reader, Grounding DINO), which makes the recipe relatively easy for teams to adapt. The reported ability to run on a single 8-GPU node, and the evidence that reinforcement learning on hard examples produces the biggest gains, are directly relevant to teams building search-augmented assistants where retrieval cost and noise are practical bottlenecks.

Future Directions

• Multilingual and cross-cultural extension: The authors explicitly flag generalization to multilingual or cross-lingual visual question answering, and to culturally specific visual knowledge, as unexplored.
• More reliable search infrastructure: The reported 1–5% failure rate from network instability, API timeouts, and malformed pages needs addressing before deployment-grade reliability is possible.
• Scaling and transfer: The experiments use a 7B and an 8B backbone; whether the complexity-adaptive RL recipe transfers to larger models, or to other tool sets and modalities, is not reported.
• Reducing dependence on human-verified trajectories: Stage 1 relies on expert annotation of every trajectory; cheaper or automated alternatives could widen applicability.

Target Audience

Researchers and engineers working on multimodal agents, tool-augmented LMMs, retrieval-augmented generation, or reinforcement learning for language and vision models. It is most useful to readers already comfortable with fine-tuning and RL pipelines who want a concrete recipe for teaching a model to decide when and where to search — and to practitioners building knowledge-intensive visual question answering systems who need evidence on whether selective visual search beats indiscriminate retrieval.

Authors’ abstract

Large Multimodal Models (LMMs) have achieved remarkable success in visual understanding, yet they struggle with knowledge-intensive queries involving long-tail entities or evolving information due to static parametric knowledge. Recent search-augmented approaches attempt to address this limitation, but existing methods rely on indiscriminate whole-image retrieval that introduces substantial visual redundancy and noise, and lack deep iterative reflection, limiting their effectiveness on complex visual queries. To overcome these challenges, we propose Glance-or-Gaze (GoG), a fully autonomous framework that shifts from passive perception to active visual planning. GoG introduces a Selective Gaze mechanism that dynamically chooses whether to glance at global context or gaze into high-value regions, filtering irrelevant information before retrieval. We design a dual-stage training strategy: Reflective GoG Behavior Alignment via supervised fine-tuning instills the fundamental GoG paradigm, while Complexity-Adaptive Reinforcement Learning further enhances the model's capability to handle complex queries through iterative reasoning. Experiments across six benchmarks demonstrate state-of-the-art performance. Ablation studies confirm that both Selective Gaze and complexity-adaptive RL are essential for effective visual search.