EviRover: Reinforcing Agentic Perception Beyond a Glance

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically agentic perception, tool-using multimodal agents, and agentic reinforcement learning (RL).

Technical level: Advanced. The high-level idea (let a model look again or search before answering) is intuitive, but the training recipe involves GRPO-style policy optimization, task-specific reward shaping, and an analysis of token-level versus sequence-level loss reduction.

Scope: The paper defines "perception under insufficient evidence," builds two data-generation pipelines plus a human-verified benchmark (EviLens), and trains a 4B perception agent (EviRover) with supervised fine-tuning followed by agentic RL.

What This Paper Is About

Standard visual perception (grounding, segmentation, counting) is treated as a one-shot prediction from a single glance at an image, assuming the image plus the model's parametric knowledge are enough to answer. The paper argues this assumption breaks down when the answer hinges on fine-grained visual detail that needs closer inspection, or on knowledge that is unfamiliar, external, or newer than the model. The goal is to reformulate perception as an evidence-seeking process, where the agent decides what evidence is missing and which action to take, and then still produces a location, a boundary, or a count on the input image.

Key Contributions

1. Identifying "perception under insufficient evidence" and reformulating perception as an evidence-seeking process that looks beyond a single glance, treating the perceptual output itself (not a textual answer) as the objective.
2. EviRover, described as the first perception agent explicitly trained to determine what evidence is missing and how to acquire it, trained with supervised fine-tuning followed by agentic reinforcement learning.
3. Two dedicated data generation pipelines yielding EviRover-SFT-5K and EviRover-RL-12K, plus EviLens, a human-verified benchmark of 688 instances spanning segmentation, counting, and three grounding categories (localization, recognition, spot-the-difference).
4. Extensive experiments showing a 30-point average improvement over the backbone on EviLens (30.2 points), with gains transferring to WebEyes (14.4 IoU on grounding, 24.7 gIoU on segmentation), conventional perception benchmarks (ReasonSeg, RefCOCOg), and general multimodal benchmarks (MMMU, MMMU-Pro, MathVerse, BrowseComp-VL).

Main Findings

• EviLens is hard for every model family. Among proprietary models, Gemini-3.5-Flash, Seed-2.1-Turbo, and GPT-5.6-Sol perform best overall. Seed-2.1-Turbo achieves the highest segmentation score of any model at 0.779 gIoU. GPT-5.6-Sol is notably poor on spot-the-difference, with a micro-F1 of only 0.112, because it consistently predicts a single bounding box even when multiple differences are indicated.

• Open-source models lag substantially. Despite strong results on conventional perception benchmarks, segmentation specialists reach at most 0.343 gIoU on EviLens (Seg-Zero-7B), and grounding specialists achieve near-zero IoU on localization (Perception-R1: 0.007 IoU, 0.000 R@0.5; UniVG-R1: 0.051 IoU). The authors attribute this to EviLens testing evidence acquisition rather than perceptual precision.

• EviRover leads on localization and recognition. On EviLens, EviRover reaches 0.444 IoU / 0.500 R@0.5 on localization and 0.648 IoU / 0.687 R@0.5 on recognition. The localization margin over Gemini-3.5-Flash (0.373 IoU) is described as clear. On segmentation it ranks second behind Seed-2.1-Turbo, at 0.686 gIoU and 0.708 cIoU.

• Large gains over the backbone. Relative to Qwen3VL-4B-Instruct, EviRover improves the primary metric of every category by 30.2 points on average on EviLens. On counting, the backbone scores 19.2 and EviRover scores 50.0, coming within 1.3 points of GPT-5.6-Sol (51.3). On spot-the-difference the backbone scores 0.005 micro-F1 and EviRover reaches 0.171 micro-F1 / 0.266 macro-F1.

• SFT alone already moves the needle. The "+ SFT" model scores 0.369 IoU / 0.371 R@0.5 (localization), 0.595 / 0.626 (recognition), 0.114 / 0.179 (spot-the-difference micro/macro F1), 0.584 gIoU / 0.633 cIoU (segmentation), and 42.8 counting accuracy, with RL adding further gains on top.

• WebEyes results. EviRover achieves the best grounding performance among all evaluated models including proprietary systems, with 40.0 IoU and 48.5 R@0.5. It outperforms Pixel-Searcher-8B (a prompt-based workflow built on a larger backbone) by 10.4 gIoU on segmentation (49.5 versus 39.1). On VQA, a task absent from the training data, it performs comparably to Pixel-Searcher-8B (40.3 versus 42.2).

• Conventional perception also improves. On ReasonSeg, EviRover reaches 68.7 val gIoU / 62.3 cIoU and 62.6 test gIoU / 56.5 cIoU, versus 63.9 / 58.9 and 60.9 / 53.9 for the backbone, outperforming Seg-Zero-7B (62.6 / 62.0 and 57.5 / 52.0). On RefCOCOg it is ahead of its backbone on every split and metric,

Authors’ abstract

Visual perception is conventionally formulated as a one-shot prediction from a single glance at the image, under the assumption that the image content and the model's parametric knowledge suffice to resolve the query. This assumption often fails in real-world scenarios that hinge on fine-grained visual details or require knowledge-intensive and up-to-date information. We term such cases \textit{perception under insufficient evidence} and formulate perception as an agentic process that can obtain information beyond a single glance. To address the absence of data for this setting, we design two dedicated data generation pipelines, yielding EviRover-SFT-5K and EviRover-RL-12K for training. We further construct EviLens, a human-verified benchmark comprising 688 instances across five perception categories. Building on these data, we present EviRover, to our knowledge the first perception agent explicitly trained to resolve perceptual queries through interaction, using supervised fine-tuning followed by agentic reinforcement learning. Experiments show that the 4B EviRover outperforms its backbone by 30 points on average on EviLens, reaching performance comparable to advanced proprietary models. The gains transfer beyond EviLens to WebEyes, conventional perception benchmarks, and general multimodal benchmarks, including a 15-point improvement on BrowseComp-VL. All code, models, and data are released.