EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligence

AI summary

Overview

Research area: Computer vision / multimodal large language models — specifically video-based spatial reasoning (distance estimation, relative direction, room layout understanding from multi-view video).

Technical level: Advanced. The paper assumes familiarity with multimodal LLMs, determinantal point processes, SE(3) camera geometry, BEV representations, and reinforcement learning (GRPO).

Scope: A dual-stage framework (EagleVision) that selects geometry-aware keyframes and then lets a VLM actively request new video frames by querying poses on a Bird's-Eye-View map, trained purely with reinforcement learning.

What This Paper Is About

Existing multimodal LLMs consume a fixed set of uniformly sampled video frames and cannot ask for new viewpoints once reasoning has started, so they often miss the geometric cues (parallax, viewpoint change) needed for reliable spatial judgments. EagleVision splits this into two stages: a macro perception stage that picks a compact, semantically relevant and geometrically diverse set of keyframes under a token budget, and a micro verification stage where the model iteratively reasons in text and queries BEV poses to retrieve additional real frames, forming a closed-loop hypothesize–look–verify cycle. The goal is state-of-the-art spatial reasoning among open-source vision-language models without adding 3D encoders or human-annotated reasoning traces.

Key Contributions

1. A dual-stage framework combining geometry-aware frame selection (macro perception) with active, BEV-grounded reasoning (micro verification) for video spatial understanding.
2. SPF-DPP (Semantics–Perspective-Fusion Determinantal Point Process), a sampling strategy that jointly models semantic relevance and SE(3) viewpoint diversity to select informative keyframes under token constraints.
3. Spatial MCoT as BEV-grounded pose querying, trained purely with reinforcement learning and a spatial grounding reward — no human-annotated chain-of-thought supervision is required.
4. State-of-the-art results among open-source VLMs on VSI-Bench and SQA3D, demonstrating cross-benchmark generalization.

Main Findings

• VSI-Bench: EagleVision achieves the best average score of 63.5 among all open-source VLMs, a +4.1 improvement over its base model Qwen3-VL-8B (59.4) and a +2.6 gain over VLM-3R-7B (60.9), which uses an additional 3D reconstruction encoder. No architecture or visual input pipeline changes were made.
• SQA3D: EagleVision reaches 60.3% (EM@1), outperforming Struct2D (58.5, +1.8), Spatial-MLLM (55.9, +4.4) and SpatialMind (46.3, +14.0).
• Component ablation (on Qwen3-VL-8B, 59.4 Avg): adding Spatial MCoT alone gives 61.9; adding BEV grounding gives 62.7 (+3.3 over baseline); adding SPF-DPP on top gives the best 63.5 (+0.8). The paper describes the three components as complementary.
• SPF-DPP ablation (on the MCoT+BEV baseline of 62.7): geometry-aware diffusion alone reaches 63.2 (+0.5); semantic modulation alone gives 63.0 (+0.3); combining both gives 63.5 (+0.8).
• Reward ablation: Accuracy + Format yields 61.1; adding the Tool bonus boosts to 63.0 (+1.9); adding the Spatial term reaches 63.5 (+0.5). The paper reports the Tool reward as providing the main gain while the Spatial term further regularizes querying.
• Hyper-parameter sensitivity: the rotation weight β and temporal bandwidth b are largely robust (performance stays within 0.5 points across tested values), while diffusion scale τ and quality mixing weight α are more sensitive — yet SPF-DPP consistently outperforms naive time-based sampling across all tested configurations. Best score 63.5 at β=2, τ=2, α=0.5, b=24.
• Robustness to pose noise: performance degrades gracefully — clean 63.5; mild noise (2%, 2°) 63.3 (−0.2); medium (5%, 5°) 63.1 (−0.4, described in text as a 0.4% drop); heavy (10%, 10°) 62.9 (−0.6). Replacing the Vipe backend with VGGT yields 63.2 (−0.3).
• Not reported: the paper does not report inference latency, wall-clock cost, or throughput figures; it states that the SLAM preprocessing is a one-time offline step and per-query cost is "negligible" without giving a number.

Methodology in Plain English

Stage 1 — Macro perception (which frames to look at first). An off-the-shelf SLAM system (Vipe) is run once offline over the video to get camera poses and depth maps, which are projected onto a ground-aligned BEV plane. Separately, FG-CLIP scores how semantically relevant each frame is to the question. Two signals are then fused:

• A viewpoint graph: pairwise frame distances in SE(3) combine normalized translation and a rotation angle (arccos of the rotation-matrix trace); these become affinities w_ij = exp(−½ d_ij²), kept only within a temporal window of bandwidth b, forming a sparse adjacency matrix.
• A diffusion kernel: a heat-kernel diffusion on the graph Laplacian, K_view = exp(−τL), propagates local affinities into global geometric relations and is positive semidefinite.
• Semantic calibration: raw scores go through a temperature-scaled softmax, and a diagonal quality matrix uses q_i = (1−α) + α ŝ_i, so query-relevant frames are favored while a nonzero floor preserves diversity.
• The two are combined as a DPP L-ensemble L_dpp = Q K_view Q, and a fixed-size subset of k frames is chosen by greedy MAP with rank-one Cholesky updates, giving a (1−1/e)-approximation guarantee.

Stage 2 — Micro verification (asking for new views). Starting from the selected keyframes plus the BEV image, the model iterates: at each step it can emit a text token, issue a pose query, or answer. A pose query is a BEV tuple (x, y, r); every real frame is scored by spatial similarity in position and orientation, and the best-matching frame is retrieved if its similarity passes the coverage threshold τ_s; otherwise an error prompt is returned signaling no camera coverage. Retrieved frames are appended to the model's context, and the number of queries is capped.

Training the querying policy. The reward combines task rewards inherited from DeepEyes — answer accuracy, output format, and a tool-use bonus — with the paper's own spatial grounding reward, which assigns −1 if any query in a trajectory matches no real frame above τ_s. Optimization uses GRPO with a frozen reference policy for KL regularization, and following DeepSeek-R1 the model is trained directly with RL with no supervised fine-tuning cold start. Only model-generated text tokens contribute to the reward; retrieved observations and tool metadata are masked out.

Setup. Qwen3-VL-8B is fine-tuned with GRPO for 80 cycles on H20 GPUs, with batches of 256 prompts, 16 rollouts per prompt, at most six tool-call steps, a KL coefficient of 0.0, and a response length capped at 20,480 tokens. Training data comes from the VLM-3R training split (and SQA3D for the cross-benchmark evaluation), with no leakage to test sets. Fixed SPF-DPP hyper-parameters are σ_t=1, β=2, b=24 (about one second at 24 FPS), τ=2, T=1, α=0.5 and k=32 frames.

Supplementary detail. The preprocessing pipeline reconstructs with Vipe, back-projects depth into a global point cloud, fits an Oriented Bounding Box using an optimization-based approximation, resolves the vertical axis by comparing the spans of the lowest and highest 5% percentile segments (the tighter cluster is the ground), and rasterizes onto the XY plane to produce the BEV map.

Why This Matters

Impact on research. The paper argues that prior spatial-intelligence MLLMs either inject 3D features (point clouds, depth) or bolt on 3D reconstruction modules as black-box preprocessing — neither lets the model actively request additional views while reasoning. EagleVision reframes spatial CoT as a tool-use problem in a shared 3D coordinate system and shows the querying policy can be learned from answer-level supervision alone, without annotated reasoning traces. It also shows the approach is backend-agnostic (swapping Vipe for VGGT costs only 0.3 points), which suggests the gains come from the macro–micro framework rather than a specific SLAM system.

Real-world applications:

• Embodied AI and robotics: an agent that decides which camera viewpoints it needs before grasping or navigating.
• Autonomous driving and scene understanding: judging distances and relative directions of objects from multi-view vehicle footage.
• AR/VR and smart-home assistants: answering questions such as "which object is closest to the TV" by actively looking from the right angle.
• Long-video and video-search systems: finding the small subset of frames in a long recording that actually carries the geometric information a query needs.

Industry relevance. The offline-once, query-cheap design is pitched as amortized-cost friendly for multi-query scenarios over the same scene, contrasting with 3D-capable LLMs that must re-invoke heavy 3D encoders on every call. The goal of reaching this performance without changing the underlying model architecture or visual input pipeline matters for teams that want to improve an off-the-shelf VLM rather than train a 3D-native one.

Future Directions

• Reducing dependence on external pose estimation. The paper reports graceful degradation under injected pose noise and a VGGT swap, but the framework still starts from an offline SLAM pass; learning or jointly refining geometry during reasoning is a natural next step.
• Scaling the query budget and video length. Queries are capped and the keyframe set is fixed at k=32; it is not reported how performance scales with many more queries or much longer videos.
• Broadening benchmarks and domains. Evaluation covers VSI-Bench and SQA3D, both requiring the reconstruction and BEV pipeline; generalization to outdoor, driving, or dynamic scenes with moving objects is left open.
• Extending from single-image tool use to multi-view tool use further. The paper positions EagleVision as extending interleaved-modal CoT (ChatGPT-o3, DeepEyes) from cropping/zooming on one image to geometry-aware multi-view evidence gathering — a direction with room for richer spatial tools beyond pose retrieval.

Target Audience

Researchers and practitioners working on multimodal LLMs, video understanding, and spatial/3D reasoning; robotics and autonomous-driving engineers who need models that decide which viewpoints to inspect; and reinforcement-learning researchers interested in tool-use policies trained without supervised chain-of-thought data. Readers should be comfortable with camera geometry, DPPs, and policy-optimization methods, since the method section is dense with SE(3) and kernel formulations.

Authors’ abstract

Video-based spatial reasoning -- such as estimating distances, judging directions, or understanding layouts from multiple views -- requires selecting informative frames and, when needed, actively seeking additional viewpoints during inference. Existing multimodal large language models (MLLMs) consume a fixed set of uniformly sampled frames and cannot request new views once reasoning begins, often missing the geometric cues necessary for reliable spatial judgments. We present EagleVision, a dual-stage framework that combines geometry-aware frame selection with active, Bird's-Eye-View (BEV)-grounded reasoning. In the first stage (macro perception), a semantics-perspective-fusion determinantal point process (SPF-DPP) selects a compact set of keyframes that jointly maximize semantic relevance and viewpoint diversity under a fixed token budget. In the second stage (micro verification), the model performs iterative spatial Chain-of-Thought: at each step it can either reason in text or predict a pose on the BEV plane to retrieve the nearest real frame, forming a closed-loop hypothesize-look-verify cycle. The querying policy is trained purely via reinforcement learning with a spatial grounding reward, requiring no human-annotated reasoning traces. On VSI-Bench and SQA3D, EagleVision achieves state-of-the-art performance among open-source vision-language models.