Lie Routing Vision Language Models for View-Robust 3D Understanding

AI summary

Overview

• Research area: Computer vision and multimodal large language models (MLLMs) for 3D scene understanding from multi-view RGB video, with a specific focus on geometric deep learning (Lie groups, SO(3)).
• Technical level: Advanced. The method depends on Lie group/Lie algebra formalism (𝔰𝔬(3), SO(3), exponential maps, Padé–Cayley approximations) and assumes familiarity with 3D MLLM architectures such as VG LLM and geometry encoders such as VGGT.
• Scope: The paper diagnoses "viewpoint-induced semantic drift" in geometry-injected 3D MLLMs and proposes LieVLM, a framework in which geometry tokens predict local SO(3) rotations that adapt pretrained visual features, evaluated on 3D captioning, grounding, detection, and spatial reasoning benchmarks.

What This Paper Is About

3D multimodal LLMs that take multi-view RGB frames typically inject geometry features into visual tokens by simple addition, which adds spatial context but does not explicitly adapt visual semantics when the camera moves. The authors show that the same object and the same question can produce inconsistent, diffuse, or mislocalized responses as the viewpoint changes, a phenomenon they call viewpoint-induced semantic drift. LieVLM addresses this by using the geometry tokens to predict small, learned rotations in three-dimensional subspaces of the visual features, so that the semantic representation is adjusted in a structured, viewpoint-dependent way while the original semantics and direct 3D context are preserved.

Key Contributions

1. Diagnosis of viewpoint-induced semantic drift. The authors characterize how target-related responses in 3D MLLMs weaken, become diffuse, or shift toward irrelevant regions as camera pose changes, using query-relevant activations and an SO(3) geodesic viewpoint-variation measure ΔR, and relate this drift to downstream task degradation.
2. Lie-State Elicitation. Geometry tokens from VGGT are used to predict latent 𝔰𝔬(3) generators (reshaped into M triplet groups) plus token- and channel-wise routing coefficients γ, with the routing bias initialized to −4 so that γ starts near σ(−4) ≈ 0.018.
3. Lie-Group Routing with a closed-form Padé–Cayley residual. Projected visual triplets are transformed by the predicted generators, and only the identity-relative correction is projected back, so the correction vanishes when the operator is the identity and the pretrained semantic path is retained.
4. Geometry-Aware Routed Fusion and broad evaluation. The final representation is F^2d + F^3d + γ ⊙ Δ_op, combining the original semantics, a direct 3D context path, and the gated routed correction, with consistent gains over additive fusion across benchmarks, training scales, and backbones.

Main Findings

• 3D dense captioning (Scan2Cap): LieVLM-8B achieves the best CIDEr of 82.7, beating Video-3D LLM by 2.7 points and LLaVA-3D by 3.5 points, and improving over VG LLM-8B from 80.0 to 82.7. Other metrics are BLEU-4 41.7, METEOR 28.9, and ROUGE 62.9.
• 3D visual grounding (ScanRefer): LieVLM-8B reaches 64.5 Acc@0.25 and 57.2 Acc@0.5, improving over published VG LLM-8B by 6.9 and 6.3 points under the same Mask3D proposal refinement protocol.
• 3D video object detection (EmbodiedScan): F1 improves from 41.2 to 45.8 in the 4-frame setting and from 40.0 to 44.1 in the 6-frame setting relative to VG LLM-8B; gains are described as particularly strong on table, bed, and bathtub.
• SPAR-Bench Full (3.05M training mixture): LieVLM-8B averages 68.57 versus 67.18 for VG LLM-8B (+1.39), with the largest category gain on ViewChgI (35.26, +3.15), and gains of +0.73 (Low), +1.43 (Medium), +1.14 (High), +0.74 (CamMotion), +1.16 (DistI-OO-MV), +0.41 (ObjRel-OC-MV), and +0.62 (SpImag-OO-MV).
• CV-Bench-3D and BLINK-Spatial: CV-Bench-3D average rises from 92.6 to 94.3 (+1.7), with Depth 93.9 (+1.7) and Distance 94.6 (+1.6); BLINK-Spatial average rises from 75.9 to 77.1 (+1.2).
• VSI-Bench across scales and backbones: With the 297K setting, the average rises from 50.6 to 52.4 (+1.7); with Qwen3-VL-8B and the 3.05M mixture, from 60.6 to 62.8 (+2.2), improving seven of eight categories, including all four multiple-choice categories shown.
• Routing tracks viewpoint: Across same-scene frame pairs, latent operator state variation has the strongest monotonic association with camera rotation (Spearman ρ = 0.870), compared with 0.452 for geometry features and 0.576 for cross-attention residuals.
• Structure and conditioning both matter (three-seed ablation): Additive Only scores 80.07 CIDEr / 57.62 Acc@0.25 / 41.18 and 40.05 F1 on 4- and 6-frame detection; generic cross-attention and an unconstrained learnable 3×3 generator give modest or near-additive gains; spectral- and rational-controlled generators reach 80.76 and 81.48 CIDEr; the 𝔰𝔬(3)–Padé formulation is strongest at 82.69 / 64.56 / 45.76 / 44.10.
• Realization and bias choices: Padé-1/1 gives the highest CIDEr (82.7) and detection F1 (45.8) among the compared realizations (Rodrigues, Taylor-1/2/3, Padé-2/2); b_γ = −4 gives the best CIDEr and detection precision, while b_γ = −3 gives higher recall.
• Qualitative behavior: Lie routing preserves more localized, target-centered query-relevant activations across a fixed camera trajectory, and fitted task trends show LieVLM retaining stronger performance as sample-level viewpoint variation ΔR increases.
• Not reported: The paper does not report inference latency, parameter counts added by the routing modules, or absolute numbers for the 8M SenseNova-SI comparison beyond citing 64.8 as external scale context.

Methodology in Plain English

The model starts from a standard 3D MLLM setup: a 2D visual encoder produces semantic tokens, a geometry encoder (VGGT) produces geometry tokens aligned to the same visual token grid, and prior work simply adds the two.

LieVLM changes the second step in three stages:

1. Reading a "rotation state" from geometry. The geometry tokens are normalized and passed through two small heads. One predicts a set of axis-angle-like numbers per token, which are reshaped into groups of three channels (triplets); each triplet is interpreted as a tiny rotation generator. The other head predicts a per-token, per-channel gate between 0 and 1, initialized low (bias −4, so roughly 0.018 at start) so routing begins as a near-no-op.
2. Rotating visual features in small 3D subspaces. Visual tokens are projected into the same triplet layout, each triplet is rotated by its predicted generator, and only the change relative to doing nothing is kept. This change is computed in closed form using a Padé 1/1 (Cayley-equivalent) approximation, expressed via the first- and second-order generator actions and a rational normalization term. Because only the residual is kept, an identity rotation produces zero correction, so the model cannot accidentally destroy the pretrained representation.
3. Gated fusion with both original paths. The output is the 2D visual feature plus the direct geometry feature plus the gate-weighted rotation correction. The language model therefore still sees plain semantic content and plain 3D context; the routing branch only supplies a viewpoint-adaptive adjustment.

Training is end-to-end on the task objective with no supervision on the predicted rotation parameters or gates; the 2D encoder, 3D encoder, and multimodal connector are frozen. The evaluation covers 3D captioning, grounding, video object detection, and a 3.05M-example spatial reasoning mixture built from CA-MLLM public data sources, with controlled comparisons where baseline and LieVLM share the same backbones, data, optimization, and evaluation.

Why This Matters

Impact on research. The paper reframes geometry injection in 3D MLLMs as an operator problem rather than a feature concatenation problem: geometry tokens predict a structured transformation of semantic state rather than merely adding context. It also introduces a diagnostic (target-response localization plus the ΔR viewpoint-variation measure) that future work can use to quantify viewpoint robustness, and it shows that a properly constrained rotation family outperforms generic cross-attention and unconstrained learnable 3×3 generators in matched three-seed experiments.

Real-world applications:

• Robot manipulation and embodied agents: consistent object grounding when the agent's camera moves around a scene.
• AR/VR and telepresence: stable object references and captions as a headset or handheld camera changes pose.
• Autonomous driving and inspection drones: robust language-guided detection of the same object across a moving multi-camera rig.
• Scene digitization and asset annotation: language-driven labeling, captioning, and spatial question answering over captured indoor scans.

Industry relevance. The method is a plug-in module on existing geometry-injected video MLLMs, keeps the encoders frozen, and yields gains that hold when moving from Qwen2.5-VL-7B to Qwen3-VL-8B and from 297K to 3.05M training examples, which is attractive for teams that already have a VG LLM-style pipeline and want better viewpoint robustness without retraining the vision stack.

Future Directions

• Explicit pose at inference. The routing module deliberately does not take relative camera poses as input; testing whether optional pose conditioning improves accuracy, or removes the need for it, is a natural next step.
• Extending beyond SO(3). Only rotation is modeled. Translation, scale change, and occlusion are not represented by the Lie state, so richer groups (for example SE(3)) or separate handling of scale and visibility remain open.
• Efficiency and deployment cost. The paper reports no latency, memory, or added-parameter figures for the routing branch, leaving its cost relative to additive fusion unquantified.
• Beyond the evaluated regimes. All results are on indoor ScanNet-style and related spatial benchmarks with fixed 4- and 6-frame settings; validation on larger and more heterogeneous frame counts, outdoor scenes, and 8M-scale training (where only an external SenseNova-SI reference is cited) is not established.

Target Audience

Researchers and graduate students working on 3D multimodal LLMs, spatial reasoning, and geometry-aware representation learning, particularly those already familiar with geometry-injection pipelines such as VG LLM, SPAR, or LLaVA-3D. It also suits practitioners in embodied AI, AR/VR, and robotics who need language grounding that stays stable as cameras move, and readers interested in applying Lie group machinery to feature adaptation rather than to point-cloud equivariance. Beginners will find the core idea accessible through the fusion equation and the ablation table, but will need background in SO(3) and exponential maps to follow Section 3 in detail.

Authors’ abstract

Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVLM, a Lie routing framework for view-robust 3D understanding. Motivated by the rotational component of viewpoint change, Lie routing adapts visual representations by applying local $\mathrm{SO}(3)$ transformations to projected features within learned three-dimensional subspaces. Geometry tokens extracted by VGGT predict local rotation parameters and channel-wise routing coefficients. A closed-form Padé--Cayley map transforms projected visual features, whose identity-relative corrections are projected back and gated into the original visual and geometry features. This design retains the pretrained semantic and direct spatial context while learning view-dependent feature adjustments. Experiments on 3D captioning, grounding, detection, and spatial reasoning show consistent improvements over additive fusion across training scales and backbones. Controlled ablations examine the roles of geometric conditioning and transformation structure. Viewpoint analyses further reveal systematic variation in the predicted routing states alongside more localized target responses and stronger task performance across viewpoint changes.