REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting

AI summary

Overview

Research area: Computer vision and 3D scene understanding — specifically open-world 3D reasoning segmentation and language-driven 3D editing built on 3D Gaussian Splatting (3DGS), driven by a Multimodal Large Language Model (MLLM) agent.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, Neural Radiance Fields, SAM, and MLLMs, and builds its method on top of all of them.

Scope: This paper introduces REALM (arXiv:2510.16410), an MLLM-agent framework that segments and edits 3D objects from implicit natural-language instructions without 3D-specific post-training, and releases the REALM3D benchmark for evaluating such reasoning.

Authors and affiliations: Changyue Shi, Minghao Chen (equal contribution), Yiping Mao, Chuxiao Yang, Xinyuan Hu, Jiajun Ding (corresponding author), and Zhou Yu, with affiliations at Hangzhou Dianzi University and Peking University. Project page: https://ChangyueShi.github.io/REALM.

What This Paper Is About

Existing 3D open-vocabulary segmentation systems handle direct queries like "segment the cup" but fail on instructions requiring reasoning such as "segment the object between the lamp and the book," while 2D vision-language models that can reason about such instructions have no intrinsic 3D spatial grounding. REALM bridges this gap by using 3D Gaussian Splatting as a photorealistic proxy for the 3D world, letting off-the-shelf MLLMs reason over rendered views and then lifting those 2D decisions back into consistent 3D object masks. The framework also supports downstream 3D edits — object removal, replacement, and style transfer — driven by textual instructions.

Key Contributions

1. REALM framework. An MLLM-agent framework for 3D reasoning segmentation that uses 3DGS as a proxy to lift the 2D reasoning ability of MLLMs into the 3D domain, and that supports downstream object-level 3D interactions (removal, replacement, style transfer) from complex textual instructions.

2. MLLM-Based Instance Segmenter (LMSeg). A component that performs image-level reasoning segmentation and infers the corresponding Gaussian identity by combining MLLM priors with SAM masks and an optimized 3D feature field.

3. Global-to-Local Spatial Grounding (GLSpaG). A hierarchical aggregation procedure that first surveys the scene from multiple global views to coarsely localize the target, then synthesizes close-up views of the object for fine-grained 3D mask refinement.

4. Benchmark contributions. LERF and 3D-OVS are re-annotated with implicit queries, and REALM3D is introduced, comprising 100+ scenes and 1000+ implicit prompt–mask pairs (the statistics table reports 100 scenes and 1,444 prompt–mask pairs).

Main Findings

• State-of-the-art on implicit queries across three benchmarks. REALM reaches 92.88 mIoU / 90.12 mBIoU on LERF, 93.68 / 86.02 on 3D-OVS, and 82.30 / 70.37 on REALM3D. Baselines Gaga, GAGS, and GS-Group score substantially lower — for example, on LERF: Gaga 44.82 / 42.37, GAGS 17.84 / 15.87, GS-Group 42.43 / 40.01. The paper states baselines perform more than 50% lower on LERF and more than 35% lower on 3D-OVS compared to REALM.

• Direct single-view MLLM input is unstable. Feeding one or a few random rendered views to the MLLM makes results highly sensitive to viewpoint selection. The researchers repeated this procedure 10 times in their ablation on the "Figurines" scene, whereas REALM grounds the target with minimal stochasticity.

• Each GLSpaG stage adds measurable accuracy. On "Figurines": GS-Group baseline 0.32 mIoU / 0.30 mBIoU; GS-Group + Qwen2.5-VL 0.78 / 0.77; plus Global Reasoning 0.89 / 0.88; plus Local Refinement 0.95 / 0.94.

• Camera sampling strategy is decisive. K-means clustering plus Top-K-ID selection achieves 0.95 / 0.94, versus K-means + Random 0.76 / 0.75, Totally Random 0.59 / 0.58, and no K-means 0.38 / 0.38.

• Cluster count matters in both directions. Without K-means, mIoU is 0.38; with N_cluster = 2 it is 0.76; with N_cluster = 24 it is 0.95; with N_cluster = 128 it drops to 0.56. Both too few and too many clusters hurt.

• Robust to the number of global cameras. N_global = 4 gives 0.81 / 0.80, while N_global = 8 and N_global = 16 both give 0.95 / 0.94.

• Excessive local refinement overfits. Refinement at 10 iterations gives 0.94 / 0.93, at 50 iterations 0.95 / 0.94, but at 500 iterations 0.79 / 0.76 and at 1,000 iterations 0.74 / 0.71.

• Rendering stays fast. REALM renders at 354.72 FPS, compared with GS-Group at 305.79, Gaga at 204.49, and GAGS at 107.06, because the pipeline renders only single-channel masks.

• Inference completes in under 10 seconds. Total time is 8.68 s per prompt: 2.53 s calling the MLLM in the global stage, 2.48 s in the local stage, and 3.67 s for 50 local refinement iterations. MLLM calls for each view can run in parallel.

• Qualitative reasoning examples. In the "Teatime" scene, the query "Which is the drink the teddy bear is holding?" yields the coffee mug, where baselines latch onto keywords "teddy bear" and "drink." For "What is a refreshing beverage made of fruit?" the model infers orange juice. In a scenario where everyone else is asleep but the user wants to watch videos, REALM selects an earphone.

• Explicit-query results are not in the main text. The paper states that quantitative results for explicit queries are provided in the supplementary materials; only the implicit-query numbers appear in the article content.

Methodology in Plain English

The pipeline has four moving parts.

First, the team builds a 3D feature field on top of the Gaussian Splatting scene. SAM extracts instance masks from each input image, and a temporal propagation model associates those instances across views so every object keeps a consistent identity ID. Each Gaussian primitive is given a feature vector, which can be rendered into a 2D feature map through alpha blending. A classifier applied to that map predicts a pixel-wise instance ID, and it can later be applied directly to Gaussian features to group them into 3D instances.

Second, the LMSeg agent performs image-level reasoning. Given a rendered view and a language query, an MLLM returns a 2D bounding box, an object category, and a short explanatory rationale. The bounding box goes to SAM to produce a binary mask, and intersecting that mask with the predicted instance map identifies which instance ID the target corresponds to.

Third, global grounding. Rather than trusting one view, the method clusters the training camera poses with K-means, picks representative cameras, and selects the top N_global views that observe the largest number of distinct instances. LMSeg runs on each selected view and the resulting instance IDs are combined by majority voting to determine the target identity. That identity is then used to produce a coarse 3D mask over the Gaussians.

Fourth, local refinement. Cameras whose instance maps contain the target ID are collected, LMSeg produces fine-grained 2D masks on those close-up views, and the 3D mask is optimized so that when rendered back into those views it matches the 2D masks under an L1 loss. The paper sets N_cluster = 24, N_global = 8, and 50 refinement steps, and all results were obtained on a single NVIDIA RTX 3090 GPU.

For evaluation, the team re-annotated LERF and 3D-OVS with implicit prompts using Qwen2.5-VL plus manual curation, selecting 2 representative LERF scenes and 5 3D-OVS scenes. REALM3D was built from 100+ multiview scenes with point clouds and camera poses generated by VGGT, annotated with Qwen2.5-VL and SAM into 1k+ prompt–mask pairs.

Why This Matters

Impact on research. The paper points at a specific gap between 3D open-vocabulary segmentation (good at explicit queries) and MLLMs (good at reasoning but not spatially grounded). Prior attempts such as ScanReason and VGMamba predict 3D bounding boxes rather than fine masks, and ReasonGrounder relies heavily on a top-down view, which limits it in complex environments. REALM shows that an off-the-shelf MLLM agent plus clever multi-view aggregation can produce fine-grained, consistent 3D masks without 3D-specific post-training, and it supplies a benchmark (REALM3D, 100 scenes and 1,444 prompt–mask pairs versus 36 pairs across 5 scenes for LERF and 150 pairs across 10 scenes for 3D-OVS) to measure progress.

Real-world applications

• Robotics and embodied assistants that must follow loose instructions such as "make the table tidier" by identifying storage containers and loose clutter.
• AR/VR content authoring, where a user removes, replaces, or restyles objects in a reconstructed scene via text.
• Scene editing for film, games, and architecture, using reconstruction-based representations where edits must respect occlusion relationships.
• Accessibility and human-AI collaboration systems that let users refer to objects by function or context rather than by name.

Industry relevance. Because the pipeline runs on a single RTX 3090, processes a prompt in 8.68 s, renders at 354.72 FPS, and does not require training a 3D-specific MLLM, it is a practical template for products built on top of Gaussian Splatting reconstructions. The benchmark and re-annotated datasets also give companies a way to compare vendors on instruction-following rather than on keyword matching.

Future Directions

• Dependency on MLLM and SAM quality. The pipeline inherits the failure modes of the underlying MLLM and SAM; how it behaves when either errs is not characterized in the paper content.
• Scaling and robustness of REALM3D evaluation. The benchmark has 100 scenes and 1,444 prompt–mask pairs, and most reported evaluations use a small subset (2 LERF scenes, 5 3D-OVS scenes); broader evaluation across the full benchmark is an open step.
• Tuning the sampling pipeline. Performance swings sharply with the number of clusters (0.95 at N_cluster = 24 versus 0.56 at N_cluster = 128) and degrades badly with too much refinement (0.74 at 1,000 iterations), suggesting adaptive or learned view and step selection as a follow-up.
• Extending to dynamic and interactive settings. The current work segments and edits static reconstructed scenes; reasoning about moving objects or closed-loop robotic interaction is not demonstrated in the paper content.

Target Audience

Researchers and engineers working on 3D scene understanding, open-vocabulary segmentation, Gaussian Splatting, embodied AI, and multimodal agents who want to connect 2D vision-language reasoning to precise 3D grounding. It is also relevant to practitioners building text-driven 3D editing tools, and to benchmark designers interested in how implicit, reasoning-based prompts are collected and evaluated. Readers need prior exposure to NeRF/3DGS representations, SAM, and MLLM prompting to follow the method sections.

Authors’ abstract

Bridging the gap between complex human instructions and precise 3D object grounding remains a significant challenge in vision and robotics. Existing 3D segmentation methods often struggle to interpret ambiguous, reasoning-based instructions, while 2D vision-language models that excel at such reasoning lack intrinsic 3D spatial understanding. In this paper, we introduce REALM, an innovative MLLM-agent framework that enables open-world reasoning-based segmentation without requiring extensive 3D-specific post-training. We perform segmentation directly on 3D Gaussian Splatting representations, capitalizing on their ability to render photorealistic novel views that are highly suitable for MLLM comprehension. As directly feeding one or more rendered views to the MLLM can lead to high sensitivity to viewpoint selection, we propose a novel Global-to-Local Spatial Grounding strategy. Specifically, multiple global views are first fed into the MLLM agent in parallel for coarse-level localization, aggregating responses to robustly identify the target object. Then, several close-up novel views of the object are synthesized to perform fine-grained local segmentation, yielding accurate and consistent 3D masks. Extensive experiments show that REALM achieves remarkable performance in interpreting both explicit and implicit instructions across LERF, 3D-OVS, and our newly introduced REALM3D benchmarks. Furthermore, our agent framework seamlessly supports a range of 3D interaction tasks, including object removal, replacement, and style transfer, demonstrating its practical utility and versatility. Project page: https://ChangyueShi.github.io/REALM.