EA3D: Online Open-World 3D Object Extraction from Streaming Videos

AI summary

Overview

Research area: Computer vision, specifically online 3D scene reconstruction and open-world 3D scene understanding from streaming video.

Technical level: Advanced. The paper assumes familiarity with Gaussian Splatting, visual odometry/SLAM, vision-language models, and differentiable rendering.

One-sentence scope: The paper introduces EA3D (ExtractAnything3D), an online framework that reconstructs and semantically interprets 3D objects from a single streaming RGB video without camera poses, depth ground truth, or pre-built geometry.

What This Paper Is About

Most existing 3D scene understanding systems are offline: they need a complete set of multi-view images, camera poses, and often pre-constructed 3D geometry before they can label or segment anything. This paper asks whether a system can instead behave like a person walking into an unfamiliar room, building and interpreting 3D objects on the fly from frames as they arrive. EA3D is the authors' answer: a unified online framework that simultaneously estimates camera motion, reconstructs geometry, and extracts open-world semantics and physical properties from a streaming video.

Key Contributions

1. A unified online open-world 3D object extraction framework that performs reconstruction and understanding simultaneously, without requiring geometric or pose priors.
2. A method that takes streaming video as input and uses historical knowledge to guide 3D object extraction at the current observation, with online joint updates of integrated features.
3. A knowledge-integrated feature map that embeds vision-language semantics, physical properties, CLIP and Grounded-SAM visual features, point maps, and confidence maps into Gaussian primitives through a feed-forward online update.
4. Support for a broad task set, evaluated across multiple benchmarks: photo-realistic rendering, semantic and instance segmentation, 3D bounding box estimation, semantic occupancy estimation, and 3D mesh generation.

Main Findings

• ScanNet multi-task results (Table 1): EA3D with its own self-estimated poses reaches PSNR 25.8, SSIM 0.89, segmentation mIoU 46.3 and mAcc 71.8, bounding box AP 57.9 and mAP 39.9, and occupancy IoU 55.4 and mIoU 33.9. With COLMAP-estimated poses (EA3D*), it reports PSNR 25.5, SSIM 0.87, mIoU 45.9, mAcc 71.2, AP 59.2, mAP 39.6, IoU 55.0, mIoU 34.3.
• Better than offline baselines on reconstruction and segmentation: Offline methods report lower PSNR and mIoU, for example LangSplat (18.4 PSNR, 0.69 SSIM, 27.5 mIoU, 51.3 mAcc), GaussianGrouping (19.6, 0.74, 32.6, 56.9), FeatureGS (23.9, 0.84, 41.1, 66.0), OpenGaussian (22.1, 0.80, 35.4, 61.7), and InstanceGaussian (24.5, 0.83, 40.5, 65.7).
• Competitive against methods trained for specific 3D understanding tasks: OpenScene reports segmentation mIoU 42.8 and mAcc 68.6, EmbodiedSAM reports 44.2 and 71.4, and SAM3D reports 39.2 and 62.3, with several of these not supporting reconstruction at all.
• Stronger than enhanced baselines: HiCOM+VFM reaches PSNR 22.6, mIoU 34.8, mAcc 61.9; MonoGS+VFM reaches 24.3, 36.3, 60.5; FeatureGS+HiCOM reaches 24.5, 40.8, 66.3; EmbodiedOcc with an added RGB loss reaches 17.6, 29.2, 54.8. EA3D-full reaches 25.8, 46.3, 71.8.
• Sparse-view and online-incremental stability on LeRF (Table 2): EA3D, using self-estimated poses, reports PSNR 21.9 / 21.8 / 23.2 and segmentation mIoU 53.8 / 55.0 / 57.4 at 10, 30, and 70 views respectively. The COLMAP-based baselines degrade more at low view counts: LangSplat reports 11.3 / 14.4 / 17.8 PSNR and 28.6 / 34.4 / 51.5 mIoU; FeatureGS reports 15.2 / 18.9 / 22.4 and 29.4 / 41.2 / 53.6; OpenGaussian reports 14.9 / 19.5 / 22.7 and 30.1 / 40.5 / 55.8; HiCOM+VFM reports 18.1 / 18.6 / 21.5 and 36.1 / 39.3 / 43.3.
• Aggressive frame rates on LeRF: EA3D reports 0.235 FPS per-frame optimization speed, versus HiCOM+VFM 0.102, FeatureGS 0.018, LangSplat 0.007, OpenGaussian 0.005, and Cut3R+VFM 0.648. Cut3R+VFM is reported as not supporting reconstruction or segmentation (shown as dashes) on that benchmark.
• Ablation (Table 3): The full model reaches PSNR 25.8, mIoU 46.3, mAcc 71.8 with training speed 0.23 and rendering speed 210 FPS. Removing the CLIP encoder gives 25.3 / 41.6 / 66.4; removing the SAM encoder gives 25.4 / 42.8 / 67.1; removing the semantic-awareness regularization term gives 25.1 / 44.3 / 70.5; removing the online updating strategy gives 24.6 / 44.5 / 69.7 and a training speed of 0.07 with rendering 110; removing online visual odometry gives 25.0 / 45.4 / 70.8; removing joint optimization gives 24.8 / 45.7 / 71.4. The starting HiCoM baseline is 22.6 / 34.8 / 61.9 with training speed 0.29 and rendering 230.
• Naive combinations can hurt: The authors report that merely attaching existing models together tends to perform poorly and can degrade overall performance because of conflicts between components.

Methodology in Plain English

EA3D is built on Gaussian Splatting, a 3D representation that renders scenes as many small ellipsoids, and on the HiCoM streaming Gaussian method, which it extends to work without predefined poses or multi-view inputs.

The pipeline has three parts:

1. Knowledge extraction and integration. For each incoming frame, a vision-language model is prompted to "Find and list all the possible objects in the given image." A dynamic online semantic cache tracks newly appearing categories and encodes them as vectors using a pretrained CLIP text encoder. Physical properties of objects and their parts are also extracted from the VLM and fused into this cache. A pretrained CLIP visual encoder and the Grounded-SAM encoder produce pixel-level masks and visual features; category masks are used to aggregate those features via k-nearest neighbors, then the semantic features are normalized, combined, and written into each Gaussian as an extra knowledge-integrated feature. This is done with a fast feed-forward step rather than slow per-frame optimization. Features are carried between adjacent keyframes using a softmax matching distribution computed over the feature maps of two consecutive frames.

2. Online 3D object extraction. Camera poses are estimated incrementally with a keypoint graph, using Cut3R combined with another method to merge a dense pixel-level point map with sparse points, which the authors say helps capture tiny objects. Cut3R poses drift over time, so the graph is updated during reconstruction with a bundle-adjustment-style cost function over the reprojection error. Gaussians are initialized at timesteps 0 and 1, and new Gaussians are added per frame by back-projecting the estimated inverse depth map and point map into 3D. Co-visible regions share translation and rotation parameters for consistency, newly observed areas get new Gaussians, stale high-opacity ellipsoids that no longer belong to objects are removed, and a one-step splitting strategy grows Gaussians where gradients indicate under-reconstruction.

3. Recurrent joint optimization. A semantic-awareness regularization term pulls Gaussians of the same category toward a shared mean scale, reducing redundant scale parameters. Rendering of features uses alpha blending. The total loss combines an L1 photometric loss, a depth loss between rendered and estimated depth, an L2 loss between the knowledge-integrated feature map and the rendered feature map, and the regularization term. Gradients from the whole scene optimize Gaussian parameters, features, and camera poses together, with no post-refinement stage.

Implementation specifics: EA3D is implemented based on HiCoM with weights λ1 = 0.25, λ2 = 0.1, and λ3 = 0.15. Each incoming frame gets 100 motion steps plus another 100 steps after new Gaussians are added. Every fifth frame is used as a test view. All experiments run on a single A100 80GB GPU. The two evaluation benchmarks are LERF, an in-the-wild dataset captured with the iPhone App Polycam containing common and long-tail categories, and ScanNet, an indoor dataset with instance-level segmentation and labels across 200 categories; the authors use 10 RGB sequences selected by an earlier work, without depth ground truth or human annotations.

Why This Matters

Impact on research: The paper targets a gap it identifies as open: prior joint reconstruction-and-understanding methods are offline and need full scene observations, while prior streaming and SLAM-based reconstruction methods either need pre-computed global poses, rely on sparse keyframe tracking with expensive post-refinement, or are tailored to a single task. EA3D positions a single framework as covering online reconstruction plus segmentation, bounding boxes, occupancy, and mesh generation at once.

Real-world applications the paper implies or mentions:

• Robotics operating in unfamiliar rooms where object identities and geometry are unknown ahead of time.
• Manipulation simulation and motion emulation, which the paper lists among downstream applications.
• Controllable 3D editing and object insertion or removal in reconstructed scenes.
• Automated 3D labeling and perception, which the broader impacts section notes could reduce reliance on manual data annotation.

Industry relevance: Streaming, pose-free operation is directly relevant to augmented and virtual reality, mobile capture from handheld devices, and embodied AI agents that must act while exploring rather than after a full capture pass. The reported speed advantage over offline Gaussian pipelines (0.235 FPS versus 0.102, 0.018, 0.007, and 0.005 FPS for compared methods on LeRF) points at practical deployment constraints. The paper also flags a labor-market concern: automating 3D labeling and perception may disrupt job sectors that depend on manual data annotation.

Future Directions

• Scaling to longer and larger sequences: the reported evaluations cover ScanNet sequences of the type used in prior work and LeRF at 10, 30, and 70 views; behavior over much longer streams is not reported in the provided content.
• Reducing accumulated pose error: the paper states that Cut3R poses introduce noticeable biases that accumulate over time, which the online keypoint graph only partially corrects.
• Extending beyond the reported tasks: the paper claims potential for downstream applications such as manipulation simulation, motion emulation, controllable 3D editing, and object insertion or removal, but the provided content does not report quantitative results for them.
• Clarifying limitations: the checklist states limitations are discussed in the supplementary materials, which are not included in the provided content, so the specific failure modes and scope constraints are not reported here.

Target Audience

This paper is most useful to computer vision and robotics researchers working on 3D scene understanding, Gaussian Splatting, online or streaming reconstruction, and SLAM-style systems, as well as to practitioners building embodied agents, mobile 3D capture, AR/VR content tools, or automated 3D annotation pipelines. Readers without background in differentiable rendering and 3D representations will find the terminology dense, though the core idea of "reconstruct and understand at the same time, from video alone" is accessible.

Authors’ abstract

Current 3D scene understanding methods are limited by offline-collected multi-view data or pre-constructed 3D geometry. In this paper, we present ExtractAnything3D (EA3D), a unified online framework for open-world 3D object extraction that enables simultaneous geometric reconstruction and holistic scene understanding. Given a streaming video, EA3D dynamically interprets each frame using vision-language and 2D vision foundation encoders to extract object-level knowledge. This knowledge is integrated and embedded into a Gaussian feature map via a feed-forward online update strategy. We then iteratively estimate visual odometry from historical frames and incrementally update online Gaussian features with new observations. A recurrent joint optimization module directs the model's attention to regions of interest, simultaneously enhancing both geometric reconstruction and semantic understanding. Extensive experiments across diverse benchmarks and tasks, including photo-realistic rendering, semantic and instance segmentation, 3D bounding box and semantic occupancy estimation, and 3D mesh generation, demonstrate the effectiveness of EA3D. Our method establishes a unified and efficient framework for joint online 3D reconstruction and holistic scene understanding, enabling a broad range of downstream tasks.