OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects

AI summary

Overview

Research area: Computer vision, specifically online/feed-forward 3D object reconstruction from monocular RGB video using 3D Gaussian Splatting.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, transformer cross-attention, memory banks, and novel-view-synthesis metrics (PSNR, SSIM, LPIPS).

Scope: The paper proposes OnlineSplatter, a feed-forward framework that incrementally reconstructs freely moving rigid objects as 3D Gaussians from unposed RGB frames, without camera poses, depth priors, or bundle optimization.

What This Paper Is About

Reconstructing a freely moving object from a single moving camera is hard because most existing methods assume static scenes and rely on camera poses, depth input, or expensive global optimization across all frames. OnlineSplatter instead processes each incoming RGB frame causally, anchoring everything to the first frame's coordinate system and refining an object-centric Gaussian representation as more views arrive. The goal is a representation that improves with every new observation while keeping memory and runtime constant, and that never needs pre-computed poses or depth maps.

Key Contributions

1. A pose-free, online feed-forward reconstruction framework that operates on monocular RGB streams, requires no camera poses, depth priors, or global optimization, and is claimed to maintain constant computational complexity regardless of sequence length. The reconstruction is anchored in the canonical coordinate system defined by the first frame.

2. A dual-key 3D Object Memory module that pairs a latent appearance-geometry key with an explicit directional key, enabling spatial-guided memory reading (an Orientation-Aligned Read and an Orientation-Complementary Read) and principled memory sparsification.

3. A spatial-guided memory readout plus a sparsification mechanism that keeps the memory bank compact while maintaining comprehensive viewpoint coverage; pruning uses attention-based usage and average angular distance between direction keys.

4. A stage-wise evaluation protocol (early: 1 ≤ t ≤ 4; mid: 5 ≤ t ≤ 10; late: 11 ≤ t ≤ T, where T = N/2) with experiments on GSO and HO3D showing state-of-the-art results on freely moving objects.

Main Findings

• GSO late-stage reconstruction: OnlineSplatter reaches 31.737 PSNR, 0.969 SSIM, and 0.075 LPIPS, versus the strongest baseline (NPS_dist3) at 24.141 PSNR, 0.863 SSIM, and 0.125 LPIPS. The paper reports improvements of up to +7.596 PSNR and +0.106 SSIM on GSO.

• GSO early and mid stages: 26.329 PSNR / 0.921 SSIM / 0.084 LPIPS at early stage and 27.553 PSNR / 0.933 SSIM / 0.066 LPIPS at mid stage, versus NPS_dist3 at 23.331 / 0.862 / 0.149 (early) and 23.206 / 0.861 / 0.138 (mid).

• HO3D real-world results: 23.627 PSNR / 0.910 SSIM / 0.152 LPIPS (early), 25.803 / 0.912 / 0.122 (mid), and 27.928 / 0.952 / 0.099 (late), versus the best baseline at those stages (NPS_dist3 at 21.134 / 0.853 / 0.162 early; NPS_dist3 at 22.967 / 0.869 / 0.165 mid; NPS_dist3 at 22.947 / 0.860 / 0.163 late). Reported late-stage gains are +4.981 PSNR and +0.092 SSIM on HO3D.

• Improvement over time: The paper emphasizes that adapted baselines using explicit frame selection often show unstable or stagnant performance, while OnlineSplatter consistently improves as more observations accumulate — from 26.329 to 31.737 PSNR on GSO and from 23.627 to 27.928 PSNR on HO3D between early and late stages.

• Dual-key memory ablation (normalized averaged metric M_avg, GSO): Full model scores 0.699 / 0.734 / 0.810 across early, mid, and late stages. Removing the latent key drops to 0.545 / 0.582 / 0.596; removing the direction key yields 0.699 / 0.701 / 0.723; encoding memory from unpatchified Gaussian parameters instead of transformer tokens yields 0.541 / 0.582 / 0.611; random pruning instead of the proposed sparsification yields 0.697 / 0.728 / 0.764.

• Training strategy ablation (M_avg, GSO): Removing staged training drops to 0.545 / 0.582 / 0.588. Removing the ray alignment loss gives 0.562 / 0.599 / 0.682; removing the background penalty gives 0.675 / 0.712 / 0.795; removing the depth loss gives 0.691 / 0.728 / 0.805. Sequential-only frame sampling gives 0.645 / 0.682 / 0.688 and random-only sampling gives 0.697 / 0.728 / 0.764.

• Constant cost claim: The paper states that every new frame triggers a single O(1) memory-and-time update and that the system maintains constant memory and runtime regardless of sequence length. Specific runtime or memory measurements in absolute units are not reported in the provided content.

• Ablation text attributions: Removing the latent key is described as degrading performance at all stages due to loss of visual-geometrical cues; removing the direction key is described as impacting later stages, reflecting its role in spatial coverage.

Methodology in Plain English

Input handling. The system takes a stream of RGB frames. An off-the-shelf online video segmentation model, XMem, produces an object mask for each frame, and the background is removed before encoding. This means the method deliberately ignores background surfaces, which is a departure from methods that rely on large static backgrounds for stability.

Dual image encoding. Each masked frame is encoded by concatenating features from two encoders: a frozen DINO backbone for self-supervised appearance cues, and a trainable encoder with the same architecture that is trained end-to-end to capture geometry-aware features.

Transformer with three token types. The OnlineSplatter Transformer processes reference-view tokens (from the first frame), source-view tokens (from the current frame), and memory-readout tokens (fetched from the object memory). Learnable view embeddings and positional embeddings are added to distinguish the token types. Output tokens are decoded by an unpatchifier into 4N Gaussian primitives per timestep: 2N from the memory group, N from the reference group, and N from the source group, where N = H × W for the input resolution.

No accumulation of outputs. Rather than appending newly predicted Gaussians over time, the model fetches its current understanding of the object from memory and refines it with the latest observation, avoiding explicit global aggregation.

Memory design. The memory is a key-value bank where each entry has two keys and one value. The latent key is produced by a lightweight learnable encoder from tokenized features. The direction key is a unit directional vector derived from azimuth and polar angles predicted by a pre-trained zero-shot 3D orientation estimator. Values come from a trainable encoder applied to the transformer's source-view output tokens. Direction keys are broadcast from a 1×3 vector to p×3 so all patches in a view share the same directional key.

Memory reading. The current latent key serves as the latent query. The direction query is the normalized average of the current and reference direction keys. Two reads happen per timestep: an Orientation-Aligned Read that scores entries matching both queries, and an Orientation-Complementary Read that matches the latent key but rewards differing orientations to capture complementary viewpoints. Scores are divided by a temperature τ_t = 2.5 − σ_t, where σ_t in [0,1] is the orientation estimator's confidence, damping potential inaccuracies. Retrieved features are attention-weighted sums of memory values.

Memory pruning. When the memory hits its maximum size S, the system drops 20 percent of entries. It sorts entries by coverage C_i (average angular dot product of an entry's direction key with all others), keeps the top 50 percent as a dense subset, and from that subset removes the 40 percent lowest-ranked entries by usage U_i (averaged accumulated cross-attention weight). The pruned entries collectively constitute 20 percent of the full memory cap.

Training. A two-stage strategy is used. Warm-up training optimizes the view encoder, positional and view embeddings, transformer, and unpatchify decoder without the memory module. Main training adds the memory module and trains the whole network end-to-end. The loss combines a photometric term (MSE on object regions, plus a background penalty that penalizes Gaussian color and opacity outside the object's visual hull) and a geometric term (a ray alignment component and a relative depth MSE), balanced by a weight λ_g. Rendering supervision uses a differentiable rasterizer, and the reference and source Gaussian subgroups are additionally rendered with the poses of V_0 and V_t to encourage specialization.

Training setup. The model is trained and evaluated at 256×256 resolution on 8× A100 GPUs: 250K steps with batch size 64 in warm-up, and 500K steps with batch size 16 in main training, sampling 3-5 views per object during warm-up and 6-12 views during main training. Training data is 100K objects sampled from Objaverse, rendered with a custom trajectory script producing look-at jitter, varying focal lengths, and randomized lighting, with a unique trajectory per object. Evaluation uses 36 rendered frames per GSO object and HO3D for real-world hand-object interaction sequences with occlusions.

Baselines. Because no prior feed-forward model supports pose-free RGB-only online reconstruction, the authors adapt FreeSplatter (with random-4 and distance-4 frame selection, denoted FSO_rand4 and FSO_dist4) and NoPoSplat (fine-tuned on Objaverse with object mask supervision, using 2 or 3 diverse frames, denoted NPS_dist2 and NPS_dist3).

Why This Matters

Research impact. The paper targets a gap that most pose-free reconstruction methods explicitly side-step: freely moving objects. By removing the requirements for camera poses, depth input, and bundle adjustment, and by showing that a feed-forward memory module can substitute for cross-frame optimization, it opens a route to online object reconstruction where the representation is not a fixed output but a state that improves causally.

Real-world applications.

• Autonomous robots that must build object models on the fly in unpredictable environments without calibrated pose or depth sensors.
• Augmented reality devices that need to track and reconstruct physical objects in real time from a single RGB stream.
• Interactive 3D content creation, where a user films an object and continuously gets a better model while filming.
• Robotic manipulation and in-hand object scanning scenarios, since the method is evaluated on HO3D hand-object interaction sequences.

Industry relevance. The constant-memory, constant-runtime claim is the practical selling point: many reconstruction pipelines scale cost with sequence length, which is prohibitive for embedded or always-on devices. Removing the depth sensor requirement also matters for hardware cost and for platforms that only ship RGB cameras. The project page is listed as markhh.com/OnlineSplatter.

Future Directions

• Extending beyond rigid objects. The method explicitly targets a freely moving rigid object; articulated, deformable, or multi-object scenes are not addressed in the reported setup.
• Reducing dependency on segmentation and orientation modules. The pipeline relies on XMem for masks and a pre-trained zero-shot orientation estimator whose confidence is used as a temperature term; errors or failures in either module propagate into the memory keys and readout.
• Quantifying the constant-cost claim. The paper asserts constant memory and runtime but does not report absolute runtime or memory measurements in the provided content, so concrete efficiency numbers would strengthen the practical case.
• Broadening evaluation. Results are reported on GSO (rendered) and HO3D (hand-object interaction). Behavior on cluttered real-world scenes, longer sequences, and other object categories is not reported.

Note: the provided paper content is truncated during the "Mesh Visual Comparison" discussion, so the appendix details referenced throughout (custom trajectory script, hyperparameter values, normalization of M_avg, progressive sampling schedule, and the mesh comparisons in Figure 4) are not included in the content analyzed here.

Target Audience

Researchers and engineers working on 3D reconstruction, 3D Gaussian Splatting, neural rendering, or SLAM-adjacent online perception; practitioners building AR/VR, robotics, or interactive 3D capture systems who need pose-free, sensor-light, bounded-cost reconstruction; and readers already comfortable with transformer memory mechanisms and novel-view-synthesis evaluation who want to understand how temporal memory can replace bundle adjustment for moving objects.

Authors’ abstract

Free-moving object reconstruction from monocular video remains challenging, particularly without reliable pose or depth cues and under arbitrary object motion. We introduce OnlineSplatter, a novel online feed-forward framework generating high-quality, object-centric 3D Gaussians directly from RGB frames without requiring camera pose, depth priors, or bundle optimization. Our approach anchors reconstruction using the first frame and progressively refines the object representation through a dense Gaussian primitive field, maintaining constant computational cost regardless of video sequence length. Our core contribution is a dual-key memory module combining latent appearance-geometry keys with explicit directional keys, robustly fusing current frame features with temporally aggregated object states. This design enables effective handling of free-moving objects via spatial-guided memory readout and an efficient sparsification mechanism, ensuring comprehensive yet compact object coverage. Evaluations on real-world datasets demonstrate that OnlineSplatter significantly outperforms state-of-the-art pose-free reconstruction baselines, consistently improving with more observations while maintaining constant memory and runtime.