AnchorComplete: Recovering Complete Scene Meshes from Partial Point Clouds

AI summary

Overview

• Research area: 3D computer vision — generative scene reconstruction and shape completion from partial point clouds (indoor scenes, meshes).
• Technical level: Advanced. The paper assumes familiarity with latent diffusion / rectified-flow generative models, 3D latent representations (structured latents, O-Voxels), and standard 3D reconstruction metrics.
• Scope (one sentence): The paper presents AnchorComplete, a generative framework built on TRELLIS.2 that turns a partial, noisy scene-level point cloud (e.g., from a feed-forward reconstructor) into a complete metric-scale scene mesh, evaluated on Replica and SCRREAM.

What This Paper Is About

Image-based 3D reconstructors (DUSt3R-style feed-forward models, SLAM systems, scanners) only recover what the cameras actually saw, so their point clouds contain large holes and leave everything outside the observed views unresolved. The paper asks whether a learned 3D generative prior can fill in those unobserved regions and output a single complete mesh, rather than a point cloud or voxel grid. The stated goal is to do this while (i) staying faithful to the geometry that was actually observed and (ii) working at any physical scene size, from a small room to a large apartment, without per-scene optimization.

Key Contributions

1. A new problem setting and protocol. The authors frame "complete scene meshes from partial point clouds" as its own task — no densely sampled posed images, no oriented normals, no per-scene optimization — and formulate strong baselines plus evaluation protocols for it (including separating visible-region accuracy from full-scene completeness in the metrics).
2. Partial-point conditioning. A hybrid conditioning scheme that anchors generation to the input cloud through (a) global anchor tokens (m = 768) read by cross-attention in the flow transformer, and (b) local spatial injection of visible-voxel evidence directly into the first-stage structure latent, via a warm start and a per-step visibility-mask concatenation.
3. Metric-scale coarse-to-fine generation. An octree of windows with fixed physical size (level 0 is the whole scene cube; level 1 and level 2 windows cover occupied space, with level-2 windows half as wide as level-1), each level using a 512³ mesh grid, so mesh resolution in metres is decoupled from scene extent. Windows inherit their parent's geometry through a "parent warm start" (α = 0.8) and overlap by a quarter of their width.
4. A working instantiation on TRELLIS.2 that produces a single merged mesh in the original scene coordinate frame, and beats the evaluated baselines on complete-scene metrics on Replica and SCRREAM, degrading more slowly as input views are removed.

Main Findings

• Best complete-scene metrics at N_views = 4. On Replica, AnchorComplete reaches CD_all 0.067, F 0.1 = 0.819 and F 0.05 = 0.684, versus 0.132 / 0.777 / 0.647 for the closest baseline Seen2Scene and 0.211 / 0.633 / 0.527 for NKSR. On SCRREAM it reaches CD_all 0.076, F 0.1 = 0.786 and F 0.05 = 0.584, versus 0.167 / 0.692 / 0.525 for Seen2Scene.
• Visible-surface fidelity is not sacrificed. CD_vis is 0.055 on Replica and 0.057 on SCRREAM; DA3 remains slightly better on CD_vis (0.048 and 0.052), which the authors attribute to DA3 reconstructing only observed surfaces.
• Runtime. About 3 minutes per scene on an RTX PRO 6000 Max-Q — faster than Seen2Scene (~10.4 min) and GenRC (~6.6 min), slower than the feed-forward reconstructors (<1 s) and NKSR (1 s).
• Gains come from conditioning, not just scene-level training. A scene-fine-tuned TRELLIS baseline (TRELLIS*) improves over the released object model but still trails AnchorComplete substantially (Replica CD_all 0.288 vs. 0.067).
• Ablation: local injection matters most. Removing local spatial injection degrades Replica most (CD_vis 0.055 → 0.072, CD_all 0.067 → 0.105); removing global anchor tokens gives a smaller drop (CD_vis 0.060, CD_all 0.086), which the authors read as the tokens supplying scene-level context rather than localization.
• Ablation: finer levels add detail. On five held-out real-world scenes and one Matterport3D house, going from level 0 to level 2 lowers accuracy error from 0.045 to 0.028 and raises F 0.02 from 0.485 to 0.801 (F 0.05 changes from 0.835 to 0.872 from level 0 to level 2).
• Sparse-view robustness. Across N_views ∈ {2, 4, 6, 8, 10}, AnchorComplete leads CD_all and F-scores over DA3, NKSR and Surflo at every view count tested, with the widest CD_all gap to NKSR at N = 2. At N = 10, F 0.05 is 0.829 vs. 0.754 for NKSR on Replica and 0.697 vs. 0.674 on SCRREAM.
• Robust to imperfect input without retraining. Performance stays stable when only a quarter of the input points are kept and under Gaussian noise up to roughly the voxel scale; beyond that it degrades gradually.
• Diverse completions. An appendix figure shows varied plausible completions from the same partial observation, suggesting the model is not simply replaying one dataset-specific shape prior.

Methodology in Plain English

The system takes a colored point cloud of a room and produces a closed mesh of the whole room, including parts no camera ever saw. It is built by fine-tuning TRELLIS.2, a pretrained two-stage 3D generator: the first stage decides where geometry exists (an occupancy/active-voxel layout) and the second stage decides what it looks like (shape latents that get decoded into a mesh).

Two changes make that object-scale prior work for scenes:

1. Telling the model what was actually seen. The point cloud is encoded by a frozen Utonia backbone plus a small trained Q-Former into 768 "anchor tokens," which the generator attends to — much like it would normally attend to an image or text prompt. Separately, the observed points are voxelized and pushed straight into the first-stage latent: visible cells start from the encoded observation instead of pure noise, and a binary visibility mask is fed in at every denoising step so the model always knows which cells are evidence and which are guesses.
2. Not squeezing a whole apartment into one cube. If the entire scene is normalized into a single cube, each grid cell covers more and more centimetres as the scene grows, and thin things like chair legs vanish. Instead, the model first generates a coarse whole-scene layout at level 0, then refines it in windows of a fixed physical size at level 1, and again at half that size at level 2. Inside a window the model behaves exactly as before and never needs to know how big the scene is. Each window starts not from scratch but from a noisy version of its parent's prediction (α = 0.8, a "warm start"), so neighbouring windows agree on where walls and furniture are. At the finest level the decoded windows are merged into a common grid and a single mesh is extracted, avoiding explicit stitching.

Training uses about 16k scenes from InternScenes and about 7k rooms from SAGE, rendered as multi-view RGB-D at 640×480 with an 85° horizontal field of view; both stages are fine-tuned for 40k steps with rectified-flow objectives, and the parent fed to a window is usually (p = 0.7) a prediction rather than a damaged ground-truth parent, so the model learns to correct imperfect parents.

Why This Matters

• Research impact: It reframes scene completion as conditional generation on a metric 3D observation and shows that a fixed-resolution generative prior can be scaled to arbitrary scene sizes, which is a structural problem for any latent-cube 3D generator. It also supplies baselines and metrics (splitting accuracy from completeness) for a task that previously had no standard protocol.
• Real-world applications (as motivated in the paper):
  • Novel-view AR/VR rendering of captured environments.
  • Large-scale physical simulation, where closed, watertight geometry is required.
  • Robotics, where complete meshes are needed for collision detection and physical interaction.
  • General digital twins of physical spaces, since output lands directly in the original scene coordinate frame.
• Industry relevance: The method needs no per-scene optimization, takes a partial point cloud from existing reconstructors or scanners, and produces a mesh downstream tools can consume — attractive for pipelines built on DUSt3R/VGGT/DA3-style reconstruction or SLAM, where holes currently block rendering and simulation.

Future Directions

• Pushing to further levels. The framework allows levels 3, 4, … by halving the window again; the paper only reports up to level 2, so the benefit and cost of deeper levels are open.
• Better handling of severely degraded input. Performance degrades gradually beyond roughly voxel-scale noise or below a quarter of the input points; making the model robust to much sparser or noisier scans is an explicit limitation.
• Scoring unobserved regions. Accuracy is only computed inside input-view frustums because completion outside them is multi-modal and the ground truth is a single realization; evaluating diverse completions fairly remains an open problem.
• Beyond indoor rooms. Evaluation covers Replica and SCRREAM (indoor) plus two qualitative real-world rooms; generalization to other scene types and to fully automated large-scale deployment is not reported.

Target Audience

Researchers and graduate students in 3D vision, generative modeling, and neural rendering who work on scene reconstruction, shape completion, or 3D generative priors; practitioners building digital-twin, AR/VR, simulation, or robotics pipelines who need complete meshes from sparse captures; and anyone wanting a clear case study in adapting an object-scale 3D generator (TRELLIS.2) into a scalable, condition-anchored scene-level system.

Authors’ abstract

Most image-based 3D reconstructors primarily recover visible surfaces, producing partial point clouds with large holes and leaving unobserved geometry unresolved. To address this limitation, we introduce AnchorComplete, a framework that reconstructs complete scene meshes from partial point clouds while preserving observed geometry and supporting scenes of varying scales. Building on TRELLIS.2, our method combines partial-point conditioning, which integrates global scene tokens with local spatial evidence to anchor generation to the input, and metric-scale coarse-to-fine generation, which progressively refines scene geometry within fixed-size windows while maintaining global structural consistency. Experiments on Replica and SCRREAM demonstrate that AnchorComplete consistently outperforms strong baselines in complete-scene reconstruction, particularly under sparse-view settings.