PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-Forward Planar Splatting

AI summary

Overview

Research area: Computer vision, specifically metric 3D reconstruction of indoor scenes and planar scene representations.

Technical level: Intermediate. The paper assumes familiarity with Vision Transformers, stereo matching, differentiable rendering, and standard 3D reconstruction metrics (Chamfer distance, F-score, pose error), but the core ideas are explained without requiring deep prior expertise in planar splatting.

Scope: The paper introduces Plana3R, a pose-free, feed-forward model that predicts metric relative camera poses and a compact set of sparse 3D planar primitives from two unposed indoor images, trained without any 3D plane annotations.

What This Paper Is About

Indoor scenes are full of flat surfaces, so representing them with 3D planar primitives is more compact than using dense point clouds or per-pixel Gaussians. Existing planar reconstruction methods usually need either dense plane-level 3D annotations (which limits how much data they can train on and hurts zero-shot generalization) or accurate multi-view camera poses from densely captured images. Plana3R removes both dependencies: it learns planar 3D structure from only depth and normal supervision and predicts relative pose jointly with the geometry, in a single feed-forward pass from an unposed image pair.

Key Contributions

1. A pose-free feed-forward framework for metric planar reconstruction. Plana3R takes two images with known intrinsics and outputs both a 6-DoF relative camera pose and a set of sparse 3D planar primitives at metric scale, with no per-scene optimization and no known poses at inference.

2. Planar splatting used as a supervision mechanism without plane annotations. Instead of training against plane masks or 3D plane correspondences, the model is trained by rendering depth and normal maps from predicted primitives through PlanarSplatting's CUDA-based differentiable rasterizer and comparing them to ground truth. This makes large-scale stereo datasets usable.

3. A Hierarchical Primitive Prediction Architecture (HPPA). Primitives are predicted at two resolutions (H/16 × W/16 and H/8 × W/8), and a gradient-magnitude heuristic on predicted normals selects where high-resolution primitives are needed. Higher-resolution primitives are only used for pixels whose normal gradient exceeds a threshold g_th.

4. Instance-level plane segmentation as an emergent capability. Because the representation is explicitly planar, merging predicted primitives by normal and distance thresholds yields semantically meaningful 2D and 3D plane segmentation, despite the model never being trained on plane masks.

Main Findings

• Two-view reconstruction and pose on ScanNetV2 (4051 image pairs, 303 scenes): Plana3R reaches a Chamfer distance of 0.11 and F-score of 92.52, versus 0.21 and 74.92 for MASt3R and 0.26 and 61.86 for NOPE-SAC. Translation error is 0.07 m median / 0.13 m mean; rotation error is 2.01° median / 3.16° mean.

• Zero-shot transfer to Matterport3D (6083 image pairs, 13 scenes): Even though the model was never trained on Matterport3D, it achieves 0.32 Chamfer and 56.63 F-score, compared with 0.49 and 30.01 for MASt3R, and it outperforms methods trained specifically on that dataset (SparsePlanes, NOPE-SAC). Translation error is 0.24 m median, 0.45 m mean. Rotation median/mean (2.00°/4.49°) is worse than MASt3R's (0.98°/4.66°) on this dataset.

• Zero-shot monocular metric depth on NYUv2 (654 test frames): Plana3R reports Rel 0.132, RMSE 0.463, δ1 86.4, δ2 96.3, δ3 98.4, and log10 0.059. MASt3R reports Rel 0.152, RMSE 0.51, δ1 83.0, δ2 95.6, δ3 98.7, and log10 0.058 — so Plana3R leads on most but not all of these metrics.

• Single-view plane segmentation and reconstruction on Replica (498 sampled images): Plana3R scores RI 0.89, VOI 1.62, and SC 0.63, versus 0.85, 1.81, and 0.58 for PlaneRecTR, a method trained with plane annotations. Plane recall improves substantially: 7.79 vs 1.78 at 0.1 m depth, 30.74 vs 13.46 at 0.6 m depth, 28.52 vs 3.79 at 5° normal error, and 36.31 vs 18.92 at 30°.

• Multi-view (more than two views) evaluation on 50 eight-view ScanNetV2 samples: Plana3R is run pairwise and merged. It achieves RRA@5 0.9000, RTA@5 0.3935, RRA@10 0.9985, RTA@10 0.7442, RRA@15 1.0000, RTA@15 0.8614. MASt3R, also run pairwise, gets RRA@5 0.9828, RTA@5 0.2657, RRA@10 0.9964, RTA@10 0.5371, RRA@15 1.0000, RTA@15 0.6878 — MASt3R is better on rotation accuracy at the tighter thresholds, Plana3R on translation.

• Primitive sparsity ablation: With g_th = 0 the model uses 3072 primitives per view (Chamfer 0.10, F-score 93.10 on ScanNetV2; NYUv2 RMSE 0.45, δ1 86.8). With g_th = 0.5 it uses 1417 primitives for reconstruction and 1565 for depth (Chamfer 0.11, F-score 92.52; RMSE 0.46, δ1 86.4). With g_th = 10 it uses 768 primitives (Chamfer 0.11, F-score 92.32; RMSE 0.49, δ1 85.3). Roughly half the primitives perform comparably; only using low-resolution primitives degrades accuracy noticeably.

• Effect of non-overlapping training pairs: The training set contains approximately 0.57M image pairs with no overlap and 3.43M pairs randomly sampled from nearby frames (mainly within the next 10 frames). Ablations on ScanNetV2 and MP3D stratified by overlap (easy >50%, medium 15–50%, hard <15%, very hard / non-overlap) show that including the non-overlapping pairs improves pose accuracy, particularly in the hard and non-overlap regimes.

• Training scale: The model was trained for 256 GPU-days on NVIDIA H20 GPUs with per-GPU batch size 6, at input resolution 512 × 384, using AdamW with learning rate decaying from 1×10⁻⁴ to 1×10⁻⁶, one warm-up epoch on the patch and pose losses followed by 10 epochs with all losses. The number of model parameters is not reported.

Methodology in Plain English

The approach starts by encoding both images through a shared (Siamese) Vision Transformer, producing feature maps at 1/16 of the input resolution. Two transformer decoders with cross-attention turn these into low-resolution embeddings. A separate head reads the concatenated low-resolution features and regresses the relative camera pose as a quaternion and a translation, both kept in metric units rather than normalized.

For geometry, three regression heads predict, for each spatial patch, a planar primitive described by its center depth, its two radii, and a quaternion defining its orientation; the plane normal is the rotated z-axis. A deconvolution network upsamples the embeddings to 1/8 resolution, and the same heads predict a second, finer set of primitives there. To decide which regions need the finer set, the model computes the spatial gradient of the low-resolution predicted normals and only keeps high-resolution primitives where that gradient exceeds a threshold, merging the two sets via binary masks.

Training proceeds in two stages. Early on, primitives are randomly initialized and often land outside the camera frustum, so directly rendering them fails. A patch loss first supervises predicted per-patch depth and normals against ground-truth maps resized to the two patch grids, stabilizing position and orientation. After warm-up, a rendering loss is added: the primitives are rasterized by PlanarSplatting's differentiable renderer into full-resolution depth and normal maps, which are compared with ground truth. This renders gradients back through the primitives, refining radii and overall geometric fidelity. A pose loss combines translation and quaternion errors with a relative-angle term. No plane masks or plane annotations are used anywhere in training.

At inference, a single forward pass produces the pose and the primitives. To obtain planar surfaces rather than a primitive soup, primitives are merged using thresholds on normal and distance error between pairs, following PlanarSplatting, which also yields instance-level plane semantics.

Training data comes from ScanNetV2, ScanNet++, ARKitScenes, and Habitat, totaling roughly four million image pairs. Ground-truth depth comes from the datasets; ground-truth normals are generated as pseudo-labels with Metric3Dv2 where only depth exists. Only a small subset of Habitat images is used because of their low rendered visual quality.

Why This Matters

The paper shows that structured indoor geometry can be learned at scale without the expensive plane-level annotation that has bottlenecked feed-forward planar reconstruction, and that a sparse planar representation can outperform dense point-cloud representatives like MASt3R on several metrics while remaining compact enough for instance-level plane semantics to fall out for free. It also argues that indoor scenes are a good training ground for metric 3D foundation models, because human-scale object dimensions are distributed similarly across scenes.

Real-world applications:

• Digital twins of buildings: Reconstructing floors, walls, ceilings, and furniture from casually captured image pairs supports virtual tours, floorplan generation, and building documentation.
• AR/VR content creation: Compact planar primitives are a more efficient scene format for real-time rendering and interaction than dense point clouds.
• Robotics and navigation: Metric pose plus an explicit plane representation is directly useful for obstacle avoidance, layout understanding, and manipulation planning.
• Interior design and e-commerce: Instance-level plane segmentation of walls and furniture enables automatic room measurement and furniture placement from photos.

Industry relevance is reinforced by the author affiliations (Ant Group, HKUST, Penn State, Wuhan University, Zhejiang University) and by funding through the Ant Group Research Intern Program and Ant Group Postdoctoral Program. The use of ViT backbones pre-trained on DUSt3R 512-DPT weights means the method can be built on existing open foundation-model checkpoints.

Future Directions

• Single-pass multi-view inference. Plana3R currently handles more than two views only pairwise, constructing N−1 image pairs and merging results for N images. A unified many-view forward pass is an open extension.
• Closing the remaining metric gaps. On Matterport3D, MASt3R still has better rotation median and mean error, and on NYUv2 Plana3R does not lead on log10 or δ3. Targeted improvements in pose and depth at fine scales remain open.
• Richer primitive models. The current representation handles only planar surfaces; non-planar, texture-rich, or curved regions are approximated by small primitives, and the paper notes these cases are relatively rare but not eliminated.
• Scaling and generalization beyond indoor scenes. The paper frames indoor regularity as the enabling condition for metric prediction; whether the same annotation-free planar splatting supervision transfers to outdoor or mixed environments is not tested.

Target Audience

Researchers and engineers working on 3D reconstruction, indoor scene understanding, stereo foundation models, and neural rendering will get the most from this paper. It is also relevant to practitioners in AR/VR, robotics, and digital-twin pipelines who need metric geometry and pose from unposed image pairs without per-scene optimization or plane annotation. Readers should have some grounding in Vision Transformers and differentiable rendering to follow the architecture and loss design fully.

Authors’ abstract

This paper addresses metric 3D reconstruction of indoor scenes by exploiting their inherent geometric regularities with compact representations. Using planar 3D primitives - a well-suited representation for man-made environments - we introduce PLANA3R, a pose-free framework for metric Planar 3D Reconstruction from unposed two-view images. Our approach employs Vision Transformers to extract a set of sparse planar primitives, estimate relative camera poses, and supervise geometry learning via planar splatting, where gradients are propagated through high-resolution rendered depth and normal maps of primitives. Unlike prior feedforward methods that require 3D plane annotations during training, PLANA3R learns planar 3D structures without explicit plane supervision, enabling scalable training on large-scale stereo datasets using only depth and normal annotations. We validate PLANA3R on multiple indoor-scene datasets with metric supervision and demonstrate strong generalization to out-of-domain indoor environments across diverse tasks under metric evaluation protocols, including 3D surface reconstruction, depth estimation, and relative pose estimation. Furthermore, by formulating with planar 3D representation, our method emerges with the ability for accurate plane segmentation. The project page is available at https://lck666666.github.io/plana3r