ShapeR: Robust Conditional 3D Shape Generation from Casual Captures

AI summary

Overview

• Research area: Computer vision, specifically 3D generative modeling and object-centric 3D scene reconstruction from casually captured image sequences.
• Technical level: Advanced. The paper assumes familiarity with latent generative models, flow matching/rectified flow, transformer architectures, signed distance fields, and visual-inertial SLAM.
• Scope: The paper introduces a multimodal, rectified-flow-based generative model that reconstructs complete, metrically accurate 3D meshes of individual objects from ordinary hand-held image sequences, along with a new evaluation benchmark of 178 objects across 7 real-world scenes.

What This Paper Is About

Most modern 3D shape generation models assume clean, unoccluded, well-segmented inputs, but real-world captures are messy: users move freely, objects are occluded, scenes are cluttered, and sensors are noisy. ShapeR addresses this mismatch by conditioning a generative 3D model on several complementary signals extracted from a casual image sequence, including sparse SLAM points, posed images, and machine-generated captions, so that it can reconstruct complete object geometry even when the input data is far from ideal. The authors also build a new benchmark specifically designed to measure performance under these realistic conditions, since existing datasets are either controlled studio captures or lack complete object-level ground-truth geometry.

Key Contributions

1. A multimodally conditioned rectified flow model for metric 3D shape generation from casually captured sequences. The model conditions on sparse point clouds, posed images, and captions, and is trained with on-the-fly cross-modal augmentations plus a two-stage curriculum spanning object-centric and scene-level datasets.
2. An implicit segmentation strategy: rather than depending on explicit 2D segmentation masks, ShapeR learns to identify the target object using 3D instance points and their 2D projections, making it more robust to imperfect or machine-generated segments.
3. A new evaluation benchmark of casually captured sequences containing paired images, SLAM points, camera parameters, captions, and complete 3D shape annotations for 178 objects across 7 diverse real-world scenes.
4. A demonstrated large margin over prior work in this challenging setting, reported as a 2.7× improvement in Chamfer distance compared to the state of the art.

Main Findings

• Large quantitative improvement on the ShapeR evaluation set: ShapeR reaches a Chamfer distance (CD, ×10²) of 2.375, Normal Consistency of 0.810, and F-score of 0.722, compared to the best listed non-ablated baseline LIRM at 8.047 CD, 0.683 NC, and 0.384 F1. Other baselines in the same table include DP-Recon (8.364 CD, 0.661 NC, 0.436 F1), FStereo/FoundationStereo-based fusion (6.483 CD, 0.677 NC, 0.435 F1), and EFM3D (13.82 CD, 0.614 NC, 0.276 F1). The paper summarizes the gap as a 2.7× improvement in Chamfer distance over the state of the art.
• Human preference also favors ShapeR: In a user study with 660 responses, ShapeR's meshes were preferred over TripoSG in 86.67% of cases, over Amodal3R in 86.11%, over Direct3DS2 in 88.33%, and over Hunyuan3D-2.0 in 81.11%.
• SLAM points are a critical input modality: Removing SLAM points degrades performance sharply (CD rises from 2.375 to 4.514; NC drops from 0.810 to 0.765; F1 falls from 0.722 to 0.486), showing that sparse metric geometry supplies shape information that images alone do not.
• Both image and point-cloud augmentations matter: Removing point augmentation worsens results (3.276 CD, 0.805 NC, 0.667 F1), and removing image augmentation gives 3.397 CD, 0.778 NC, 0.649 F1. The authors report that the no-image-augmentation variant must rely on explicit foreground segmentation and therefore suffers with noisy masks.
• Scene-level fine-tuning helps: Dropping the two-stage curriculum (3.053 CD, 0.801 NC, 0.689 F1) performs worse than the full model, supporting the combination of large-scale synthetic object pretraining with realistic scene fine-tuning.
• 2D point mask prompting disambiguates objects: Without it, the model sometimes reconstructs neighboring objects (2.568 CD, 0.813 NC, 0.701 F1 versus 2.375 CD, 0.810 NC, 0.722 F1 for the full model).
• Strong results on third-party casual datasets (one-directional recall-based metrics): On ScanNet++, ShapeR achieves 1.09 CD (×10²), 0.84 NC, 0.91 recall versus DPRecon's 7.69, 0.73, 0.45. On Replica, ShapeR achieves 1.77, 0.84, 0.82 versus DPRecon's 4.65, 0.75, 0.57. The authors note ShapeR often exceeds the ground-truth scans in completeness because those scans lack geometry in occluded regions.
• Competitive on controlled tabletop data: On DTC Active, ShapeR scores 0.94 CD (×10²), 0.91 NC, 0.94 F1 versus LIRM's 0.90, 0.94, 0.92 — roughly matching LIRM. On the more casual DTC Passive set, ShapeR scores 0.95, 0.91, 0.95 versus LIRM's 1.37, 0.91, 0.88, surpassing it. Each DTC variant has approximately 100 sequences.
• Monocular extension without retraining: Although trained on multi-view posed data, ShapeR can be configured for single-image metric reconstruction by pairing it with a metric point cloud and camera estimator such as MapAnything.
• Robustness trend: Across DTC Active, DTC Passive, and the ShapeR evaluation dataset, baseline metrics deteriorate as captures become more casual, while ShapeR remains comparatively stable. The authors describe the progression as non-linear, with the ShapeR evaluation set substantially harder than DTC Passive.

Methodology in Plain English

The pipeline takes an ordinary image sequence and breaks it into four steps.

1. Extract geometry and camera motion. An off-the-shelf visual-inertial SLAM system produces a sparse 3D point cloud and camera poses. In the implementation, this comes from Project Aria's Machine Perception Services using monochrome cameras and IMUs. During inference, sparse metric point clouds are obtained by tracking and triangulating high-gradient image regions.
2. Find objects. A 3D instance detection model operating on the posed images and point cloud predicts 3D bounding boxes. Points belonging to each object are refined inside the box using SAM2 to remove points from neighboring objects.
3. Assemble conditioning signals per object. For each object, the system gathers its sparse points, the frames in which it is visible (up to sixteen at 280×280 pixels at inference; two randomly sampled views per object during training), binary masks formed by projecting the object's 3D points into those images, camera intrinsic/extrinsic parameters encoded as Plücker rays, and a caption generated by a vision-language model prompted on a representative image.
4. Generate the shape. A rectified flow transformer built on a FLUX-style dual/single-stream diffusion transformer denoises a latent VecSet representation, conditioned on point tokens (encoded by a ResNet-style 3D sparse convolutional encoder), image tokens (encoded by a frozen DINOv2 backbone), text tokens (frozen T5 and CLIP encoders), and the 2D projection masks. The denoised latent is decoded into a signed distance field by a Dora-style VecSet VAE decoder, and marching cubes extracts the final mesh. Each object's points are normalized into a [-1, 1]³ cube before generation, and the resulting mesh is rescaled back to the original metric coordinates so the output matches real-world scale and placement. Sampling integrates the learned flow with midpoint sampling.

Training strategy. The model is first pretrained on over 600K artist-created object meshes across diverse categories, with heavy on-the-fly augmentations applied compositionally in the data loader: background compositing, occlusion overlays, visibility fog, resolution degradation, and photometric perturbations for images, plus simulated partial trajectories, varied point dropout, Gaussian noise, and point occlusion for SLAM points. It is then fine-tuned on object crops from Aria Synthetic Environments, which are less diverse but exhibit realistic image occlusions, inter-object interactions, and SLAM noise. No segmentation masks are used as input; the object of interest is learned implicitly from 3D point tokens and 2D projected point masks.

Model scale. The 3D VAE uses 8 encoder transformer layers and 16 decoder layers with hidden width 768 and 12 attention heads, trained for 200K steps with effective batch size 640 on 64 NVIDIA H100 GPUs. The rectified flow transformer has 16 dual-stream and 32 single-stream blocks, each with 16 attention heads and hidden width 1024, trained for 550K steps on 128 H100 GPUs with effective batch size 512, progressively increasing the latent sequence length. Both are optimized with Adam at a learning rate of 5×10⁻⁵. The latent code has variable length L in {256, 512, …, 4096} with feature width d = 64.

Benchmark construction. The evaluation dataset was recorded with Project Aria Gen 1 or Gen 2 glasses while an annotator walked casually through each scene. For each annotated object, the annotator moved the object to an uncluttered area, captured a high-resolution image, manually segmented it, and ran a state-of-the-art image-to-3D model to generate the geometry. This geometry was manually verified and aligned back into the original sequence using a web interface guided by SLAM points, with placement and dimensions checked by projecting the mesh onto the original images.

Why This Matters

Impact on research. The paper argues that the gap between clean laboratory inputs and messy real-world captures has been under-measured, and that multimodal conditioning (particularly sparse metric points alongside images) is a practical route to closing it. The released benchmark, code, and model weights give the community a way to evaluate object-centric reconstruction under conditions that match actual deployment rather than curated datasets. It also positions generative shape modeling and metric scene reconstruction as converging rather than separate lines of work.

Real-world applications:

• Augmented and virtual reality: Building persistent, metrically accurate 3D object twins from a user's walkthrough of a room, for spatial anchoring and content placement.
• Robotics and embodied agents: Generating complete object geometry from noisy sensor sweeps so manipulators know the full extent of an object, including surfaces they could not see.
• Digital content creation and e-commerce: Turning casual phone or glasses footage of a physical product into a usable 3D asset without a scanning rig or manual cleanup.
• Scene understanding and inventory: Automatically cataloging objects in cluttered environments, since each object is reconstructed independently with consistent scale and can be re-composed into a whole scene.

Industry relevance. The work comes from Meta Reality Labs Research with collaborators at Simon Fraser University, which signals direct relevance to smart-glasses and mixed-reality hardware pipelines where such capture conditions are the norm. The reliance on off-the-shelf SLAM, detection, segmentation (SAM2), and vision-language models means the approach composes existing production-grade components rather than requiring new sensing hardware, which lowers the barrier to adoption. The stated plan to release code, weights, and the evaluation dataset also makes it a candidate foundation for downstream product research.

Future Directions

• Improving low-evidence reconstructions. The authors report that objects captured with low image fidelity or seen in very few views produce incomplete or low-detail meshes. Better priors or view-selection strategies for such cases remain open.
• Separating closely stacked or attached objects. When objects sit on or against one another (for example, items on a table), reconstructed meshes sometimes absorb parts of adjacent structures. Cleaner isolation under physical contact is unresolved.
• Reducing dependence on upstream detection. ShapeR relies on 3D instance detection; missed detections or inaccurate boxes propagate directly and missed objects cannot be recovered, so tighter coupling between detection and generation is a natural next step.
• Extending to real monocular data. The authors show single-image metric reconstruction is possible without retraining via a point cloud and camera estimator such as MapAnything, and note that fine-tuning on real data collected in that monocular setup could improve it further.

Target Audience

This paper is most valuable to 3D vision and graphics researchers working on generative shape modeling, object-centric scene reconstruction, or 3D asset creation, particularly those interested in robustness under real-world capture conditions. It will also be useful to engineers building augmented reality, robotics, or spatial computing systems who need to turn casual device footage into complete metric 3D geometry. Readers without background in flow matching, latent 3D representations, or SLAM will find the method sections demanding, though the problem framing and results are accessible to a broader technical audience.

Authors’ abstract

Recent advances in 3D shape generation have achieved impressive results, but most existing methods rely on clean, unoccluded, and well-segmented inputs. Such conditions are rarely met in real-world scenarios. We present ShapeR, a novel approach for conditional 3D object shape generation from casually captured sequences. Given an image sequence, we leverage off-the-shelf visual-inertial SLAM, 3D detection algorithms, and vision-language models to extract, for each object, a set of sparse SLAM points, posed multi-view images, and machine-generated captions. A rectified flow transformer trained to effectively condition on these modalities then generates high-fidelity metric 3D shapes. To ensure robustness to the challenges of casually captured data, we employ a range of techniques including on-the-fly compositional augmentations, a curriculum training scheme spanning object- and scene-level datasets, and strategies to handle background clutter. Additionally, we introduce a new evaluation benchmark comprising 178 in-the-wild objects across 7 real-world scenes with geometry annotations. Experiments show that ShapeR significantly outperforms existing approaches in this challenging setting, achieving an improvement of 2.7x in Chamfer distance compared to state of the art.