Easy-3Dfy: Object-Level Geometric Composition for Single-Image 3D Scene Construction

AI summary

Overview

Research area: Computer Vision — single-image 3D scene construction, combining 3D object generation, monocular geometry (pointmap) estimation, and point-cloud registration.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, monocular depth and pointmap estimation, similarity transforms, and standard 3D reconstruction metrics.

Scope: Easy-3Dfy is a modular, training-efficient pipeline that builds a full 3D scene from one image by generating each object independently and then geometrically registering each generated mesh onto a monocular scene pointmap, evaluated on Scene-Maker (1,000 scenes) and 3D-FUTURE200 (200 scenes).

What This Paper Is About

Reconstructing a complete, correctly arranged 3D scene — all objects plus their positions, orientations and scales — from a single image is hard because it requires two different skills: making detailed 3D objects, and understanding where things sit in space. End-to-end systems such as SAM-3Dfy do both at once, but the authors argue this coupling limits the fine-grained quality of both the meshes and the layouts, producing interpenetrating or floating objects.

Easy-3Dfy instead splits the problem in two: a replaceable image-to-3D model builds each object on its own, and a learned registration module (a Rectified Point Flow, or RPF, aligner) figures out where that object belongs in a point cloud estimated from the input image. The goal is better scene geometry without ever training on complete, calibrated 3D scenes.

Key Contributions

1. A compositional workflow for single-image 3D scenes that decouples object-asset generation from scene-level spatial reconstruction, connected by three explicit interfaces: object-local meshes, a scene pointmap, and local-to-pointmap-scene transforms. Each module can be swapped without redefining the problem.

2. An RPF-based geometric aligner that recasts scene construction as object-level point-cloud registration: given an object's anchor point cloud and a partial query cloud extracted from the pointmap, it predicts pointwise correspondences, from which a uniform-scale similarity transform (scale, rotation, translation) is recovered by Umeyama fitting.

3. An object-level training scheme that synthesizes alignment pairs from single meshes via rendered RGB–depth views, the same monocular geometry model used at inference, random occlusions, and independent random rotations/scales/translations on the anchor and query sides — avoiding any need for complete scene-level supervision.

4. A controlled validation and scalability study, including a comparison against SAM-3Dfy using identical meshes to isolate layout quality, and a mesh-provider sweep (GT, SAM-3Dfy, TRELLIS-2.0, HY3D-2.0, HY3D-3.1, Meshy-T2, Tripo-V3) showing the workflow inherits gains from better upstream object generators without retraining.

Main Findings

• Better layouts with identical assets (Scene-Maker): Using the same SAM-3Dfy meshes, Easy-3Dfy improves Scene-CD from 0.261 to 0.242, Scene-FScore from 0.615 to 0.649, and BBox IoU from 0.366 to 0.399; Object-CD (0.188) and Object-FScore (0.695) are identical by construction, isolating the layout effect.

• Better layouts with identical assets (3D-FUTURE200): Scene-CD improves from 0.146 to 0.140, Scene-FScore from 0.820 to 0.836, and BBox IoU from 0.517 to 0.554, again with identical Object-CD 0.139 and Object-FScore 0.807.

• Outperforms other baselines on all reported metrics: On Scene-Maker, MIDI scores Scene-CD 0.442 / Scene-F 0.329 / Object-CD 0.379 / Object-F 0.384 / BBox IoU 0.065 and SceneGen scores 0.455 / 0.368 / 0.264 / 0.543 / 0.085, both well behind Easy-3Dfy. On 3D-FUTURE200, MIDI scores 0.275 / 0.627 / 0.171 / 0.736 / 0.319 and SceneGen 0.320 / 0.584 / 0.168 / 0.747 / 0.292.

• Mesh quality helps, but geometric compatibility matters more than detail: With everything else fixed on Scene-Maker, GT meshes give the ceiling (Scene-CD 0.242, Scene-F 0.660, BBox IoU 0.415). SAM-3Dfy meshes reach BBox IoU 0.399; Tripo-V3 0.391; HY3D-3.1 0.385; Meshy-T2 0.381; TRELLIS-2.0 0.317; HY3D-2.0 0.291. The authors attribute the gap to how well an asset's dimensions and proportions match the observed pointmap, not to texture or surface detail alone.

• Surface normals matter; RGB does not: Removing surface-normal conditioning degrades layout on Scene-Maker (Scene-CD 0.245, Scene-F 0.641, BBox IoU 0.394 vs. 0.242 / 0.649 / 0.399) and more clearly on 3D-FUTURE200 (0.150 / 0.812 / 0.535 vs. 0.140 / 0.836 / 0.554). A retrained 9-channel checkpoint including RGB performs worse than the 6-channel XYZ+normal model on both datasets — e.g., 3D-FUTURE200 Scene-F 0.7973 vs. 0.8136 and BBox IoU 0.5231 vs. 0.5428.

• Point-cloud cleaning is nearly free to drop: Omitting rule-based cleaning leaves Scene-Maker metrics essentially unchanged (Scene-CD 0.242, Scene-F 0.646, BBox IoU 0.399) but costs more on 3D-FUTURE200 (0.148 / 0.824 / 0.544), which the authors credit to RPF's noise-tolerant augmented training.

• Pointmap choice dominates completion-model choice: With Meshy-T2 fixed, swapping MogeV2 for Depth Anything V3 hurts (Scene-Maker Scene-CD 0.267 vs. 0.253, BBox IoU 0.356 vs. 0.381; 3D-FUTURE200 0.169 vs. 0.146 and 0.476 vs. 0.543). Replacing FLUX-2 with GPT-Image-2 is mixed: slightly better scene metrics on 3D-FUTURE200 (Scene-CD 0.142 vs. 0.146, Scene-F 0.822 vs. 0.814) but worse object metrics (0.156 vs. 0.151, 0.775 vs. 0.782) and no gain on Scene-Maker.

• Qualitative failure modes: Small or heavily occluded objects, contaminated masks, and unreliable pointmaps for reflective or transparent surfaces can leave the transform underconstrained and produce inaccurate poses.

• Metrics have blind spots: Scene-CD, Scene-FScore and BBox IoU do not capture appearance, texture, material, semantic identity, scene completeness, perceptual plausibility, or fine topology. Object-CD and Object-FScore, computed with independent object-local normalization followed by FilterReg/ICP registration, are asset-geometry diagnostics rather than evidence about layout.

Methodology in Plain English

The pipeline has two halves that never need to be trained together.

Building the objects. Given an image and one mask per object (masks come from the dataset, since segmentation is out of scope), the method crops each object, feeds it to a prompted image-editing model that removes occluders and hallucinates the hidden parts on a black background, and then passes the completed image to any off-the-shelf image-to-3D generator. The output is a mesh in its own local coordinate frame. Crucially, the method throws away whatever pose or layout the provider might output.

Figuring out where each object goes. A separate monocular geometry model turns the input image into a pointmap (a per-pixel 3D point) and surface normals, which define the common scene coordinate frame. Masking gives a partial point cloud of the visible part of each object. The aligner is fed two point sets: the anchor (8,192 points sampled from the generated mesh) and the query (8,192 points from the pointmap mask), each represented only by XYZ plus normal — 16,384 points total, no color.

The aligner. RPF is a DiT-based conditional generative model that, for every query point, predicts where that point sits on the complete object — e.g., a visible headlamp in the photo gets mapped to the headlamp region of the generated car. Those point-to-point correspondences are then fed to a closed-form Umeyama fit that recovers one uniform scale, rotation and translation. That transform is applied to the entire mesh, not just the sampled anchors, and all transformed meshes are assembled into the scene.

How the aligner is trained. No scene data is needed. For each training mesh, the authors render pixel-aligned RGB and depth from random viewpoints, run the same monocular geometry model on the render to get a realistic partial query, add random occlusions using 8 crops, and read the ground-truth correspondence positions straight off the rendered depth. Independent random rotations, scales (0.8–1.2) and translations (0–0.3), plus axis-wise shape scaling (0.9–1.1) on each side, force the model to learn geometry rather than memorize a canonical frame. The normal channel is zeroed with probability 0.5 during training. Training uses roughly 21,000 × 40 asset instances with 20 rendered views each, batch size 128, 2 nodes of 8 NVIDIA A100 GPUs, and 50 flow steps.

How it is scored. Before metrics, each predicted scene is aligned to ground truth with a fixed uniform scale from the AABB diagonal ratio, then 60 Adam iterations optimizing only rotation and translation under a Chamfer objective — one global transform for the whole scene. F-score uses τ = 0.1, following MIDI's scene-level threshold.

Why This Matters

Impact on research. The paper is evidence that decoupling beats joint training for scene construction under a fixed-mask protocol: with the same meshes, explicit geometric registration improves layout over an end-to-end scene model. It also argues that object-level registration data is far cheaper to synthesize than scene-level supervision, since no calibrated multi-object scenes are required — which makes the workflow a clean testbed where improving either 3D generation or 3D foundation models immediately improves the whole system.

Real-world applications:

• E-commerce and furniture retail: turning a single product-in-room photo into a placeable 3D asset in a metric room layout.
• AR/VR content authoring: populating a captured room with detailed, correctly scaled 3D objects from one photograph.
• Robotics and simulation: generating scene geometry for simulators or grasp planning from a single RGB view.
• Digital twins and interior design: rapid, editable room reconstructions where each object remains a separate, replaceable asset.

Industry relevance. The modular design is the commercial pitch: each stage is a swappable vendor-style component (image editor, mesh generator, pointmap model), and the mesh-provider table shows customers can upgrade their 3D generator alone and get better layouts with no retraining. The authors report a project page and code release at https://easy3dfy.github.io, and commit to publicly releasing the method.

Future Directions

• Fixing underconstrained cases. Small objects, severe occlusion, bad masks, and reflective or transparent surfaces still break the transform; the authors point to Appendix A.7 for a failure-pattern analysis but do not propose a remedy.
• Richer evaluation. The paper explicitly calls for protocols combining geometric, visual, semantic and completeness criteria, since current metrics miss texture, material, identity and perceptual plausibility — no perceptual or appearance metrics are reported here.
• Joint reasoning about inter-object relationships. The object-level formulation deliberately ignores object-object interaction during training; whether explicit relational constraints further reduce interpenetration is untested.
• Absorbing better foundation models. Because the workflow showed gains when swapping HY3D-2.0 → HY3D-3.1 or MogeV2 → Depth Anything V3, an open question is how far the pipeline scales purely by upgrading its providers, and whether an RGB-conditioned aligner could ever beat the geometry-only design (the retrained 9-channel model underperformed here).

Target Audience

Researchers and engineers working on 3D content generation, single-image scene reconstruction, and point-cloud registration who already know the basics of generative 3D models and similarity-transform estimation. It is also useful for practitioners building modular asset pipelines who want evidence that swapping components pays off, and for benchmarking researchers interested in the paper's critique of geometry-only metrics.

Authors’ abstract

Single-image 3D scene construction is a fundamental problem in 3D vision. Recent methods such as SAM-3Dfy enable end-to-end reconstruction of 3D assets and their scene layouts, but still struggle with fine-grained scene geometry and asset quality. Meanwhile, specialized 3D object generators provide detailed object geometry, while 3D foundation models offer strong and generalizable spatial understanding. We propose Easy-3Dfy, a compositional workflow that decouples object generation from scene layout estimation, allowing specialized models to fully exploit their respective strengths. An RPF-based aligner then bridges these capabilities by establishing geometric correspondence between object-local meshes and partial scene pointmaps, recovering object layout for scene composition. Without requiring complete scene-level supervision, the proposed workflow consistently outperforms SAM-3Dfy across multiple geometric metrics on Scene-Maker and 3D-FUTURE200. Our results demonstrate that explicit geometric composition can effectively combine high-quality object generation with strong spatial understanding, while enabling the workflow to benefit independently from advances in both 3D object generation and foundation models.