CoLa3D: Composable Latent 3D Decomposition

AI summary

Overview

Research area: Computer Vision — 3D scene generation, single-image 3D reconstruction, and generative 3D latent representations.

Technical level: Advanced. The paper assumes familiarity with sparse voxel/structured 3D latents, transformer attention blocks, Hungarian matching, and standard 3D reconstruction metrics such as Chamfer Distance and F-Score.

Scope in one sentence: CoLa3D is a feed-forward transformer that takes one whole-scene 3D latent (from a single image or an existing mesh) and splits it into a set of object latents that decode in parallel into spatially aligned object meshes.

What This Paper Is About

Recovering a usable 3D scene from a single image usually means detecting objects in 2D, generating each object separately, and then gluing the pieces back together — which produces scenes where objects float, overlap, or sit at the wrong scale. CoLa3D instead performs the split inside a single shared 3D scene latent, so every object latent is expressed in the same scene coordinate frame as the whole scene. The goal is a set of individual object meshes that reassemble into a coherent, physically plausible scene that downstream systems can edit, rearrange, or simulate.

Key Contributions

1. A new decomposition paradigm. Instead of per-object generation in canonical spaces followed by fragile pose and scale estimation, CoLa3D extracts object latents from one whole-scene latent in a shared scene coordinate frame, then decodes them in parallel into per-object meshes plus a recomposed scene mesh.
2. A lightweight query-based decomposition transformer. It performs latent-space object extraction in a single forward pass and accepts three query types — 2D masks, 3D points, and purely learnable queries — with 3D semantic features from Utonia and 2D semantic features from DINOv3 added for generalization.
3. State-of-the-art results on synthetic and real data. The paper reports gains over MIDI, SceneGen, PartCrafter, SAM 3D, 3D-Fixer, Gen3DSR, and a Point-SAM variant on object asset quality and global scene consistency, while running roughly 2× faster and using less than 1% of SAM 3D's training data.
4. A scalability study. Retraining on 3D-FRONT combined with ARSG-110K (100k retained scenes, 870k images, up to 20 objects per scene) with K_max = 20, plus a new MetaScenes-hard benchmark of 50 scenes with 9–20 visible objects each.

Main Findings

• Synthetic scene-level gains: On 3D-FRONT, CoLa3D (Free) reduces scene Chamfer Distance by 27.9% and improves object F-Score by 8.8% over the best baseline, reaching scene CD 0.0075 and object F-Score 0.8252. On MetaScenes, CoLa3D (Point) reduces scene CD by 57.1% and improves object F-Score by 26.9%, reaching scene CD 0.0060 and object F-Score 0.7996.
• Speed: CoLa3D variants run in about 16.6 seconds per scene on a single NVIDIA L40S 48 GB GPU, versus 32.15 s for SAM 3D, 49.30 s for SceneGen, 51.04 s for PartCrafter, 67.45 s for MIDI, 98.35 s for 3D-Fixer, and 229.40 s for Gen3DSR.
• Zero-shot real-world transfer: On Replica, CoLa3D (Point) reduces one-side scene CD by 53.6% and improves object Recall by 20.9% over the best baseline; on ScanNet++, CoLa3D (Mask) reduces scene CD by 20.8% and CoLa3D (Free) improves object Recall by 4.1%.
• Denser scenes: On MetaScenes-hard, CoLa3D (Free) reduces scene CD by 17.0% over the strongest baseline (SAM 3D, 0.0112 → 0.0093) and raises scene IoU from 0.6339 to 0.7229; CoLa3D (Point) is best at the object level, reducing CD by 7.8% over SceneGen and improving F-Score by 12.6% over SAM 3D.
• Visible geometry is preserved: On 512×512 rendered depth maps from 3D-FRONT, CoLa3D reduces AbsRel by 14.9% and RMSE by 10.3% over the best 3D baseline, while achieving the highest completion score (0.9175–0.9337 across variants). The monocular depth estimator MoGe2, shown only as a reference and excluded from ranking, is stronger on depth (AbsRel 0.0423 vs. CoLa3D (Mask) 0.0797) but produces no decomposable meshes.
• Semantic features matter most: Removing Utonia features increases object CD by 54.7% over the learnable-query baseline (0.0318 → 0.0492); removing DINOv3 features drops object IoU by 0.0197 (0.6912 → 0.6715).
• Loss design matters: Removing the query diversity loss drops object IoU by 3.1%, and replacing multi-hot mask supervision with single-hot labels reduces object IoU by 5.0%.
• Query type trades off: Mask queries give the best scene-level result in the ablation (scene CD reduced by 6.9%, to 0.0067), while point queries give the best object-level result (object CD reduced by 15.1%, to 0.0270).
• Fewer training data: The paper states CoLa3D uses less than 1% of SAM 3D's training data, and separately that SAM 3D is trained on 100× more data.
• Qualitative artifacts: MIDI and Gen3DSR generate wrong shapes under heavy occlusion, while SceneGen and SAM 3D produce more accurate shapes but show spatial inconsistency and interpenetration, especially for objects in close contact; CoLa3D is reported to avoid these artifacts.

Methodology in Plain English

The model reuses an existing image-to-3D generator, TRELLIS.2, as its backbone. That system compresses a mesh into a compact set of features attached to active cells in a sparse 3D grid, and can decode such a latent back into a mesh. CoLa3D first obtains a geometry latent for the whole scene — either by encoding a supplied scene mesh or by generating one from a single image — and then adapts the pipeline for scene-level rather than object-level generation.

That geometry latent is augmented with semantic features. A mesh is passed through Utonia to get 3D semantic features, which are projected to match the geometry feature dimension and simply added. DINOv3 supplies 2D image features, which the transformer also attends to.

The core module is a small transformer, six attention blocks deep with a hidden dimension of 768. A small set of queries — 2D object masks encoded with a CNN plus positional encoding of the mask centre, 3D points encoded with positional encoding plus the latent features at that point, or eight learnable slots with no explicit prior — self-attend, then attend to the frozen DINOv3 features, then alternate between reading from and writing to the scene latent. 3D RoPE is applied in the query self-attention and query–latent cross-attention to preserve relative 3D position.

Each refined query is compared to each refined latent cell with a temperature-scaled cosine similarity passed through a sigmoid, producing a soft assignment matrix. Cells scoring above 0.5 belong to that object, and an argmax fallback assigns any unclaimed cell. Because the latent grid is coarse, a cell can be assigned to several queries, which the authors argue suits neighbouring objects that share contact regions. Each object's latent is then decoded by the shared decoder in parallel, and the meshes come out already in the same coordinate frame, so no pose or scale fitting is needed.

Training supervises the assignment masks in latent space, using BCE plus Dice loss after Hungarian matching for learnable queries (mask and point queries use their known prompt-to-object correspondence). A diversity regularizer penalizes pairwise cosine similarity between queries to stop several queries collapsing onto the same object. Training used about 13k scenes and 82k images from 3D-FRONT (combining the MIDI and InstPIFU splits), 60k steps, Adam at a learning rate of 1e-4, per-GPU batch size 16, loss weights λ_bce = 1.0, λ_dice = 1.0 and λ_reg = 0.1, taking about 2 days on 4 NVIDIA H100 GPUs.

Why This Matters

The paper argues that the dominant per-object recipe — segment in 2D, generate each object in a canonical space, then estimate pose and scale to reassemble — is structurally prone to spatial inconsistency and interpenetration. Moving the decomposition into a shared latent space avoids that glue step entirely, and the reported result is meshes that are already aligned to the scene and therefore directly loadable into physical simulators.

Real-world applications:

• Embodied agents and robotics, which need object-level structure and relationships, not just a fused mesh, to plan interaction and manipulation.
• AR/VR content authoring, where scenes must be editable, rearrangeable and interactive rather than a single untextured blob.
• Physical simulation and synthetic data generation, using the decomposed assets for rearrangement and dynamics experiments.
• Real-to-simulation pipelines built from single photographs or existing scanned/artist-created meshes.

Industry relevance: The method runs in roughly 16 seconds per scene on a single GPU and trains on a small fraction of the data used by SAM 3D, which matters for deployment cost. It also drops into an existing foundation 3D generator; the authors note the paradigm should transfer to other backbones such as Hunyuan3D.

Future Directions

• Backbone independence: The paper asserts the latent decomposition paradigm is not specific to TRELLIS.2 and should apply to other foundational 3D generators such as Hunyuan3D, but this is stated as a claim rather than evaluated.
• Object count limits: Learnable queries default to K_max = 8 and the scalability retraining uses K_max = 20; how the approach behaves on scenes with far more objects, and whether query capacity or matching becomes the bottleneck, is left open.
• Depth fidelity versus decomposition: CoLa3D's visible-region depth trails the MoGe2 reference (AbsRel 0.0797 vs. 0.0423), raising the question of whether stronger monocular depth priors can be folded into the shared latent without losing decomposability.
• Interpenetration and physical plausibility: The paper motivates its approach with physical plausibility and contact regions, but the evaluation uses CD, F-Score, IoU and Recall — quantitative physical-plausibility or contact metrics are not reported and remain a natural next step.

Target Audience

Researchers and graduate students working on 3D generative models, single-image 3D reconstruction, and scene-level 3D understanding; engineers building real-to-simulation, AR/VR, or robotics pipelines who need editable object-level assets rather than monolithic meshes; and practitioners familiar with TRELLIS-style latent 3D representations who want to see how far a lightweight transformer on a frozen latent can go. Readers without background in 3D generative modelling will find the metrics and architectural details demanding.

Authors’ abstract

We introduce CoLa3D, a simple and effective model for composable 3D scene decomposition that, given a single whole-scene latent from either a single image or an existing scene mesh as input, directly outputs a set of object meshes with consistent spatial layout that can be assembled back into a coherent scene. CoLa3D introduces a novel paradigm for 3D scene decomposition within a shared whole-scene latent, unlike alternatives that perform per-object generation in canonical spaces and then glue the objects back together with fragile pose and scale estimation. The network is a lightweight, flexible, and scalable query-based transformer that accepts various types of queries, including 2D masks, 3D points, and learnable queries, and learns to decompose the scene into objects in a compact latent space. On both synthetic and real-world datasets, CoLa3D achieves state-of-the-art performance on composable 3D scene decomposition, substantially improving global scene-level consistency and object-level quality while running 2 times faster and using less than 1% of SAM 3D's training data. Visible depth on par with MoGe2 further confirms that latent decomposition preserves accurate visible scene geometry while producing physically complete, decomposable object meshes.