WorldSculpt: Generating Compositional Worlds from Grounded Videos

AI summary

Overview

• Research area: Computer vision, specifically 3D generative modeling and compositional scene reconstruction from multi-view observations of cluttered real-world scenes.
• Technical level: Advanced. The paper assumes familiarity with native-3D generative priors, flow matching, structured latents, LoRA adaptation, and multi-view geometry.
• Scope: The paper proposes WorldSculpt, a framework that adapts a single-object 3D generative prior (Pixal3D) with a multi-view conditioning pathway to produce scenes as collections of individually addressable object meshes, and introduces UE-MeshyScene, a photorealistic benchmark of densely cluttered scenes.

What This Paper Is About

Generative world models such as Marble (World Labs, 2025) and HY-World 2.0 (Team et al., 2026) can synthesize explorable 3D environments, but their output is a single unified representation — a fused mesh or a set of Gaussians — so individual objects like a chair or a vase cannot be selected, moved, or transferred into a physics simulator. Geometry-centric reconstruction methods also typically recover the scene as one fused representation and leave missing geometry in occluded regions, while existing compositional generative approaches have mostly been demonstrated on relatively simple scenes such as a small collection of objects on a tabletop. WorldSculpt's goal is to generate a compositional 3D representation of a densely cluttered scene containing hundreds of objects, where each object is an individual mesh placed in a shared world frame.

Key Contributions

1. The WorldSculpt framework. Given multi-view scene observations, it maps them into an anchor-aligned canonical frame per object, conditions a strong object-level 3D generative prior on the aligned observations, and places the generated meshes into a shared world frame through canonical-to-world transformations — producing a scene of hundreds of individually addressable object meshes rather than a fused representation.
2. Demonstration that no scene-level training is needed. A generative prior finetuned entirely on individual objects in canonical space generalizes at test time to large-scale scenes containing hundreds of densely occluded objects, through multi-view conditioning and a targeted augmentation curriculum.
3. The UE-MeshyScene benchmark. A photorealistic benchmark of densely cluttered indoor and outdoor scenes containing hundreds of objects, author- and rendered in Unreal Engine, with exact per-object annotations and ground-truth meshes for evaluating compositional 3D generation under complex occlusion.
4. Broader applicability. A demonstration of converting generated 3DGS worlds, such as Marble and HY-World 2.0, into compositional mesh scenes.

Main Findings

• Consistent superiority across settings: Across single-object, controlled multi-object, and UE-MeshyScene evaluations, the method consistently outperforms prior approaches, with larger gains as scene complexity and occlusion increase.
• Single-object behavior on Toys4k: With 1 input view and 0% occlusion, the method achieves CD-ℓ₂ of 2.59, CD-ℓ₁ of 3.85, EMD of 7.8, F@.02 of 0.739, F@.05 of 0.932, against Pixal3D's 2.17 / 3.74 / 7.8 / 0.749 / 0.934 and TRELLIS.2's 7.55 / 6.35 / 10.7 / 0.587 / 0.839. Performance improves monotonically with more views: at 16 views and 0% occlusion the method reaches CD-ℓ₂ 1.19, CD-ℓ₁ 2.89, EMD 6.9, F@.02 0.827, F@.05 0.967.
• Robustness to heavy occlusion: At 75% per-view occlusion with 16 views, the method reports CD-ℓ₂ 2.25, CD-ℓ₁ 3.97, EMD 8.0, F@.02 0.713, F@.05 0.929, whereas the single-view priors degrade sharply — Pixal3D at 1 view and 75% occlusion reports CD-ℓ₂ 63.95 and TRELLIS.2 reports 56.71.
• Real-world transfer on HouseCat6D: The method reports CD-ℓ₂ 0.28, CD-ℓ₁ 1.53, EMD 3.4, F-Score 0.995. The strongest listed prior, ShapeR, reports 1.26 / 1.88 / 4.8 / 0.973; other listed methods range from RecGen (7.22 / 6.98 / 8.6 / 0.805) to SceneGen (31.4 / 13.08 / 15.9 / 0.626).
• Cluttered synthetic scenes on Toys4k-Scene: The method reports CD-ℓ₂ 0.61, CD-ℓ₁ 1.78, EMD 3.7, F-Score 0.981, versus ShapeR at 8.38 / 7.81 / 9.4 / 0.746 and SAM3D at 52.7 / 17.45 / 17.5 / 0.457.
• Large-scale scenes on UE-MeshyScene: Aggregated over the six scenes, the method reports CD-ℓ₂ 2.48 (per-instance median 0.25), CD-ℓ₁ 2.66 (median 1.69), EMD 5.0, and F-Score 0.951 at τ = 0.05, compared with ShapeR at 7.42 (median 2.47), 6.37 (median 5.07), 8.8, and 0.813.
• UE-MeshyScene composition: Six environments — an aircraft hangar, an abandoned-city interior, an old cathedral, an architectural-visualization office, a Japanese school, and a desert town — contain 93 to 701 objects each (2,299 objects total), rendered at 2560 × 1440 with 265 to 1758 views per scene (5,964 views total). Per-scene object counts are 171 (Hangar), 93 (Abandoned City), 145 (Cathedral), 678 (Office), 511 (Japanese School), and 701 (Desert Town).
• Qualitative advantages: On Toys4k-Scene the method produces more detailed and accurately registered object geometry closely matching ground truth, while on HouseCat6D it recovers detailed object geometry accurately placed in the world frame and closely matching the scanned ground truth.

Methodology in Plain English

The pipeline assumes it is already given posed images with known intrinsics and camera-to-world extrinsics, per-view instance masks for each object, and a coarse world-space localization box per object. The authors state that recovering these masks, camera parameters, and coarse object localizations is well studied and lies outside the scope of the work.

For each object, the method builds an "anchor-aligned virtual canonical cube." The coarse localization box is not used directly as the generation volume because its side lengths are generally unequal while the generative model works in a normalized cubic domain, and its orientation need not match the camera-relative canonical orientation the pretrained prior expects. Instead, one view is chosen as the anchor — at inference the view in which the object is most fully observed, and during training a randomly sampled view — and this anchor determines the orientation of the canonical frame. The canonical-to-world transformation is a similarity transformation built from the box center, an isotropic scale, and a rotation induced by the anchor camera, which preserves the proportions of the generated object. The scale is allowed to increase (keeping center and rotation fixed) until the cube's projection covers the object masks in all selected views, so the object is not clipped by the per-view crops.

Each selected view's image is projected, cropped to the cube's extent, masked with the instance mask, and resized to the model's input resolution. Because cropping and resizing change the image coordinate system, the camera intrinsics are adjusted accordingly, and a projection function maps any canonical voxel into the resized crop, accounting for the object's position, orientation, scale, and off-center image location.

The per-object generator is instantiated from Pixal3D, a native-3D generative model built on the TRELLIS.2 structured-latent backbone that generates objects through a cascade of flow-matching stages. Only the first two stages are used — a sparse structure stage predicting coarse occupancy on a 64³ grid, and a shape stage generating a high-resolution sparse structured latent over the occupied voxels which is then decoded into a mesh. Texture and material are left to future work.

To condition on multiple views, each canonical observation is encoded with DINOv3 into a dense feature map, and each feature map is lifted into a shared canonical voxel volume by sampling at the voxel's projected image location, so observations of the same canonical location are aligned before aggregation. An IBRNet-style aggregator that is permutation-invariant and accepts a variable number of views then fuses the features: it computes the cross-view mean and variance at each voxel, refines each feature and predicts an aggregation logit with two lightweight MLPs, and outputs the mean plus a softmax-weighted residual. The final layer of the feature MLP is initialized to zero so the aggregator begins as an exact cross-view mean and gradually learns view-dependent residual corrections. Global image tokens for the original Pixal3D cross-attention interface are extracted only from the anchor observation, preserving the single-image conditioning convention.

The aggregated 3D condition is injected into both geometry stages at each transformer block by aligning it with current 3D token locations, projecting it to the block feature dimension, and adding it to the block features. The original Pixal3D parameters stay frozen, with LoRA adapters adapting attention and per-block projection layers; the multi-view aggregator and zero-initialized conditioning projections are trained in full.

Training uses random-view sampling: a variable number of conditioning views is sampled per object and one is randomly chosen as anchor, so the anchor defines the camera-relative canonical orientation for that example. The sparse-structure and shape stages are trained independently on their respective ground-truth latent targets, and only the multi-view aggregator, conditioning injection layers, and LoRA parameters are optimized.

A conditioning-view augmentation curriculum closes the gap between clean training renders and the partial, degraded observations in cluttered scenes. Four degradation types are sampled independently — occlusion simulated with random 2D masks or 3D-consistent occluders, perturbation of non-anchor camera poses (the anchor pose is unchanged), mask errors from perturbed and degraded segmentation boundaries, and downsampling to simulate objects occupying only a small image region. Augmentations are applied only to inputs, never to the supervision target, and each degradation's strength is gradually increased from zero to its maximum during training; the curriculum reaches maximum strength over the first 3k training iterations.

At inference, each object is processed independently: the anchor is chosen as the view where the object is most fully observed, the cube is expanded if needed, a bounded number of conditioning views is selected with priority to observations with larger visible object regions, and the two Pixal3D geometry stages run with multi-view conditioning. The resulting canonical mesh is placed into the scene by the canonical-to-world transformation, and the final scene is the collection of these world-space meshes — no cross-object fusion or joint shape optimization is performed.

Implementation: the two Pixal3D geometry stages are initialized from released checkpoints and finetuned independently on the TexVerse dataset. Conditioning images are 512 × 512 px for the sparse-structure stage and 1024 × 1024 px for the shape stage. LoRA uses rank r = 32 and scaling factor α = 32, applied to DiT attention and per-block projection layers. Each stage trains for 15k iterations in bf16 with AdamW, learning rate 10⁻⁴, betas (0.9, 0.95), no weight decay, and an exponential moving average of 0.9999, with batch size 48 for sparse-structure and 12 for shape. Each iteration samples 1 to 20 views for the conditioning set. Gradient checkpointing is applied to half the shape stage's transformer blocks, and each conditioning view is capped at 32,768 tokens.

Why This Matters

The paper targets a structural mismatch between what generative world models produce and what downstream applications consume. As the authors put it, these applications "do not consume an undifferentiated soup of surface geometry" — they require a scene decomposed into constituent objects, each a mesh that can be independently selected, moved, and re-simulated. WorldSculpt also shows that a prior finetuned entirely on individual objects in canonical space can generalize to scenes with hundreds of densely occluded objects without any scene-level training, which is a scalability claim relevant to anyone building on object-level 3D priors.

Real-world applications named in the paper:

• Gaming and content creation, where individually selectable and movable object assets are needed.
• AR/VR, where scenes must be represented as separate entities rather than fused surfaces.
• Simulation, including physics simulators that require objects as independent entities.
• Robotics, where cluttered environments and object-level geometry matter.

Industry relevance: the framework is positioned as a bridge from monolithic world-model outputs to asset-level pipelines, and the authors demonstrate converting generated 3DGS worlds such as Marble and HY-World 2.0 into compositional mesh scenes. UE-MeshyScene supplies exact geometric ground truth that the authors note is difficult to obtain from real-world captures, giving the field a way to measure progress on genuinely cluttered scenes rather than sparse tabletop arrangements.

Future Directions

• Texture and materials. The paper uses only the first two Pixal3D geometry stages and explicitly leaves the subsequent texture and material stage to future work, so generated objects currently lack appearance.
• Extending the evaluated settings. The authors position UE-MeshyScene and the Toys4k/HouseCat6D/Toys4k-Scene evaluations as covering single-object, controlled multi-object, and large-scale compositional generation, leaving room for broader real-world scene evaluation where per-object mesh ground truth is hard to obtain.
• Reducing reliance on given inputs. The method assumes known intrinsics, poses, per-view instance masks, and coarse world-space localizations; the authors state that recovering these lies outside the scope of this work, suggesting integration with segmentation, pointmap, and 3D detection pipelines as a natural extension.
• Using the released annotations. UE-MeshyScene provides camera poses, per-object instance masks, per-object 3D bounding boxes, and metric depth maps; the authors state that depth is not used by the method but release the full annotation set to support related tasks.

Target Audience

Researchers and engineers working on 3D generative models, compositional scene generation, and amodal or occlusion-aware reconstruction. It is also relevant to practitioners in graphics, simulation, AR/VR, gaming, and robotics who need scene representations decomposed into individually manipulable object meshes, and to benchmark builders interested in photorealistic, densely cluttered 3D datasets with exact per-object ground truth. A strong background in multi-view geometry and diffusion or flow-matching generative models is needed to follow the method sections.

Authors’ abstract

We study the problem of generating a compositional 3D representation of a cluttered scene containing hundreds of objects. The goal is to represent the scene as a collection of individual object meshes placed in a shared world frame, as required by downstream applications such as gaming, AR/VR, simulation, and robotics. This task is challenging in densely cluttered scenes, where objects heavily occlude one another and each view reveals only a fraction of their geometry. Geometry-based approaches typically reconstruct the scene as a single representation and leave incomplete geometry in occluded regions, while existing compositional methods with generative priors are largely limited to relatively simple scenes. We show that complex scenes with hundreds of objects can instead be generated compositionally by adapting a strong single-object 3D generative prior to multi-view observations. We instantiate this paradigm with Pixal3D, extending it with a multi-view conditioning pathway that grounds object generation in multiple posed observations. Although the model is finetuned entirely on single objects in canonical space, it generalizes to large scenes with severe occlusion without any scene-level training, demonstrating the feasibility and scalability of this paradigm. We further introduce UE-MeshyScene, a photorealistic benchmark of densely cluttered scenes with hundreds of objects, per-object annotations, and ground-truth meshes. Across single-object, controlled multi-object, and UE-MeshyScene evaluations, our method consistently outperforms prior approaches, with larger gains as scene complexity and occlusion increase. Finally, we demonstrate broader applicability by converting generated 3DGS worlds, such as Marble and HY-World 2.0, into compositional mesh scenes.