ArtiLatent: Realistic Articulated 3D Object Generation via Structured Latents

AI summary

Overview

Research area: Computer vision and 3D generative modeling, specifically articulated 3D object synthesis (joint-based part motion for human-made objects).

Technical level: Advanced. The paper assumes familiarity with variational autoencoders, latent diffusion / flow matching, sparse voxel representations, and 3D Gaussian Splatting.

Scope: The paper introduces a diffusion-based framework that jointly generates geometry, part-level articulation parameters, and photorealistic appearance for human-made articulated 3D objects, conditioned on a single real-world image.

What This Paper Is About

Human-made objects such as drawers, doors, and appliance lids consist of parts joined by hinges or sliders with constrained motion. Generating such objects from scratch requires getting three tightly coupled things right at once: detailed geometry, physically valid joint behavior, and realistic texture. Existing methods either handle only rigid static objects, rely on coarse bounding boxes plus retrieval-based part assembly, or model geometry without texture. ArtiLatent addresses this by embedding geometry, part semantics, and articulation attributes into a single shared latent space and decoding that latent into animatable 3D Gaussians.

Key Contributions

1. A structured global articulation representation. Rather than modeling each part independently with a fixed part-count upper bound, the method represents an object as sparse voxels, where each active voxel carries occupancy, part semantic label, part bounding box, joint type, joint axis and origin, and motion range. All voxels belonging to one part share the same semantic and articulation attributes.

2. A unified articulation-aware VAE and latent diffusion model. A VAE with 3D convolutional blocks compresses the articulation-aware voxel tensor into a compact latent (from C_in = 35 input channels at 64×64×64 to C_z = 8 at 16×16×16). A transformer-based flow matching network (𝒢^Arti) then samples new latents under conditions such as image embeddings or category labels, with classifier-free guidance.

3. An articulation-aware Gaussian decoder with fine-tuning. Because a pretrained static-object decoder assigns poor textures to surfaces that are hidden when closed but revealed when opened (e.g., the inside of a drawer), the authors fine-tune the encoder, Gaussian decoder, and diffusion model using rendered images of the same object across multiple articulation states.

4. Image-conditioned generation of complete articulated objects. Given a single RGB image of an object in its rest state (near-frontal view), the framework produces a textured 3D Gaussian representation plus joint parameters, enabling motion by joint-driven transformation without re-running inference.

Main Findings

• Geometric and articulation accuracy on PartNet-Mobility: ArtiLatent reaches RS-d_CD of 0.0063 and AS-d_CD of 0.0043, the lowest of all articulation-aware baselines. SINGAPO scores 0.0168 and 0.0905, NAP-ICA scores 0.0173 and 0.0914, and URDFormer scores 0.5502 and 0.8374.

• Appearance fidelity on PartNet-Mobility: ArtiLatent achieves an FID of 137.18 versus 175.85 for SINGAPO. TRELLIS, which cannot model articulation, reports a CD of 0.0051 and an FID of 153.45.

• Zero-shot generalization on ACD: On 135 unseen ACD objects, ArtiLatent obtains RS-d_CD 0.0690, AS-d_CD 0.0751, and FID 128.34, against SINGAPO's 0.1011, 0.1679, and 201.60.

• Fine-tuning matters: The ablation shows that removing articulation-aware fine-tuning worsens results from RS-d_CD 0.0063 / AS-d_CD 0.0043 / FID 137.18 to 0.0076 / 0.0051 / 156.02, with visible texture artifacts in articulated regions.

• Attribute prediction quality: On PartNet-Mobility, the method reports occupancy recall of 98.94%, part type accuracy of 96.27%, joint type accuracy of 99.17%, joint axis error of 1.14°, joint origin error of 0.10, joint range (angle) error of 7.37°, and joint range (translation) error of 0.0159. Compared with SINGAPO, it is better on bbox center (0.0357 vs 0.0440), joint type, joint axis, joint origin, and joint range (translation), but worse on bbox size (0.0832 vs 0.0651), part type (96.27% vs 97.89%), and joint range (angle) (7.37° vs 6.73°).

• Spatial consistency of per-voxel predictions: Intra-part standard deviation is reported as low across bounding box center ([0.0211, 0.0159, 0.0277]), bounding box size ([0.0377, 0.0260, 0.0391]), joint axis ([0.1012, 0.0972, 0.0875]), joint origin ([0.0392, 0.0233, 0.0384]), joint range angle ([6.96°, 7.56°]), and joint range translation ([0.007, 0.0123]), supporting the averaging strategy used to enforce per-part coherence.

• Inference cost: Generation takes about 25.85 seconds per object on an NVIDIA A6000: 16.25 seconds for the articulation-aware voxel structure, 9.54 seconds for voxel-level appearance feature sampling, and 0.06 seconds for Gaussian decoding. SINGAPO takes about 2.9 seconds per object on the same hardware.

• Qualitative advantage over retrieval: In the washing machine example, SINGAPO fails to retrieve correct door geometry, while the generative approach keeps part and motion consistent.

Methodology in Plain English

The pipeline has two stages.

Stage one: generating the articulation-aware structure. Each object is converted into a 64×64×64 occupancy grid, and every occupied voxel is annotated with a part label (base, drawer, door, handle, knob, tray, shelf, wheel), a bounding box, a joint type (fixed, revolute, prismatic, continuous, screw), a joint axis and origin, and a motion range — 34 attribute channels plus one occupancy channel, giving 35 input channels. A VAE compresses this into an 8-channel 16×16×16 latent. Training combines a KL term (weighted at α_kl = 0.001), a Dice loss for occupancy to handle class imbalance, cross-entropy losses for part semantic labels and joint types, and an L2 regression loss over axes, origins, ranges, and bounding boxes. A transformer-based flow matching network then learns the distribution of these latents, conditioned on things like a DINOv2 image embedding or a category label, with classifier-free guidance at inference.

Stage two: generating appearance. The method builds on TRELLIS, sampling a latent feature per voxel and decoding it into 3D Gaussians (32 Gaussians per active voxel). Because TRELLIS's decoder was trained on static objects, it produces artifacts on surfaces that only become visible once a part moves. The fix is to supervise the encoder and decoder with images of the object rendered in multiple articulated poses. For each object, the authors uniformly sample k = 8 articulation states and render n = 48 views per state, extract DINOv2 features per view, average them per voxel across states and views, and assign the averaged feature to the rest-state voxel. During fine-tuning, decoded Gaussians are transformed by the joint parameters (translation or rotation) and rendered against ground-truth views. The diffusion model is then re-tuned on the updated latents.

Inference. After sampling and decoding a voxel structure, voxels are grouped into parts using predicted semantic labels, and a DBSCAN clustering step over bounding box attributes separates adjacent parts sharing a label. Articulation parameters are averaged within each segment and reassigned to all its voxels, producing coherent part-level motion.

Training setup. The articulation VAE and the diffusion model were each trained from scratch for 1 day on 4×A6000 GPUs; fine-tuning used the same hardware over 2 days. Adam with a learning rate of 1×10⁻⁴ and a batch size of 4 per GPU. At inference, CFG strength is 3 and sampling uses 50 steps. The diffusion model is initialized from the TRELLIS structure diffusion model.

Data and evaluation. Training uses 3,063 articulated objects from a PartNet-Mobility subset spanning Storage, Table, Refrigerator, Dishwasher, Oven, Washer, and Microwave, preprocessed following SINGAPO. Evaluation uses 77 held-out instances with two randomly rendered views, plus 135 unseen ACD objects for zero-shot testing. Evaluation renders two views in the rest state, randomly selects one as input, generates a single rest-state 3D Gaussian, and then applies joint transformations to simulate five target articulation states; all metrics are computed over those five outputs.

Why This Matters

Research impact. The paper argues that modeling geometry, semantics, and articulation jointly in one latent space captures correlations that independent part modeling misses — for example, that drawers tend to translate while doors rotate. It also identifies a specific failure mode in inheriting a static-object appearance decoder (poor supervision on occluded-then-revealed surfaces) and offers a general fine-tuning remedy. Its results across in-distribution and zero-shot ACD benchmarks give a reference point for controllable articulated object generation with appearance included, a combination earlier work such as NAP, CAGE, and MeshArt did not provide.

Real-world applications:

• Populating physics simulators and robotics training environments with interactable furniture and appliances.
• Building digital twins of household or industrial equipment that can be opened, closed, and manipulated.
• Creating assets for immersive virtual environments and games without manual rigging.
• Generating interactive assets from a single photograph for design, e-commerce, or AR previews.

Industry relevance. Companies working on embodied AI, simulation platforms, AR/VR content pipelines, and digital-twin software all need articulated assets with correct joints and believable interiors. The paper's reported ~25.85-second generation time and its reliance on accurate part-level bounding boxes and voxel-level semantic labels (per the truncated limitations section) indicate where the practical constraints currently sit; SINGAPO's ~2.9-second runtime is notably faster despite its retrieval-based approach.

Future Directions

• Scale and generalization. The authors state they evaluate on objects with relatively simple kinematic structures and acknowledge that generalization could improve with a larger dataset.
• Reducing dependence on supervision. The limitations discussion notes reliance on accurate part-level bounding boxes and voxel-level semantic labels (the sentence is truncated in the provided content, so the full dependency is not reported).
• More complex kinematics and categories. Extending beyond furniture-like PartNet-Mobility categories and the ACD dataset to richer joint structures and unseen object types is an open question.
• Narrowing the runtime gap. Reducing the gap between the reported ~25.85 seconds per object and SINGAPO's ~2.9 seconds would make the approach more practical for interactive or batch pipelines.

Target Audience

Researchers and graduate students in 3D generative modeling, computer graphics, and computer vision who work on latent diffusion, structured 3D representations, or 3D Gaussian Splatting. It is also relevant to practitioners in robotics simulation, embodied AI, and digital twin or AR/VR asset pipelines who need articulated objects with plausible interiors and controllable joints. Readers without a background in VAEs, flow matching, and Gaussian splatting will find the method section demanding.

Authors’ abstract

We propose ArtiLatent, a generative framework that synthesizes human-made 3D objects with fine-grained geometry, accurate articulation, and realistic appearance. Our approach jointly models part geometry and articulation dynamics by embedding sparse voxel representations and associated articulation properties, including joint type, axis, origin, range, and part category, into a unified latent space via a variational autoencoder. A latent diffusion model is then trained over this space to enable diverse yet physically plausible sampling. To reconstruct photorealistic 3D shapes, we introduce an articulation-aware Gaussian decoder that accounts for articulation-dependent visibility changes (e.g., revealing the interior of a drawer when opened). By conditioning appearance decoding on articulation state, our method assigns plausible texture features to regions that are typically occluded in static poses, significantly improving visual realism across articulation configurations. Extensive experiments on furniture-like objects from PartNet-Mobility and ACD datasets demonstrate that ArtiLatent outperforms existing approaches in geometric consistency and appearance fidelity. Our framework provides a scalable solution for articulated 3D object synthesis and manipulation.