Skip to content
AI.info

Research

Direct3D-V3: Latent-Free Diffusion Transformers in the Native Voxel Domain

Most recent 3D generation systems are two-stage: a VAE first compresses a shape into a compact latent code (a vecset, triplane, or sparse voxel latent), and a diffusion model then generates inside…

arXiv
2611.26511
Published
2026-11-19
Authors
Shuang Wu, Jiachen Qian, Youtian Lin, Feihu Zhang, Philip Torr, Yao Yao, Xun Cao

AI summary

Overview

  • Research area: Computer Vision / 3D generative modeling — specifically image-to-3D shape generation with diffusion transformers.
  • Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching, transformer architectures (DiT, AdaLN), signed distance fields (SDF), Marching Cubes mesh extraction, and latent diffusion pipelines.
  • Scope: The paper proposes and describes Direct3D-V3, a framework that trains and samples a diffusion transformer directly on raw SDF voxel grids at resolutions up to 1536³, removing the variational autoencoder (VAE) used by latent-based 3D generation methods.

Note on completeness: the supplied paper content is truncated before the experiments section. The specific benchmark numbers, dataset sizes, baseline comparison tables, and ablation values are therefore not reported in the material available, and none are stated below. Only the qualitative claims made in the abstract and introduction are summarized.

What This Paper Is About

Most recent 3D generation systems are two-stage: a VAE first compresses a shape into a compact latent code (a vecset, triplane, or sparse voxel latent), and a diffusion model then generates inside that latent space. The authors argue this design has two built-in costs: the VAE throws away fine geometric detail before the generator ever sees the data, and the VAE is trained for reconstruction while the diffusion model is trained for distribution matching, so the two stages optimize different objectives. Direct3D-V3 asks whether the compression step can be removed entirely, and instead models raw SDF voxel grids directly during both training and sampling.

Key Contributions

  1. A latent-free DiT for native voxel-domain 3D generation. The framework trains and samples directly on raw SDF voxel grids with no VAE, and reaches a resolution of 1536³ through a dual-granularity architecture that splits work between a patch-level DiT and a lightweight voxel-level DiT.
  2. Three complementary signals added to the native voxel formulation. Implicit manifold regularization (IMR) enforces locally consistent SDF transitions between neighboring voxels to suppress non-watertight surfaces; rendered normal supervision is applied by extracting meshes with Marching Cubes directly from the predicted SDF; and an auxiliary camera-prediction branch aligns the generated geometry with the conditioning image.
  3. Extension to generative reconstruction. Because the model outputs an SDF grid and predicts a camera, the generated geometry can be refined at test time against normals estimated from the input view, interleaved with diffusion sampling, without repeatedly passing through a heavy VAE decoder.
  4. Qualitative comparison against latent-based state of the art. The authors report that Direct3D-V3 is competitive with latent-based methods while producing meshes with finer geometric detail and better alignment to the input image (the underlying quantitative results appear after the truncation point and are not available in the supplied content).

Main Findings

  • The VAE is presented as a bottleneck, not a convenience. The paper's central claim is that VAE reconstruction removes fine geometric detail before the diffusion model sees the data, and that the two-stage pipeline creates an objective mismatch because the VAE is trained for reconstruction while the diffusion model is trained for distribution matching.
  • Dual granularity makes native SDF diffusion tractable. Operating directly on dense voxel grids is computationally prohibitive for DiT attention. Splitting the problem into a patch-level DiT (coarse semantics and global layout over sparse surface patches) and a voxel-level DiT (fine detail inside each patch) is what allows the method to reach 1536³.
  • Removing the VAE also removes an implicit geometric prior. In latent pipelines the VAE decoder implicitly learns plausible local SDF structure and tends to suppress holes. The authors observe that a latent-free DiT can instead produce locally inconsistent fields, so they add implicit manifold regularization; they report this "substantially reducing non-watertight artifacts in practice."
  • Eikonal regularization was rejected in favor of a discrete Lipschitz constraint. The authors state that Eikonal losses (unit-gradient-norm constraints at arbitrary points) can conflict with ground-truth TSDF supervision, making optimization unstable and in severe cases causing training collapse. Their alternative penalizes only neighbor pairs whose SDF values differ by more than one voxel length.
  • Native normal supervision scales where latent-based normal supervision does not. The paper contrasts its approach with Hi3DGen, which relies on VAE decoding for normal rendering; that memory overhead is described as limiting Hi3DGen to 256³, whereas Direct3D-V3 applies the same style of supervision at resolutions as high as 1536³.
  • Early latent-free 3D work was resolution-limited. The paper cites SDF-Diffusion as a prior VAE-free 3D approach whose generation is limited to 128³, restricting geometric precision.
  • Camera prediction is used as an auxiliary learning signal. Predicting the conditioning image's camera parameters is described as encouraging the DiT to learn the mapping from image observations to 3D shape, improving image–geometry alignment.
  • Quantitative results: not reported in the supplied content (the paper truncates before the experiments section).

Methodology in Plain English

The model represents a 3D shape as a uniform grid of signed distance values — each grid cell stores how far it is from the object's surface. Rather than compressing that grid into a learned code, the diffusion transformer denoises the grid itself, predicting the clean distance field directly from a noisy version at each step.

To keep this affordable, the grid is split into small cubic patches, and only patches near the surface are kept — which requires a cheap first stage that produces a coarse mesh and identifies the occupied patches. Patching alone still leaves the model with very high-dimensional tokens at high resolution, so the work is split in two: one transformer handles whole patches to get global shape and semantics right, and a second, lighter transformer refines the individual distance values inside each patch, guided at every layer by the patch-level features.

Because the surface is defined as the zero crossing of the distance field, small errors in predicted values can create inconsistent crossings and meshes with holes. The authors add a local smoothness penalty that fires only when neighboring voxels differ by more than one voxel's physical length, which keeps the field coherent without over-constraining valid variation.

The fact that the model outputs a distance grid — not a latent code — lets the researchers convert predictions into a mesh on the fly with Marching Cubes, render its normals from random camera angles using Nvdiffrast, and compare those normals to normals rendered from the ground-truth mesh. Gradients flow back to the predicted voxel values. A separate small branch predicts the camera of the input image (azimuth, elevation, log distance, log field of view, and two principal-point offsets) from the patch features and DINOv2-Large image features, which nudges the generated shape into agreement with the input view.

For reconstruction from a single image, the same setup is reused at test time: a pretrained normal predictor estimates the input image's normal map, a mask predictor estimates the foreground, the predicted camera is corrected by aligning rendered and estimated masks, and the SDF grid is updated so its rendered normals match the estimated normals. This correction is interleaved with denoising steps, so observations fix the visible parts while the generative prior fills in what was never seen.

Why This Matters

  • Impact on research: The paper challenges a premise that unifies otherwise very different 3D generation methods — vecset, triplane, and sparse-voxel latents all assume geometry must be compressed before it is generated. If latent-free generation is competitive at 1536³ with a deliberately minimal SDF representation, that undercuts the case for ever more elaborate VAE and latent designs, and shifts attention to how native signal-domain supervision can be applied instead.
  • Real-world applications (as framed in the paper's introduction):
    • Game and film asset creation from reference images.
    • AR/VR content authoring.
    • Robotic simulation environments that need clean, watertight object meshes.
    • Any image-to-3D workflow where fine surface detail and fidelity to the observed view matter more than raw throughput.
  • Industry relevance: Removing the VAE removes a whole separately trained, separately tuned component from the pipeline, and it removes the memory cost of a decoder at high resolution. The paper explicitly contrasts itself with the latent-based Hi3DGen, which must decode through a VAE before rendering normals and is consequently capped at 256³. A pipeline that supervises geometry directly in its evaluation space is also simpler to extend to test-time refinement, which is exactly what the generative-reconstruction application demonstrates.

Future Directions

  • Other volumetric representations. The authors state the latent-free formulation is in principle compatible with FlexiCubes, Sparc3D, and O-Voxel, but deliberately use the minimal SDF grid to test the core hypothesis. Whether the same recipe holds for those richer representations is left open.
  • Higher resolutions. 1536³ is presented as the current reachable scale; the dual-granularity design and the sparse-patch scheme are the stated enablers, which raises the question of how far the approach extends before attention or memory becomes binding again.
  • The first stage is explicitly out of scope. Surface patch localization comes from an external coarse stage, and the paper says this model "is not a focus of this paper" and that the method is agnostic to how the patch set is produced. Jointly optimizing or improving that stage is an obvious next step.
  • Quantitative grounding of the qualitative claims. The paper claims competitive results and finer geometry against latent-based state of the art; the specific benchmarks, datasets, and ablations that would let a reader check those claims fall outside the supplied content and are not reported here.

Target Audience

Researchers and practitioners in 3D generative modeling who are already comfortable with diffusion transformers and implicit surface representations — particularly those working on latent representations for 3D, high-resolution shape synthesis, or test-time reconstruction with generative priors. It is also relevant to graphics engineers evaluating whether a VAE-based pipeline can be dropped in favor of direct voxel-domain generation. Readers without background in flow matching, AdaLN-conditioned transformers, or SDF/Marching Cubes conventions will find the method sections dense.

Authors’ abstract

Native 3D generation has largely been built around latent diffusion, with recent advances concentrating on stronger variational autoencoders (VAE) and increasingly elaborate latent representations, from vecset and triplanes to structured sparse voxels. Despite their architectural differences, these methods all share the same premise: geometry is first compressed into a latent space and only then modeled by the generator. We take a different route and propose Direct3D-V3, a…

The full abstract on arXiv
Read the original paper