TEXTRIX: Latent Attribute Grid for Native Texture Generation and Beyond

AI summary

Overview

Research area: Computer Vision / 3D generative modeling — specifically native 3D texture generation, latent 3D representations, and 3D part segmentation.

Technical level: Advanced. The paper assumes familiarity with diffusion models, diffusion transformers (DiT), variational autoencoders, sparse voxel grids, cross-attention, and UV parameterization of meshes.

Scope: The paper introduces a sparse voxel "latent attribute grid" plus an attribute VAE and an image-conditioned DiT that generates 3D textures natively in volumetric space, and then reuses the same architecture to predict semantic attributes for 3D part segmentation.

What This Paper Is About

Most 3D texturing methods either generate multiple 2D views and fuse them onto a mesh, or generate textures directly in flattened 2D UV space. The first approach produces seams, inconsistent lighting, and missing coverage on occluded surfaces; the second breaks the natural neighborhood relationships of a 3D surface by cutting it into disconnected UV islands. This paper instead represents texture as a native 3D attribute grid — a sparse voxel field storing per-voxel attributes such as color, semantic labels, and PBR materials — and learns to generate that grid directly, avoiding both multi-view fusion and UV fragmentation. The same grid representation is then repurposed to predict part labels, giving high-precision 3D part segmentation with the same architecture.

Key Contributions

1. A native 3D texturing framework built on a sparse attribute grid. It generates textures by coloring the model in volumetric space rather than by fusing multi-view projections or predicting UV maps, thereby resolving inter-view inconsistency and seam artifacts.
2. A sparse latent conditioning strategy. The input image is projected into a 3D voxel grid using a front-view position map, encoded by the pretrained attribute VAE, and injected into the DiT via cross-attention — giving spatially explicit conditioning that anchors frontal alignment and detail preservation.
3. An extensible unified framework. The same sparse grid architecture, trained to predict semantic attributes instead of color, achieves state-of-the-art results on 3D part segmentation, and extends naturally to PBR material generation (roughness, metallic, normal maps) within the same grid structure.
4. An end-to-end attribute VAE. Unlike TRELLIS, which relies on DINOv2 features from multi-view images and decodes into FlexiCubes or 3D Gaussian splatting at a 256³ resolution limit, this VAE is fully end-to-end and scales to 1024³.

Main Findings

• Single-view texture generation beats all baselines. On front-view reconstruction, the method reaches SSIM 0.9421, PSNR 30.0985, and LPIPS 0.0627, versus TRELLIS (0.9150 / 25.5405 / 0.0856), TexGen (0.8976 / 22.4177 / 0.1005), and Paint3D (0.8903 / 21.5729 / 0.1051). On novel-view generation it also leads: CLIP Score 0.8545 and CLIP FID 19.8543 versus TRELLIS (0.8346 / 21.3961), TexGen (0.8206 / 22.1541), and Paint3D (0.8013 / 23.1599).
• It improves existing multi-view pipelines. Combined with MVAdapter, the method posts CLIP 0.8357 and CLIP-FID 21.7571, ahead of MVAdapter alone (0.8305 / 21.9120) and Hunyuan2.0 (0.8124 / 23.1458).
• Segmentation is state-of-the-art on complex geometry but trails on simple geometry. On the Objaverse (Random) 100-mesh subset, the method scores 72.26 mIoU versus PartField's 74.63, SAMPart3D's 46.34, and SAMesh's 44.84. On the Objaverse (Complex) subset — 100 meshes with over 10,000 sharp edges identified via DORA — it leads decisively at 60.82 mIoU versus PartField 51.79, SAMPart3D 36.67, and SAMesh 31.07. The authors attribute the Random-subset gap to grid-based querying introducing minor inconsistencies on large, simple faces that span multiple voxel cells.
• The sparse latent condition is essential. Removing it drops performance from SSIM 0.9421 / PSNR 30.0985 / LPIPS 0.0627 to 0.9052 / 24.3984 / 0.0942.
• Rendering-based supervision beats direct grid supervision. For the VAE, a rendering loss yields SSIM 0.9841, PSNR 33.21, LPIPS 0.0246, compared with a cube MSE loss at 0.9788, 31.59, 0.0421.
• The VAE reconstructs better than TRELLIS's VAE. The paper reports sharper edges and better color fidelity from the latent representation compared with the TRELLIS VAE, even at a lower resolution.
• Fidelity gains come from removing the fusion step entirely. Qualitative comparisons show Paint3D with low fidelity, seams, and poor input consistency; TexGen producing incoherent textures on occluded regions; and TRELLIS showing discoloration artifacts and failing to reproduce fine frontal detail.

Methodology in Plain English

The researchers start by changing what a 3D model's texture is. Instead of a flat 2D image wrapped onto a mesh, texture lives in a sparse grid of small 3D cells (voxels) placed near the object's surface, where each cell stores an attribute vector — color channels, semantic channels, PBR channels, and others, concatenated into one representation. Attributes are read back at any surface point by converting a UV pixel into a 3D world coordinate via a precomputed UV position map, locating the enclosing voxel, and trilinearly interpolating from its eight corners. This lets the model learn a smooth, non-linear attribute field from a sparse grid.

To make generation tractable, they train a variational autoencoder over this grid. Its encoder alternates sparse 3D convolutions (which downsample and extract local features) with a sparse transformer (which models long-range dependencies), and its decoder mirrors this structure in reverse. The VAE compresses a 1024³ input grid to a 128³ latent grid with an embedding dimension of 16. Pruning operations after each upsampling stage remove redundant grid elements, and supervision comes from view-dependent renderings rather than directly from the input voxels, combined with L1, pruning (BCE), KL, LPIPS, and adversarial losses.

On top of this latent space they train a diffusion transformer using a rectified flow objective and a spatial sparse attention mechanism similar to Direct3D-S2. The key conditioning trick: rather than feeding only global CLIP-style features, they first canonicalize the input by generating a front-view image from the reference using a fine-tuned diffusion model and a front-view position map, then project that front-view image's pixels into a 3D volume. That projected volume is fed through the frozen attribute VAE encoder, so the condition lives in the same latent space as the tokens being generated — closing the domain gap. A DINOv3 global feature is added alongside through a second cross-attention branch.

For segmentation, the same model is trained but targets part labels instead of colors, using ground-truth UV maps where each part gets a unique random RGB color. At inference, an existing 2D segmentation method (SAM-based, reference [3]) produces oversegmented masks from rendered views, DINO features merge similar adjacent segments, and the generated label grid is clustered into the final segmentation.

Training used the Objaverse, ObjaverseXL, and private data, with 2048×2048 color-coded UV maps for texture and segmentation as targets. The VAE handles 1024³ grids; the DiT was trained with AdamW at a learning rate of 1×10⁻⁴ and betas (0.9, 0.999), batch size 16 per GPU across 32 NVIDIA A100 GPUs, a flow matching scheduler with 1000 training timesteps, roughly 30,000 iterations, and classifier-free guidance with a 0.1 drop probability. Inference uses the same scheduler with 15 steps and a guidance scale of 3.0.

Why This Matters

Impact on research: The paper challenges the assumption that 3D texturing must pass through 2D machinery — either multi-view projection or UV space. By showing that a single sparse volumetric attribute grid can serve generation and perception, it argues for unifying tasks that are typically handled by separate specialized frameworks. Its segmentation results on high-polycount, high-frequency meshes suggest that native volumetric resolution preserves geometric detail that feature-field methods lose.

Real-world applications:

• Populating immersive virtual environments and digital media with fully textured 3D assets at high fidelity.
• Robot simulation, where assets must be both geometrically and visually coherent for training perception systems.
• Content pipelines for games and film that currently require manual seam cleanup and UV authoring.
• Asset processing tools that need automatic, class-agnostic 3D part segmentation with precise boundaries on complex meshes.
• PBR material authoring (roughness, metallic, normal maps) within the same representation.

Industry relevance: 3D texture generation is a recognized bottleneck in 3D content creation despite rapid progress in geometry generation. A feed-forward native approach that avoids slow per-instance optimization and incomplete surface coverage is directly relevant to anyone building 3D asset pipelines, and the demonstrated ability to boost an existing multi-view adapter (MVAdapter) suggests the method can be layered onto deployed commercial systems rather than replacing them.

Future Directions

• Mesh robustness. The stated limitation is that performance is sometimes sensitive to the quality and topology of the input mesh, particularly with mesh defects or severe self-intersections. Improving robustness to defective geometry is the most direct next step.
• Closing the segmentation gap on simple geometry. The method trails PartField on the Objaverse (Random) subset because grid-based querying causes minor inconsistencies on large, simple faces spanning multiple voxel cells. Adaptive grid resolution or resolution-aware querying could address this.
• Broader attribute unification. The representation is explicitly designed to hold color, semantic, PBR, and "other 3D attributes" in one vector. The paper demonstrates PBR generation but leaves open how far the framework extends to other attribute types and whether they can be generated jointly rather than one task at a time.
• Scaling behavior. The VAE is described as capable of scaling to 1024³, but the practical limits of the sparse volumetric grid at higher resolutions — and the trade-off against the reported grid-query artifacts — are not fully characterized.

Target Audience

Researchers and practitioners in 3D generative modeling, neural rendering, and 3D asset creation, particularly those working on texture synthesis, sparse voxel representations, latent diffusion models, or class-agnostic 3D segmentation. It is also relevant to graphics and simulation engineers who need to convert raw 3D assets into textured, semantically segmented, PBR-ready models, and to readers following the broader debate over whether 3D generation should be "native" or built on lifted 2D models.

Authors’ abstract

Prevailing 3D texture generation methods, which often rely on multi-view fusion, are frequently hindered by inter-view inconsistencies and incomplete coverage of complex surfaces, limiting the fidelity and completeness of the generated content. To overcome these challenges, we introduce TEXTRIX, a native 3D attribute generation framework for high-fidelity texture synthesis and downstream applications such as precise 3D part segmentation. Our approach constructs a latent 3D attribute grid and leverages a Diffusion Transformer equipped with sparse attention, enabling direct coloring of 3D models in volumetric space and fundamentally avoiding the limitations of multi-view fusion. Built upon this native representation, the framework naturally extends to high-precision 3D segmentation by training the same architecture to predict semantic attributes on the grid. Extensive experiments demonstrate state-of-the-art performance on both tasks, producing seamless, high-fidelity textures and accurate 3D part segmentation with precise boundaries.