SUPRA-TEX: 3D Generation with Super-Aligned Texture

AI summary

Overview

Research area: Computer Vision — generative 3D asset creation, specifically relightable physically based rendering (PBR) texture generation from a single reference image.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching models, sparse voxel ("O-Voxel") latents, PBR material channels, and transformer attention design, though the central idea is explainable without that background.

Scope (one sentence): The paper introduces SUPRA-Tex, a framework that converts a reference photograph into surface-aligned PBR material observations and then completes the full 3D surface around those fixed observations using an interleaved flow transformer.

What This Paper Is About

Generating a relightable 3D asset from one photo requires reproducing fine details — text, logos, patterns — at exactly the right surface locations while plausibly filling in the parts of the object the camera never saw. The authors call this requirement super-alignment, and argue that existing methods either blur sharp reference details during multi-view projection and fusion, or force a single 3D generator to simultaneously infer physical materials from visual features, reconstruct fine detail, and invent unseen regions. SUPRA-Tex's goal is to split those responsibilities: let 2D priors estimate materials in the reference view, ground them on the 3D surface through camera projection, and let the 3D generator focus on completing the unobserved surface.

Key Contributions

1. Native observation encoding. Reference-view PBR estimates (albedo, metallic, roughness) are projected onto valid visible surface voxels and encoded with the frozen pretrained material VAE encoder into latents that live in the same space as the generation target — so the 3D model receives material evidence rather than abstract image features. Unobserved voxels are omitted rather than zero-padded.
2. Interleaved flow transformer. A generator that keeps the observation latents as fixed, read-only "memory" supplying keys and values at every block, while alternating between updating only unobserved cells (even blocks) and the full surface (odd blocks).
3. A spatially indexed conditioning interface. Because observations are indexed by 3D surface coordinates, the same interface accepts additional views (including 2D-synthesized back views) and interactive 2D edits without retraining the 3D backbone.
4. State-of-the-art results from public data only. SUPRA-Tex leads all shaded and PBR metrics on two held-out benchmarks for geometry-conditioned texturing, leads all 9 metrics for end-to-end single-image-to-3D generation, and with synthetic back-view conditioning reaches reference fidelity and visual quality comparable to or better than five commercial systems in paired comparisons.

Main Findings

• Geometry-conditioned texturing (Table 1, 16 evaluation views). Against ten baselines, SUPRA-Tex reaches shaded PSNR 30.89 / 30.05 and foreground shaded PSNR 22.34 / 20.64 on DTC-Tex / Complex-Tex, with albedo foreground PSNR 19.55 / 18.12 and the lowest metallic/roughness MAE (0.043/0.041 and 0.109/0.084). Relative to the feature-conditioned Pixal3D⋆ baseline this is +1.7 dB and +1.2 dB shaded PSNR, and +2.0 dB and +0.9 dB albedo foreground PSNR.
• A stronger 2D prior transfers directly. The variant "Ours ‡", whose 2D material estimator is fine-tuned from Qwen-Image-2.1 instead of FireRed-Image-Edit-1.1, improves further to shaded PSNR 32.08 / 30.66 and albedo foreground PSNR 21.14 / 18.27 without changing the 3D completion model.
• Reference-view and novel-view gains separate cleanly (Table 2). SUPRA-Tex has the best mean in every reported metric on both view splits and both datasets. Against the strongest baseline per split, reference-view foreground shaded PSNR gains are 2.56 dB (DTC-Tex) and 0.95 dB (Complex-Tex), while novel-view gains are 1.76 dB and 1.09 dB. Reference-view LPIPS is 0.0132 (DTC-Tex) and 0.0202 (Complex-Tex).
• End-to-end generation (Table 3, no ground-truth geometry or cameras). SUPRA-Tex leads all 9 metrics on both datasets, reaching PSNR 28.65 / 28.73, LPIPS 0.025 / 0.029 and CLIP-I 0.971 / 0.968 on DTC-Tex / Complex-Tex. Versus Pixal3D⋆ (which supplies the shared geometry), four-view PSNR improves by 1.77 dB and 2.49 dB, foreground PSNR by 0.89 dB and 1.60 dB, and LPIPS drops by 64.8% and 58.6%.
• Competitive with commercial systems (Table 4, 45 real and synthetic images, two-view conditioning). Human pairwise win rates are 84.7% (HY3.1), 75.8% (Rodin 2.5), 71.6% (Tripo H3.1), 60.8% (Meshy 7.1) and 51.1% (Hi3D 3.0) on input alignment, and the paper reports 51.1%–84.8% preference in visual-quality comparisons. SUPRA-Tex's VLM scores are 8.08 (alignment), 8.67 (PBR) and 8.34 (quality), tying the highest VLM PBR score of 8.67.
• Read-only observations with interleaved updates beat other injection schemes. On the ablation protocol, SUPRA-Tex improves shaded foreground PSNR over residual injection by 0.87 dB (DTC-Tex) and 0.29 dB (Complex-Tex), matches joint-update attention on shaded fidelity while improving albedo foreground PSNR by 0.15 and 0.16 dB with lower MR MAE, and samples 1.20× and 1.18× faster than joint-update attention.
• Predicted metallic/roughness carry real information. Replacing predicted MR with fixed values (M=0, R=0.5) or dataset means costs 0.44 and 0.26 dB shaded foreground PSNR on DTC-Tex, and 0.69 and 0.57 dB on Complex-Tex. Replacing all channels with ground-truth PBR (Oracle PBR) gains 2.31/1.12 dB shaded and 2.62/2.51 dB albedo foreground PSNR — a remaining gap the Qwen-Image-2.1 variant narrows to 0.98/0.47 dB and 1.03/2.36 dB.
• Partial encoding is not a slice of complete encoding. Using ground-truth geometry, cameras and PBR on Complex-Tex, the authors encode complete materials and the filtered visible subset separately and compare latents at identical observed coordinates, showing the two differ. (The figures for this analysis fall in the truncated part of the provided text and are not reported here.)

Methodology in Plain English

The pipeline has three stages.

Stage 1 — Estimate reference-view materials with a 2D model. An MLLM-based image-editing model is trained to output lighting-separated albedo and packed metallic–roughness maps (plus an auxiliary normal map) from a single photograph in one forward pass, using "factorized modality-axis joint attention" so the output channels share information at each spatial location. It is trained on Objaverse RGB images with rendered ground-truth material and normal maps using a latent flow-matching objective at a fixed maximum-noise endpoint, so inference needs no iterative denoising: one pass takes 1.2 s on one H800 and produces 1024 × 1024 material maps.

Stage 2 — Ground those materials on the surface. Using the mesh and camera projection, each foreground pixel is mapped to its first visible surface hit, unreliable boundary correspondences are filtered out (foreground-mask erosion, depth-discontinuity filtering), and the surviving materials are voxelized. The frozen pretrained material VAE encoder compresses only the observed voxels, producing "native observation latents" that share the exact latent space the 3D generator already uses. Each observation also carries metadata: a coverage ratio (how much of a latent cell's voxels were actually seen) and a boundary distance in graph steps to the nearest unobserved region.

Stage 3 — Complete the surface around fixed evidence. A flow-matching transformer generates the full material latents conditioned on geometry latents and the observation set. Two streams coexist in the same 3D coordinate frame: a dynamic generation stream and a static, read-only observation stream that never receives updates and only provides keys and values. Blocks alternate between querying only unobserved cells (to concentrate capacity where synthesis is needed) and querying the whole surface (to keep the result globally coherent). Training supervises the generation stream against complete-material targets everywhere, including observed cells. Inference starts from noise and integrates the learned velocity field with 25 Euler steps and no classifier-free guidance, then decodes with the frozen material decoder; optional KV caching reuses the observation stream's keys and values across steps (the paper reports runtime and memory effects in its Table 6, which is outside the provided text).

Why This Matters

The paper's core argument is architectural: with 3D training data scarce, forcing one 3D network to learn material inference, fine-detail preservation and surface hallucination at once is wasteful. Offloading material estimation to mature 2D priors and supplying the result as spatially aligned material evidence measurably improves fidelity at the reference view while also improving novel views — the two need not trade off. The released code and checkpoints (supra-3d.github.io/supratex) and the fact that training uses only public datasets make the recipe reproducible in a way commercial systems are not.

Real-world applications (as motivated or implied by the paper):

• Product digitization and e-commerce: turning a single product photo into a relightable 3D asset with legible labels, logos and patterns.
• Embodied AI and simulation: assets whose appearance can be re-lit consistently for training and evaluation.
• Digital twins and catalog work: the DTC-Tex benchmark is drawn from the Digital Twin Catalog of product scans.
• Content pipelines for AR/VR and games: fast asset texturing with optional interactive 2D edits that propagate without retraining.

Industry relevance: The head-to-head comparison against five commercial systems on real photographs, with roughly half-to-over-four-fifths pairwise preference and a tied top VLM PBR score, positions this as a direct competitor to closed 3D asset services while publishing weights and training on public data only.

Future Directions

• Better 2D material estimation is the main lever. The measured gap to Oracle PBR (2.31/1.12 dB shaded, 2.62/2.51 dB albedo foreground PSNR) shows that 3D texture quality is currently limited by the 2D estimator; the Qwen-Image-2.1 variant's partial closure of that gap suggests further gains from stronger 2D priors without touching the completion model.
• Richer multi-view and editable conditioning. The authors note that additional views, 2D-generated back views and interactive 2D edits plug into the same coordinate-indexed interface; how far this extends (more views, conflicting edits, video input) is left open.
• Scalable 3D training data. The paper's premise is that 3D data scarcity motivates the division of labor; how the design behaves as native 3D material data grows is an open question.
• Efficiency of the observation stream. KV caching is presented as an optional optimization with runtime and memory characterized in a separate table, leaving room for further sampling-cost reduction.

Target Audience

Researchers and practitioners in generative 3D vision and graphics — particularly those working on texture synthesis, PBR/material generation, sparse-latent 3D diffusion, and image-to-3D systems — as well as industrial teams building product-digitization or asset-creation pipelines. Readers wanting only the practical takeaway can focus on the three-stage pipeline description and the benchmark tables; the transformer attention formulation in Section 3.4 and the latent-encoding analysis are aimed at readers implementing similar systems.

Authors’ abstract

Single image 3D generation aims to dress a given 3D mesh with relightable materials while faithfully preserving the visual details of a reference image. Existing methods either introduce artifacts when fusing inconsistent multi-view predictions or struggle to preserve fine details through implicit image-feature conditioning. To resolve these challenges, we introduce SUPRA-Tex, built on the principle of decoupling visible-material estimation, surface alignment, and observation-guided 3D completion. This division of labor allows the 3D generator, trained on limited 3D data, to focus its capacity on preserving visible evidence and completing unobserved surfaces. Specifically, we first train an MLLM-based image editing model to predict lighting-separated PBR maps in the reference view. We then project these maps onto the visible surface and encode them with the pretrained material VAE encoder into native observation latents that share the generation target's latent space. An interleaved flow transformer holds these latents as fixed, read-only anchors while alternating unobserved-region and full-surface updates to generate complete material latents. This design achieves super-alignment with the reference, delivering What You See Is What You Get fidelity on visible regions and coherent completion on unobserved surfaces. Trained solely on public datasets, SUPRA-Tex achieves SOTA performance in both geometry-conditioned texturing and end-to-end asset generation, matching or even exceeding leading commercial systems on in-the-wild images. Project page: https://supra-3d.github.io/supratex/