RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generation

AI summary

Overview

Research area: Computer Vision, specifically diffusion-based image generation conditioned on 3D assets. Technical level: Advanced (requires familiarity with diffusion models, transformers, LoRA, and 3D representations). One-sentence scope: RefAny3D introduces a dual-branch diffusion framework that takes a 3D mesh as reference and generates photorealistic images that faithfully match the asset's geometry and texture by jointly producing RGB images and point maps.

What This Paper Is About

Current subject-driven image generation methods can preserve a subject's identity from 2D reference photos, but they cannot accept 3D assets (meshes) as references, which limits their usefulness for creators who work in 3D. This paper asks how to condition a diffusion model on a 3D asset so that the generated 2D images are simultaneously consistent in identity, geometry, and texture. The proposed solution, RefAny3D, generates the RGB image and its corresponding point map together, using the point map as a structural anchor that ties 2D pixels to 3D object coordinates.

Key Contributions

1. A 3D asset-referenced image generation framework that uses multi-view RGB images and point maps of a mesh as conditioning signals, achieving faithful geometry and texture alignment with the reference asset.
2. A spatially aligned, domain-decoupled dual-branch architecture that simultaneously generates RGB images and point maps in a unified diffusion process, giving the model explicit 3D structural awareness.
3. Two decoupling mechanisms — Domain-specific LoRA (separate Reference-LoRA and Domain-LoRA) and Text-agnostic Attention — that prevent information leakage between the RGB and point-map domains.
4. A new pose-aligned training dataset built on Subjects200k, augmented with 3D meshes (via Hunyuan3D) and 6D object poses (via FoundationPose), with rigorous filtering using Mask IoU and LPIPS.

Main Findings

• Superior GPT-based evaluation scores: RefAny3D achieves the best scores on texture consistency (6.315 vs. 5.631 next best), geometric consistency (7.368 vs. 6.684), aesthetic quality (7.687 vs. 7.105), and overall (7.123 vs. 6.367) against five baselines.
• Strongest fine-grained matching: The GIM keypoint count reaches 3901.316, substantially higher than all baselines (~3100–3570), indicating much closer correspondence between generated images and real 3D asset details.
• Competitive semantic metrics: The method leads on CLIP and DINO average/max similarities except against IP-Adapter on one metric, while also achieving the highest CLIP text-image score (0.340).
• Best user study ranking: Human raters rank RefAny3D first overall (Rank 1.579, lower is better), with the strongest Faithfulness (4.655) and ID (4.737) scores.
• Ablations confirm every component matters: Removing shared positional embeddings causes cross-domain misalignment; removing Text-agnostic Attention or Domain-specific LoRA causes background bleeding from the RGB/text branches into the point map; removing the point-map branch makes training unstable and degrades 3D consistency.
• View-count scaling: Performance improves monotonically as conditional views increase from 4 to 6 to 8, but the method still works reasonably with fewer views.
• Editing-based alternatives fail: Using a rendering-plus-editing pipeline (Qwen-Image-Edit-2509) produces hallucinations and foreground–background mismatches that RefAny3D avoids.

Methodology in Plain English

The researchers start from a 3D mesh and render it from multiple viewpoints, capturing both color images and "point maps" — images where each pixel stores the 3D coordinate of the surface point it represents. These two kinds of images are fed into a diffusion model (Flux.1-dev) as conditioning signals. The model is trained to generate a new RGB image and a matching point map at the same time.

Because point maps encode geometry uniquely (unlike textures, which can repeat or look ambiguous across views), they serve as reliable anchors that tie 2D pixels to 3D positions. To keep the two output streams aligned, the model uses shared positional embeddings with a shift trick that places conditional tokens next to but separate from target tokens.

Generating color and geometry together creates a problem: the point map has no background, while the RGB image does, so information from the RGB branch or text prompt can "bleed" into the point map. The authors solve this in two ways. First, two separate LoRA adapters — a Reference-LoRA that learns general appearance from references and a Domain-LoRA activated only for point-map tokens — keep domain knowledge separate. Second, a Text-agnostic Attention mask prevents text tokens from influencing the point-map branch, so geometry stays clean while the RGB branch can still use both text and geometry.

Training data was built by taking Subjects200k, extracting objects with GroundingDINO and SAM, converting them to meshes with Hunyuan3D, and estimating poses with FoundationPose, then filtering out bad samples with Mask IoU and LPIPS thresholds.

Why This Matters

Impact on research: This is the first work to formulate 3D asset-referenced image generation as a distinct task and to propose a diffusion framework that jointly models RGB and point-map distributions. It bridges two previously separate research lines — 2D subject-driven generation and 3D-aware image synthesis — and provides a reusable recipe (dual-branch generation with domain decoupling) for future cross-domain diffusion systems.

Real-world applications:

• E-commerce and advertising, where marketers need to place a specific product mesh into diverse lifestyle scenes without re-shooting.
• Game and film production, allowing artists to visualize how a 3D asset will look in different scenes, lighting conditions, and viewpoints.
• Interior and industrial design, letting designers preview catalog meshes in candidate environments.
• AR/VR content creation, generating context imagery around existing 3D content.

Industry relevance: The method requires no per-asset fine-tuning, unlike DreamBooth or Textual Inversion, which is a practical advantage for pipelines that must handle many assets. It integrates with existing multi-view-to-3D generation tools, making it a natural component in asset-centric production workflows.

Future Directions

• Handling non-rigid and deformable objects (ropes, cushions, cloth), which the current model treats as rigid due to dataset limitations.
• Reducing the computational and time overhead when conditioning on many viewpoints, possibly via more efficient attention mechanisms.
• Extending to full scene-level conditioning where multiple 3D assets and physical interactions between them are modeled.
• Improving the data pipeline to include higher-quality in-the-wild 3D reconstructions, since errors in Hunyuan3D outputs propagate into training.

Target Audience

Researchers and graduate students working on diffusion models, subject-driven generation, or 3D-aware image synthesis will get the most from this paper. Practitioners in e-commerce, game development, and content creation who need to render 3D assets into photographic scenes will also find it relevant. A working knowledge of diffusion models, transformer architectures, LoRA, and 3D coordinate representations is assumed.

Authors’ abstract

In this paper, we propose a 3D asset-referenced diffusion model for image generation, exploring how to integrate 3D assets into image diffusion models. Existing reference-based image generation methods leverage large-scale pretrained diffusion models and demonstrate strong capability in generating diverse images conditioned on a single reference image. However, these methods are limited to single-image references and cannot leverage 3D assets, constraining their practical versatility. To address this gap, we present a cross-domain diffusion model with dual-branch perception that leverages multi-view RGB images and point maps of 3D assets to jointly model their colors and canonical-space coordinates, achieving precise consistency between generated images and the 3D references. Our spatially aligned dual-branch generation architecture and domain-decoupled generation mechanism ensure the simultaneous generation of two spatially aligned but content-disentangled outputs, RGB images and point maps, linking 2D image attributes with 3D asset attributes. Experiments show that our approach effectively uses 3D assets as references to produce images consistent with the given assets, opening new possibilities for combining diffusion models with 3D content creation.