GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer

AI summary

Overview

Research area: 3D computer vision and generative modeling — specifically appearance transfer (texture plus fine geometric detail) onto 3D assets.

Technical level: Advanced. The paper builds on structured latent representations, rectified flow models, conditional flow matching, and universal diffusion guidance, so it assumes familiarity with modern generative modeling.

One-sentence scope: GuideFlow3D is a training-free framework that steers a pretrained rectified flow model during sampling with differentiable, part-aware guidance objectives so that the appearance of one object can be transferred onto the geometry of another, even when the two objects differ substantially in shape.

What This Paper Is About

3D appearance transfer means taking an input 3D object and giving it the texture and fine surface detail of a separate "appearance" object, while keeping the input object's overall geometry unchanged. Existing methods break down when the input and appearance objects have significantly different geometry, and the authors show that simply applying a 3D generative model directly also fails to produce appealing results. The paper proposes a principled, optimization-guided approach that periodically injects guidance into the sampling process of a pretrained rectified flow model, conditioned on either an image or text.

Key Contributions

1. A training-free guided rectified flow framework. GuideFlow3D applies universal, differentiable guidance to a pretrained rectified flow model, enforcing constraints the model was never originally trained for — extending the concept of universal guidance to 3D generation.
2. Two new guidance losses. A part-aware appearance loss built on geometric co-segmentation, and a self-similarity (structure) loss that preserves intrinsic structure within regions of the input while promoting separability between parts.
3. Flexibility across appearance modalities. The framework handles mesh-image pairs, plain images, and text under a single scheme: when a mesh is available the appearance loss is used (affecting geometry and texture), otherwise the self-similarity loss guides generation from an image or text (texture alone).
4. An evaluation protocol for a ground-truth-free task. Because no ground-truth transferred texture exists and conventional encoder/reconstruction metrics cannot compare dissimilar geometries, the authors build a GPT-based pairwise ranking system across six criteria and validate it against a human user study.

Main Findings

• Guidance alone is not enough, and flow alone is not enough. Optimizing only the proposed objectives shifts latents away from the distribution modeled by the generative network, producing implausible results (Fig. 4a), while applying rectified flow without constraints fails to reflect style texture or semantic structure (Fig. 4b). Interleaving latent optimization with flow sampling yields appealing assets (Fig. 4c).
• Best results in intra-category image conditioning. On the simple-complex intra-category set with image conditioning (L_appearance), GuideFlow3D attains the best (lowest) rankings across all six GPT criteria — Style Fidelity 1.89, Clarity 2.41, Integration 2.07, Detail Quality 2.23, Shape Adaptation 2.28, Overall 2.12 — versus Trellis (2.51, 2.58, 2.53, 2.69, 2.61, 2.62), Cross Image Attention (3.56, 3.48, 3.32, 3.83, 3.47, 3.59), EasiTex (3.18, 4.30, 4.08, 3.17, 4.18, 3.81), MambaST (4.94, 3.55, 4.56, 4.90, 4.42, 4.87) and UV Nearest Neighbor (4.12, 3.84, 4.30, 4.10, 4.43, 4.33).
• Best results under text conditioning as well. With text conditioning (L_structure) on the same set, GuideFlow3D reports 1.54 Fidelity, 1.63 Clarity, 2.01 Integration, 2.15 Detail Quality, 2.44 Shape Adaptation and 1.95 Overall, compared with Trellis (2.01, 1.89, 2.67, 2.75, 2.55, 2.39), SDXL + Cross Image Attention (2.88, 2.52, 3.25, 3.38, 3.29, 2.98) and UV Nearest Neighbor (3.12, 3.21, 3.82, 3.61, 3.43, 3.64).
• Robustness to cross-category transfer. On simple-complex and complex-complex sets with image conditioning, GuideFlow3D leads on Fidelity, Clarity and Adaptation in both intra- and inter-category settings; for example in simple-complex inter-category it reports 1.99 Fidelity, 2.75 Clarity, 2.45 Adaptation, versus Trellis at 2.64, 2.85, 2.76.
• Baseline failure modes are diagnosed qualitatively. MambaST blends textures from input and appearance (a grey hue), EasiTex produces non-smooth texture repetitions and leaves parts untextured, Cross Image Attention omits texture detail and weak local geometry, and Trellis can alter input geometry — for example closing a side hole on a bunk bed and texturing the altered region.
• In-the-wild generalization. Using assets from Objaverse-XL and ABO, GuideFlow3D transfers appearance across strongly mismatched categories (giraffe to chair, chair legs to bike wheels, bed legs to golden retriever legs, cabinet handles to airplane propeller), where Trellis shows weaker structural grounding and inconsistent material mapping.
• Metric choice matters. The authors state that encoder-based metrics such as DINOv2, CLIP and DreamSim, and reconstruction metrics such as PSNR, SSIM, LPIPS and FID, are not representative here because they need ground truth. They instead use GPT-5 pairwise ranking over multi-view renders of input, appearance and output, and report that GPT-5 judgments align with human preferences on this task. The specific user-study agreement figures are not reported in the provided content.
• Runtime cost. The method runs in 96 s on an NVIDIA 4090 GPU, against 78 s for the Trellis baseline, since the approach is optimization-based and not intended for real-time use.

Methodology in Plain English

The starting point is a pretrained generative model (Trellis) that represents a 3D object as a set of local latent codes anchored on a sparse voxel grid — active voxel positions describe coarse shape, and the latent vectors at those positions describe fine geometry and appearance. For appearance transfer, the authors keep the input object's voxel positions fixed (so global geometry is preserved) and only make the latent codes learnable, initializing them from a normal distribution.

Two differentiable objectives then drive those codes. The first, used when the appearance object comes with a textured mesh, tries to pull each input latent toward a semantically and geometrically corresponding latent from the appearance object. Because such correspondences do not exist directly, they are approximated: the authors extract PartField features per voxel, run k-means clustering to co-segment input and appearance shapes into parts, and match latents to nearest neighbours based on those part-based features. The second objective, used when no mesh is available (image or text only), is a part-aware contrastive loss on geometric features — features of voxels in the same geometric cluster are pulled together, and features of voxels in different clusters are pushed apart — so structure is preserved while appearance is applied.

Rather than optimizing these losses on their own, the authors interleave them with the rectified flow sampling steps: each reverse-time step takes a flow update conditioned on the global signal (image or text) and then adds the gradient of the guidance loss. The guidance is framed through a Bayesian view where the conditioned flow models the prior and likelihood, and the guidance terms act as additional factors. The optimized latent feeds the next flow step, and the result is decoded into a mesh (or other 3D representation) with the pretrained decoder.

For evaluation, the authors build a benchmark because none existed: procedural models generate "simple" inputs, an ABO subset supplies "complex" appearance objects, and results are ranked by GPT-5 across six criteria.

Why This Matters

Impact on research. The paper shows that a generative model's inductive bias can be repurposed for an unseen task at inference time through guided rectified flow sampling, without retraining. It also argues that standard perceptual and reconstruction metrics are misleading for this class of problem and offers an LLM-based, human-validated ranking alternative, which is a methodological contribution beyond the specific 3D task.

Real-world applications (as framed by the authors):

• Game and XR content creation, where simple CAD-style primitives can be turned into detailed, textured assets.
• Rapid prototyping and 3D design iteration, since the method accepts mesh, image or text appearance cues.
• Augmented reality and digital twin generation, where coherent stylized geometry matters.
• Scene editing and downstream stylization tasks (mentioned as an appendix demonstration).

Industry relevance. The authors argue the approach could simplify and democratize 3D content creation, making tools more accessible to artists and accelerating development and creativity in XR and gaming. The explicit ethical discussion notes risks around manipulation and deepfakes, misinformation, intellectual property, and bias amplification, and calls for disclosure of AI-generated 3D content and respect for source content licenses.

Future Directions

• Faster inference. Because the method is optimization-based, the authors propose training self-supervised feed-forward models for faster inference as a direction. Current runtime is 96 s versus 78 s for Trellis.
• Reducing dependence on upstream models. The implementation relies on Trellis and PartField, so failures in those models propagate to GuideFlow3D; addressing this dependence is left open.
• Handling noisy real-world meshes. The approach assumes noiseless meshes, which the authors identify as a limiting factor for some application scenarios.
• New guidance objectives. Developing novel guidance objectives for new applications is described as an interesting future research direction, alongside the general claim that the method could be extended to different diffusion models and guidance functions.

Target Audience

Researchers and practitioners working on 3D generative modeling, 3D stylization and appearance transfer, and controllable generation — particularly those already familiar with diffusion or rectified flow models and structured 3D latents. It is also relevant to graphics and XR/gaming engineers interested in asset stylization pipelines, and to researchers interested in LLM-based evaluation protocols for tasks without ground truth. Readers without background in generative modeling and 3D latent representations will find the framework difficult to follow.

Authors’ abstract

Transferring appearance to 3D assets using different representations of the appearance object - such as images or text - has garnered interest due to its wide range of applications in industries like gaming, augmented reality, and digital content creation. However, state-of-the-art methods still fail when the geometry between the input and appearance objects is significantly different. A straightforward approach is to directly apply a 3D generative model, but we show that this ultimately fails to produce appealing results. Instead, we propose a principled approach inspired by universal guidance. Given a pretrained rectified flow model conditioned on image or text, our training-free method interacts with the sampling process by periodically adding guidance. This guidance can be modeled as a differentiable loss function, and we experiment with two different types of guidance including part-aware losses for appearance and self-similarity. Our experiments show that our approach successfully transfers texture and geometric details to the input 3D asset, outperforming baselines both qualitatively and quantitatively. We also show that traditional metrics are not suitable for evaluating the task due to their inability of focusing on local details and comparing dissimilar inputs, in absence of ground truth data. We thus evaluate appearance transfer quality with a GPT-based system objectively ranking outputs, ensuring robust and human-like assessment, as further confirmed by our user study. Beyond showcased scenarios, our method is general and could be extended to different types of diffusion models and guidance functions.