Controllable 3D Object Generation with Single Image Prompt

AI summary

Overview

Research area: Computer vision, specifically image-to-3D generative modeling — synthesizing a 3D object from a single input image using diffusion priors.

Technical level: Advanced. The paper assumes familiarity with Neural Radiance Fields (NeRF), DMTet, score distillation sampling, Stable Diffusion, ControlNet, IP-Adapter, and CLIP embeddings.

Scope (one sentence): The paper proposes a way to generate controllable 3D objects from a single image prompt without the extra training cost of textual inversion, adding a depth-conditioned warmup strategy to improve 3D consistency.

What This Paper Is About

Existing image-to-3D methods typically rely on textual inversion, which trains a pseudo text prompt to describe the input image before that prompt can drive a text-to-image diffusion model. This training step costs additional time and removes the ability to control conditions such as depth, pose, and text. The paper's goal is to generate 3D objects directly from a single image prompt — optionally combined with conditions like depth, pose, sketch, or text — while preserving the fine details and 3D consistency of the input.

Key Contributions

1. Controllable image prompt score distillation sampling. A single image is used as the prompt without any textual inversion, producing novel views of a 3D object and enabling control over optional conditions (depth, pose, text, and others such as scribble and canny).
2. Depth conditioned warmup strategy. Depth estimated from the 3D object (NeRF/DMTet) is fed as a condition to ControlNet to prevent degradation of 3D consistency, and a warmup schedule mitigates the instability of that depth in early training epochs.
3. Demonstrated controllability and quantitative/qualitative performance. The method generates diverse, controllable 3D objects and is reported to quantitatively and qualitatively outperform alternative approaches on public benchmarks.
4. User study on fidelity and 3D consistency. Participants rated generated 3D objects on a 1–10 scale for how well results match the input image and whether 3D consistency is maintained, with the proposed method reported as outperforming the alternatives.

Main Findings

• No textual inversion needed. The baseline that uses textual inversion must learn a pseudo text prompt from the input image, which the paper states requires an additional 1–2 hours of training time. The proposed method uses the input image directly, reducing training time.
• CLIP-similarity results (Table 1). On RealFusion15, the method scores 0.850 versus 0.828 for Magic123 with textual inversion and 0.827 without; LRM scores 0.705 and LGM 0.529. On NeRF4 it scores 0.782 (Magic123: 0.774 with textual inversion, 0.747 without; LRM 0.615, LGM 0.480). On CO3D it scores 0.730 (Magic123: 0.670 with textual inversion, 0.702 without; LRM 0.630, LGM 0.546). On GSO it scores 0.761, slightly below Magic123 with textual inversion at 0.763, and above LRM 0.710 and LGM 0.564. Note that the paper's table lists a downward arrow for the NeRF4 CLIP-similarity row, which is inconsistent with the other rows and with the surrounding text.
• A-LPIPS results are mixed. On RealFusion15, the method records A-LPIPS VGG 0.0342 and Alex 0.0213, versus Magic123 with textual inversion at 0.0790 / 0.0559 and without at 0.0770 / 0.0536, but LRM is far lower at 0.0071 / 0.0036 and LGM at 0.0109 / 0.0069. The paper explicitly acknowledges that "LRM and LGM perform well in terms of the A-LPIPS metric, our approach excels in CLIP-similarity."
• Warmup epoch ablation (Table 2). CLIP-similarity is 0.830 with 0 warmup epochs, 0.834 at n = 15, 0.829 at n = 30, and 0.784 at n = 45. The paper reports n = 15 as optimal.
• Training time (Table 1). Reported training time is approximately 1 minute for LRM, approximately 1 minute for LGM, approximately 30 minutes for Magic123 without textual inversion, approximately 120 minutes for Magic123 with textual inversion, and approximately 30 minutes for the proposed method.
• Ablation study (Fig. 9a). Removing IP-Adapter produces undesirable artifacts because a textual inversion trained on a single image fails to capture the input image's concepts or styles in detail; removing the depth condition causes the 3D model to fail to maintain 3D consistency; removing the depth conditioned warmup strategy also harms 3D consistency because predicted depth is unstable in early epochs.
• User study. Specific numerical scores are not reported in the paper text; the paper states that on a 1–10 scale the method shows matched or better performance than RealFusion, Zero-1-to-3, Magic123, LRM, and LGM on both fidelity and 3D consistency.
• Controllability. Fig. 7 shows the model incorporating scribble, pose, depth, and canny conditions; Fig. 8 compares against Control3D on the sketch condition, where the paper notes Control3D takes sketch and text as input rather than an image.

Methodology in Plain English

The authors build on the Magic123 architecture and train in two alternating coarse-to-fine stages. In the first stage they train a NeRF (Instant-NGP, three layers with 64 hidden dimensions) that predicts volume density and color, and therefore an image and a depth map, from an arbitrary camera pose. In the second stage they fine-tune with DMTet, which separates geometry (a deformable tetrahedral grid with a signed distance function and triangle deformation vectors) from texture, giving higher resolution.

To avoid textual inversion, the input image is passed through a pretrained Image Prompt Adapter (IP-Adapter) — a pretrained CLIP image encoder plus a single linear layer — to become a "controllable image prompt." That prompt, along with the depth predicted by the NeRF/DMTet, conditions a depth-conditioned ControlNet v1.1 built on Stable Diffusion v1.5 to predict a noise estimate, which is used to compute a 2D score distillation sampling loss. A separate 3D score distillation loss uses the pretrained Zero-1-to-3 model from the 100,500-iteration checkpoint, which the paper notes gives better 3D consistency.

The total loss combines the 2D and 3D score distillation terms with a reconstruction loss (comparing the input image and its mask, produced by a dense prediction transformer, against the rendered reference view), a monocular depth regularization loss based on negative Pearson correlation with pseudo depth from a pretrained monocular depth estimator, and a normal smoothness loss that compares normals before and after a Gaussian filter with kernel size 9. Because the NeRF-derived depth is unreliable early on, the depth-conditioned ControlNet is withheld for the first 15 epochs, during which only the Zero-1-to-3 branch is used; after the first stage, both ControlNet and Zero-1-to-3 are used.

Experiments use NeRF4, RealFusion15, GSO, and CO3D; NeRF4 contains mic, ficus, drums, and chair (a back-view case), RealFusion15 contains fifteen real-world photos, GSO uses single images of squirrel, school bus, hammer, dog, and shark, and CO3D uses the bench, car, bicycle, and hydrant scenes.

Why This Matters

The paper targets a task — turning one photograph into an explorable 3D object — that conventionally requires experts in 3D graphics tools and considerable time. Removing the textual inversion step simplifies the pipeline and cuts the training cost reported in the paper's comparison, while adding controls that text-prompt-only methods cannot offer. For research, the contribution is a concrete demonstration that an off-the-shelf image adapter can replace a learned pseudo prompt inside a score-distillation pipeline, plus evidence that a depth-conditioned warmup schedule matters for 3D consistency.

Real-world applications the paper points to:

• Augmented Reality (AR) and Virtual Reality (VR), where 3D assets must be produced quickly.
• Gaming, where content pipelines conventionally depend on 3D graphics specialists.
• Broader generative media the paper lists for generative models generally, including artwork creation, video generation, automatic colorization, virtual try-on, and 3D computer graphics.

Industry relevance comes from the practical cost profile: the paper reports about 30 minutes of training versus about 120 minutes for the textual-inversion variant it compares against, and from the ability to accept optional conditions such as depth, pose, sketch, scribble, canny, and text — useful when an artist or developer wants to steer the result rather than accept whatever the image prompt produces.

Future Directions

• Reliability of the evaluation metrics. The paper itself questions whether CLIP-similarity and A-LPIPS are reliable for image-to-3D models, which leaves open the need for better automatic metrics — especially given that LRM and LGM beat the proposed method on A-LPIPS while losing on CLIP-similarity.
• Closing the A-LPIPS gap. Since LRM and LGM achieve substantially lower A-LPIPS on RealFusion15, NeRF4, and CO3D while training in about 1 minute versus about 30 minutes, improving adjacent-view consistency without sacrificing image alignment is an open problem.
• Dependence on pretrained components. The pipeline relies on Stable Diffusion v1.5, ControlNet v1.1, Zero-1-to-3, and IP-Adapter; how the approach behaves with stronger or differently trained diffusion backbones is not explored in the reported content.
• Scope of the evaluations. The datasets are limited to a small set of objects and scenes (four NeRF4 objects, fifteen RealFusion15 photos, five GSO objects, four CO3D scenes); how well the method scales to larger or more varied object categories and to full scenes rather than objects is not established in the paper.

Target Audience

Researchers and graduate students working on 3D generative models, image-to-3D synthesis, NeRF and DMTet representations, and diffusion-based score distillation. It is also relevant to practitioners in AR/VR, gaming, and 3D content pipelines who want a controllable, lower-training-cost alternative to textual-inversion pipelines, and to readers interested in how IP-Adapter and ControlNet can be repurposed as conditioning mechanisms inside a 3D optimization loop.

Authors’ abstract

Recently, the impressive generative capabilities of diffusion models have been demonstrated, producing images with remarkable fidelity. Particularly, existing methods for the 3D object generation tasks, which is one of the fastest-growing segments in computer vision, pre-dominantly use text-to-image diffusion models with textual inversion which train a pseudo text prompt to describe the given image. In practice, various text-to-image generative models employ textual inversion to learn concepts or styles of target object in the pseudo text prompt embedding space, thereby generating sophisticated outputs. However, textual inversion requires additional training time and lacks control ability. To tackle this issues, we propose two innovative methods: (1) using an off-the-shelf image adapter that generates 3D objects without textual inversion, offering enhanced control over conditions such as depth, pose, and text. (2) a depth conditioned warmup strategy to enhance 3D consistency. In experimental results, ours show qualitatively and quantitatively comparable performance and improved 3D consistency to the existing text-inversion-based alternatives. Furthermore, we conduct a user study to assess (i) how well results match the input image and (ii) whether 3D consistency is maintained. User study results show that our model outperforms the alternatives, validating the effectiveness of our approaches. Our code is available at GitHub repository:https://github.com/Seooooooogi/Control3D_IP/