Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling

AI summary

Overview

Research area: Computer vision — 3D and 4D generative modeling, specifically subject-driven (personalized) generation and identity preservation.

Technical level: Intermediate. The paper includes diffusion-model formalism (forward/reverse processes, DDIM sampling, LoRA finetuning) but the core pipeline is conceptually describable without that math.

Scope: A three-stage pipeline (TIRE) that takes a rough 3D or 4D asset from an existing generative model and progressively repaints its unseen regions using video tracking plus a personalized 2D inpainting model, then lifts the result back into 3D.

Paper details: arXiv:2510.23605v1 [cs.CV], submitted 27 Oct 2025, by Shuhong Zheng, Ashkan Mirzaei, and Igor Gilitschenski, with affiliations at the University of Toronto, Vector Institute, and Snap Inc. Project website: https://zsh2000.github.io/track-inpaint-resplat.github.io/

What This Paper Is About

Modern 3D and 4D generation methods are optimized for photorealism, efficiency, and aesthetics, but they do not reliably preserve the semantic identity of a specific subject across viewpoints. The authors show that even a state-of-the-art 4D model, L4GM, produces side and back views of a cat that do not resemble the subject in the source view (in their example, it introduces a blueish tone on originally occluded regions).

The goal is subject-driven 3D/4D generation: given a single image (for 3D) or a video (for 4D) of one subject, produce an asset that faithfully represents that subject's identity from every angle. The authors argue that personalization for 3D/4D generation is still largely underexplored, and propose TIRE as a general, orthogonal enhancement layer on top of existing feed-forward generators.

Key Contributions

1. Framing subject-driven 3D/4D generation as an orthogonal, complementary problem. Rather than replacing feed-forward or native 3D/4D pipelines, the authors position their method as a general post-hoc enhancement that operates on rendered 2D frames and can be applied to any 3D representation.

2. A three-stage method, TIRE (Track, Inpaint, Resplat). Track identifies which regions need infilling using long-video tracking; Inpaint uses a customized, subject-driven 2D inpainting model to progressively fill unseen regions while matching the subject's identity; Resplat unprojects the infilled 2D observations back to 3D while maintaining cross-view consistency.

3. The progressive texture infilling paradigm. The inpainting model is trained on the source view and augmented versions, then applied at a ±20° "sweet spot" that acts as an anchor viewpoint, then to ±90°, and finally from ±90° to ±180°, using backward tracking at each step to reduce the area needing infilling.

4. A new benchmark and comprehensive evaluation. The authors construct the DreamBooth-Dynamic dataset from the original DreamBooth dataset (built on animatable subjects) for subject-driven image-to-3D and video-to-4D generation, and evaluate with DINO feature similarity, six vision-language models, and a user study.

Main Findings

• Identity preservation improves over baselines. In qualitative comparisons (Fig. 4 for image-to-3D; Figs. 5 and 6 for video-to-4D), TIRE produced assets with significantly better identity preservation than baselines including Wonder3D, SV3D, LGM, MeshFormer, TRELLIS, Hunyuan3D-v2.5, STAG4D, SV4D, L4GM, and Customize-It-3D.

• Recent native 3D models also fail at identity. The authors report that even the most recent image-to-3D advancements, TRELLIS and Hunyuan3D-v2.5, still face substantial challenges in producing identity-preserving 3D assets.

• Geometry improves as a side effect. Although TIRE targets appearance, it also outperformed L4GM on geometry, with fewer ghosting artifacts from cross-view inconsistency. The authors attribute this to Track and Inpaint propagating source-view pixels to target views and to the mask-aware halfway diffusion-denoising step in Resplat.

• DINO similarity is a misleading metric here. On DINO feature similarity, TIRE scored 0.5665 (ViT-S/16) and 0.5815 (ViT-B/16), beating L4GM (0.5506 / 0.5694) and STAG4D (0.5287 / 0.5592), but Customize-It-3D ranked highest at 0.5773 / 0.6087 — despite performing worse in qualitative comparisons. The authors conclude DINO similarity is not suitable for this task and discuss the limitation in App. C.

• VLM-based evaluation favors TIRE. Following DreamBench++, six VLMs (GPT-4o, OpenAI o4-mini, Gemma 3 27B, Gemini 2.0 Flash, Qwen2.5-VL-7B, Mistral-Small-3.1-24B-Instruct) scored subject consistency from 0–4 across shape, color, texture, and facial features. TIRE averaged 1.854, versus Hunyuan3D-v2.5 at 1.703 and TRELLIS at 1.476. All scores remained far from the perfect score of 4, indicating the task is far from solved.

• Users prefer TIRE. With 18 volunteers, 10 randomly selected samples from DreamBooth-Dynamic, and 540 total scores on a 1–10 overall-quality scale, TIRE scored highest. Participants were not told the work focuses on subject-driven generation.

• Backward tracking beats forward tracking. Forward tracking (source view to target views) produced many small inpainting regions and grainy results; backward tracking (target views to source view) produced more accurate masks in better shapes, because the source view contains the richest identity information.

• Progressive infilling is necessary. Without it, the model fills the target view with source-view appearance regardless of pose — e.g., painting a cat's whiskers at a target view 60° off the source view that actually shows the cat's side.

• The denoising schedule matters. Using the first 15% of the schedule left some regions unaltered; using 50% risked overly intense, less realistic texture changes; the authors empirically adopt 30%.

Methodology in Plain English

TIRE does not generate a 3D or 4D asset from scratch. It starts from a rough asset produced by an existing model (LGM for 3D, L4GM for 4D in the reported results) and renders multi-view observations from azimuth ±180° and elevation 0°.

Stage 1 — Track. The rendered views are ordered as a video along the camera path. The video tracking model CoTracker establishes correspondence between the given source view and target views. If a tracked point on a target view stays within the valid mask and its visibility flag stays active, the corresponding source point is considered visible there. Crucially, tracking runs backward, from target views to the source view, because the source view holds the most identity information; this yields cleaner masks better suited to inpainting.

Stage 2 — Inpaint. A pretrained stable diffusion inpainting model is personalized with LoRA weights, finetuned against randomly generated binary masks, with the loss applied only on foreground valid regions using the fixed prompt "A photo of sks." Because only one source image/video exists, the authors augment it with horizontal flipping and small rotations within 15°. Inpainting then proceeds progressively: first the ±20° "sweet spot," which serves as an anchor viewpoint balancing exploration and exploitation of unseen regions; then viewpoints within ±90°, tracked back to the anchor to shrink the area needing infilling; then ±90° to ±180°, using the ±90° result as the next anchor. Only the first 30% of the denoising schedule is used so the original structure is not significantly changed.

Stage 3 — Resplat. Because each frame was infilled separately, the frames may disagree. A multi-view diffusion model refines consistency with a mask-aware latent update that only updates latents on unseen viewpoints, keeping the conditional source view fixed (V = 4 views, with one extra entry for the conditional image). The first 30% of the denoising schedule is used here too. Finally, pixels from every viewpoint are splatted into 3D Gaussians using LGM or L4GM, a step the authors note is adaptable to other large reconstruction models and representations beyond Gaussians.

Why This Matters

The paper argues that personalization has been largely neglected as the 3D/4D field has optimized for speed and realism, and that identity preservation is a core requirement for real-world personalized content creation. Its practical appeal is that it does not require a new 3D generative backbone — it sits on top of existing ones and only manipulates rendered 2D frames, so it can be applied to any 3D representation.

Real-world applications:

• Personalized avatars and digital characters generated from a single photo or short clip of a person or pet.
• Consumer content creation tools where users want their own subject (a product, a toy, a pet) rendered as a rotatable or animatable 3D asset.
• E-commerce and marketing, producing turnable 3D product visualizations that stay faithful to the actual item.
• Gaming and AR/VR asset pipelines, populating scenes with identity-consistent personalized characters.

Industry relevance: Two of the three authors are affiliated with the University of Toronto and Vector Institute, and one with Snap Inc., suggesting direct relevance to consumer camera and augmented-reality applications where personalized, identity-faithful 3D content is a natural product feature.

Future Directions

• Better quantitative metrics. The authors demonstrate that DINO feature similarity ranks Customize-It-3D best despite worse qualitative results, and note that purely vision-based evaluation is inadequate for this task. Developing subject-fidelity metrics tailored to 3D/4D assets remains open.

• Closing the gap to a perfect score. All VLM-based subject consistency scores remain distant from the maximum of 4, which the authors read as evidence that subject-driven 3D/4D generation is still far from solved.

• Combining with newer backbones. Because the pipeline is model-agnostic, a natural next step is pairing it with evolving native 3D generators; the authors already show it can further enhance Hunyuan3D-v2.5 in App. F.

• Scaling and generalization. The paper reports results on the constructed DreamBooth-Dynamic dataset and on in-the-wild data from the L4GM project webpage, but the dataset size and composition details are deferred to App. B, and the study focuses on single subjects — extending to more subjects and broader in-the-wild conditions is left open.

Target Audience

Researchers and practitioners in generative 3D/4D vision who care about personalization, subject fidelity, and identity preservation — particularly those working with feed-forward reconstruction models such as LGM and L4GM, multi-view diffusion, or 2D diffusion inpainting and LoRA personalization. It is also useful for applied engineers building personalized AR/VR, avatar, or e-commerce content pipelines, and for anyone studying evaluation methodology for subject-driven generation, given the paper's finding that DINO similarity misranks methods on this task.

Authors’ abstract

Current 3D/4D generation methods are usually optimized for photorealism, efficiency, and aesthetics. However, they often fail to preserve the semantic identity of the subject across different viewpoints. Adapting generation methods with one or few images of a specific subject (also known as Personalization or Subject-driven generation) allows generating visual content that align with the identity of the subject. However, personalized 3D/4D generation is still largely underexplored. In this work, we introduce TIRE (Track, Inpaint, REsplat), a novel method for subject-driven 3D/4D generation. It takes an initial 3D asset produced by an existing 3D generative model as input and uses video tracking to identify the regions that need to be modified. Then, we adopt a subject-driven 2D inpainting model for progressively infilling the identified regions. Finally, we resplat the modified 2D multi-view observations back to 3D while still maintaining consistency. Extensive experiments demonstrate that our approach significantly improves identity preservation in 3D/4D generation compared to state-of-the-art methods. Our project website is available at https://zsh2000.github.io/track-inpaint-resplat.github.io/.