Uni-Animator: Towards Unified Visual Colorization

AI summary

Overview

• Research area: Computer vision / generative AI — sketch (line-art) colorization for both still images and video, built on Diffusion Transformer (DiT) backbones.
• Technical level: Advanced. The paper assumes familiarity with diffusion models, transformers, positional encoding (RoPE), optical flow, VAEs, DINOv2 and CLIP.
• Scope: The paper proposes and evaluates a single DiT-based framework, Uni-Animator, that performs high-fidelity sketch colorization for both images and videos using single or multiple visual references, with three targeted modules addressing reference fidelity, physical detail preservation and temporal consistency.

What This Paper Is About

Automated sketch colorization currently splits into two separate worlds: methods tuned for static images and methods tuned for video. This paper argues that existing approaches suffer from imprecise color transfer from visual references, loss of high-frequency physical detail (largely through VAE compression), and flickering or misalignment when motion is fast or non-uniform. The goal is a single model that takes a sketch sequence, a text prompt and a set of reference images, and produces colored frames for both the image case (one sketch frame) and the video case (many frames).

Key Contributions

1. A unified framework. Uni-Animator is presented as the first unified framework supporting high-fidelity sketch colorization for both images and videos, removing the need for task-specific model adaptation between still and moving content.
2. Visual Reference Enhancement. A two-stage, multi-granularity reference fusion scheme combining coarse channel-wise fusion (VAE-encoded references concatenated into the noise latent) and fine sequence-wise fusion via an Instance Patch Embedding module using stacked 3D convolutions to capture local color distributions and textures.
3. Physical Detail Reinforcement. Pre-trained DINOv2 features are used to capture high-frequency structure (material properties, surface textures, lighting), projected through an MLP "physical head", concatenated along the token dimension and injected into the DiT, alongside a CLIP-based cross-attention mechanism for resolving text-versus-reference semantic conflicts.
4. Sketch-Based Dynamic RoPE. A motion-aware positional encoding that extracts optical flow from sketch frames with RAFT, decomposes it into horizontal and vertical components, and adapts RoPE frequencies per dimension (temporal, height, width) based on motion intensity — described as requiring no extra training, and degrading to standard RoPE when normalized motion intensity is below 0.1.

Main Findings

• Video colorization leads on FID and LPIPS. On the paper's video benchmark, Uni-Animator reports FID 119.514 and LPIPS 0.193 versus ToonCrafter (124.497 / 0.287), Anidoc (158.382 / 0.463), LVCD (138.828 / 0.454) and ToonComposer (132.319 / 0.304). SSIM is 0.689 and PSNR 20.521, both the highest among the video baselines listed.
• ToonComposer leads video CLIP Score. The paper states ToonComposer leads on the CLIP metric at 0.929 versus Uni-Animator's 0.925.
• Image colorization is led on most metrics. Uni-Animator reports SSIM 0.741, CLIP 0.845, FID 23.761 and LPIPS 0.211, compared with MagicColor (0.713 / 0.798 / 28.953 / 0.231), MangaNinja (0.543 / 0.728 / 43.165 / 0.425) and ColorizeDiffusion (0.512 / 0.762 / 82.201 / 0.557).
• MagicColor leads image PSNR. MagicColor's PSNR of 23.751 exceeds Uni-Animator's 21.961, the only image metric where a baseline is ahead.
• Cross-domain ability is the distinctive claim. The paper states Uni-Animator is the only method delivering strong results on both image and video colorization, which it attributes to the unified design.
• Ablation: physical detail module matters. Removing the PhysEncoder drops image SSIM from 0.741 to 0.737 and raises FID from 23.761 to 24.955; for video, SSIM falls from 0.688 to 0.673, FID rises from 119.514 to 124.380 and temporal consistency falls from 0.971 to 0.966.
• Ablation: instance embedding matters most on video. Removing it drops video SSIM to 0.646, raises FID to 126.841 and LPIPS to 0.231, and lowers temporal consistency to 0.962. The paper describes this as the greatest degradation; note that the reported image LPIPS for this variant is 0.207, which is lower (better) than the full model's 0.211.
• Ablation: dynamic RoPE affects motion quality. Removing it gives video SSIM 0.679, LPIPS 0.201, FID 121.901 and temporal consistency 0.961. No image row is reported for this variant, though the text says image fine details blur slightly.
• Qualitative failure modes identified in baselines. ColorizeDiffusion is described as unstable and hyperparameter-sensitive; MangaNinja as weak at semantic matching and poor with multiple references; MagicColor as prone to fine-grained color errors on small textures and edges; LVCD as failing to track large-scale motion; ToonCrafter as misinterpreting instance colors with multiple references; Anidoc as losing instance identity in complex motion.
• RoPE visualization shows asymmetric motion. The paper observes sketch sequences often exhibit strong horizontal motion with gentle vertical motion, and reports that dynamic RoPE produces faster cosine oscillations for the stronger direction.

Methodology in Plain English

The model is initialized from a pretrained Wan2.1-14B image-to-video diffusion transformer and fine-tuned with LoRA (rank 64, batch size 1, learning rate 2×10⁻⁵, AdamW weight decay 3×10⁻²). Training mixes images and videos, with images sampled from the video data.

Three ideas are layered on top:

1. Getting the reference colors right. Reference images are encoded by a VAE and concatenated into the noisy latent along the channel dimension, giving the model global style and structure. Then a patch-embedding layer plus a convolution-based instance encoder injects finer, instance-level features along the sequence dimension, so the model sees both the overall look and local details of each reference.

2. Keeping fine detail. Since VAE encoding erases high-frequency texture, a frozen DINOv2 extracts structure-aware features from the references. A small MLP head projects them to the model's dimension, and they are concatenated with the fused latent and fed into the transformer. Separately, CLIP visual embeddings of the references are concatenated with T5 text embeddings and used as keys/values in a cross-attention layer, which is intended to arbitrate when the text and the reference images disagree (the paper's example is text asking for "pink hair" while the reference shows another color).

3. Stopping video flicker. For each consecutive pair of sketch frames, RAFT estimates optical flow, which is split into horizontal (u) and vertical (v) components with magnitude M = √(u² + v²). A scaling rule f_dynamic = f_base · (1 + α · M̂) rescales the rotary positional encoding frequencies, where M̂ is normalized motion intensity in [0, 1], with α_t = 0.1 for the temporal dimension, α_h = 0.3 for height (driven by vertical flow) and α_w = 0.3 for width (driven by horizontal flow). Faster motion thus gets higher frequencies and finer temporal resolution.

Data and evaluation: Training uses 5K manually curated anime video clips drawn from 5 films (the paper names Spirited Away and Doraemon) plus 30K filtered clips from Sakuga-42M, all normalized to 512×512 single-channel grayscale sketches. Testing uses 100 high-quality animation video clips and 1.5K sampled image pairs, with character instances extracted via SAM as references. Metrics are SSIM, PSNR, CLIP Score, FID, LPIPS (written as "LIPIS" in the evaluation-metric description) and a custom Temporal Consistency Score, with lower FID/LPIPS and higher SSIM/CLIP/Temporal Consistency being better. Baselines get each method's default configuration: a single reference for image methods and the original first frame for video methods, while Uni-Animator uses extracted instance-level references for both.

Why This Matters

• Research impact: The paper argues against the prevailing split between image-specific and video-specific colorization models, and demonstrates that one DiT backbone with reference-fusion, physical-feature injection and motion-aware positional encoding can serve both. It also introduces a training-free mechanism (Dynamic RoPE) for adapting positional encodings to motion, which is transferable to other video generation tasks.
• Animation production: The paper positions the work as reducing the manual, frame-by-frame color layering and flicker-suppression work that animators currently perform for long sequences.
• Film restoration: Restoring and recoloring archival footage is one of the stated application areas, where consistent per-instance color across many frames is required.
• Game development: Rapid generation of colored assets and character variants from sketches, including customizing characters by swapping visual references while keeping the original motion.
• Mixed-content pipelines: Because the same model handles a single sketch frame and a sketch sequence, studios do not need separate image and video toolchains.
• Industry relevance: The paper highlights labor cost, scalability for long sequences and production efficiency as the bottlenecks the framework targets; the reported FID/LPIPS gains on video and the ability to use five or more references for image colorization are the concrete arguments for deployment.

Future Directions

• Real-time processing and higher-resolution outputs, both explicitly named as future work in the conclusion.
• Generalization beyond anime-style 2D animation. Training data is drawn from anime video clips (5K curated clips from 5 films plus 30K Sakuga-42M clips) and the test set is described as randomly sampled from animation films; behavior on other domains is not reported.
• Cost and robustness of the motion pipeline. Dynamic RoPE depends on RAFT optical flow over consecutive sketch frames; the paper does not report the runtime overhead of this estimation, nor how it behaves on very long sequences.
• Standardizing temporal evaluation. The paper uses a custom Temporal Consistency Score defined by the authors with no baseline supplying the same number in Table 1, so how it compares across methods is not established.

Target Audience

Researchers and graduate students working on diffusion models, video generation and controllable image synthesis; graphics and animation engineers evaluating automated colorization for production pipelines; and technical artists or studio R&D teams looking for a single model that handles both still and moving sketch colorization with reference-driven style control.

Authors’ abstract

We propose Uni-Animator, a novel Diffusion Transformer (DiT)-based framework for unified image and video sketch colorization. Existing sketch colorization methods struggle to unify image and video tasks, suffering from imprecise color transfer with single or multiple references, inadequate preservation of high-frequency physical details, and compromised temporal coherence with motion artifacts in large-motion scenes. To tackle imprecise color transfer, we introduce visual reference enhancement via instance patch embedding, enabling precise alignment and fusion of reference color information. To resolve insufficient physical detail preservation, we design physical detail reinforcement using physical features that effectively capture and retain high-frequency textures. To mitigate motion-induced temporal inconsistency, we propose sketch-based dynamic RoPE encoding that adaptively models motion-aware spatial-temporal dependencies. Extensive experimental results demonstrate that Uni-Animator achieves competitive performance on both image and video sketch colorization, matching that of task-specific methods while unlocking unified cross-domain capabilities with high detail fidelity and robust temporal consistency.