Identity-Preserving Image-to-Video Generation via Reward-Guided Optimization

AI summary

Overview

Research area: Computer vision, specifically image-to-video (I2V) generation, human-centric identity preservation, and reward-guided/reinforcement-learning fine-tuning of diffusion models.

Technical level: Advanced.

Scope: This paper introduces IPRO (Identity-Preserving Reward-guided Optimization), a reinforcement-learning tuning framework that improves facial identity consistency in image-to-video diffusion models without changing model architecture or adding identity modules.

What This Paper Is About

Image-to-video models can turn a single still image into a moving video, but they often fail to keep a person's face looking like the same person as the video progresses — especially when the face is small in the frame and the person moves or changes expression a lot. Existing fixes either bolt on extra identity modules (which break the original model structure and only handle one person) or rely on supervised fine-tuning, which suffers from exposure bias and lets identity drift accumulate over frames. The authors' goal is to improve identity preservation in a general-purpose I2V foundation model by tuning it with a facial reward signal instead of altering its architecture.

Key Contributions

1. A facial reward feedback framework for I2V. The method uses ArcFace embeddings and cosine similarity as a differentiable reward, backpropagated through the sampling chain (using a DRaFT-style truncation of the last K steps) so the model learns on-policy from pure Gaussian noise. The paper states this is "the first facial reward feedback framework for I2V."
2. A facial scoring mechanism (FSM) that builds a multi-view feature pool. Instead of comparing generated frames to the first frame or to their time-aligned ground-truth frame, each generated frame is scored against the average similarity to all ground-truth video frames via cosine similarity, which broadens the reward signal, improves generalization, and reduces copy-paste behavior.
3. A multi-step KL-divergence regularization. The optimization penalizes deviations between the tuned model and the original reference model at each gradient step, expressed as a weighted sum of squared differences between velocity parameterizations. The paper reports this keeps KL divergence low and stable and suppresses reward hacking.
4. Extensive evaluation across three base models. Experiments cover an in-house 15B I2V model (with both MMDiT and single DiT blocks), Wan 2.2 5B, and Wan 2.2 27B-A14B, with comparisons against SFT, CLIP-based reward, DPO, GRPO, MoCA, and Concat-ID, plus a human study.

Main Findings

• FaceSim improves substantially across all three base models. On a small-face evaluation set of 600 scenes, face similarity rose by 45.9% for the in-house I2V model, 44.1% for Wan 2.2 5B, and 20.1% for Wan 2.2 I2V 27B-A14B.
• Raw FaceSim numbers (Table 1). In-house I2V model: 0.4769 to 0.6960 with the reward model. Wan 2.2 5B: 0.3788 to 0.5460. Wan 2.2 A14B: 0.5780 to 0.6942.
• General video quality is largely preserved. VBench-I2V metrics (Subject Consistency, Background Consistency, Aesthetic Quality, Imaging Quality, Time Flickering, Dynamic Degree, Motion Smoothness) did not degrade with the reward model; for example Wan 2.2 A14B Time Flickering went from 0.9676 to 0.9690 and Dynamic Degree from 19.45 to 19.17.
• Better than adapted T2V identity methods. Against methods adapted to I2V, FaceSim was 0.5780 for Wan 2.2, 0.5820 for MoCA†, 0.6056 for Concat-ID†, and 0.6942 for the proposed method.
• Better than DPO and GRPO. FaceSim: Wan 2.2 0.5780, DPO 0.6284, GRPO 0.6334, ours 0.6942. The paper attributes this to dense, calibrated absolute reward signals versus relative preference signals, and to low response diversity within a prompt limiting GRPO's group-wise advantage estimation.
• Better than SFT and CLIP-based reward. FaceSim: Wan 2.2 0.5780, SFT† 0.6392, CLIP† 0.6099, ours 0.6942. The paper notes SFT still struggled to preserve face identity despite explicit face-similarity supervision, and that ArcFace captured fine-grained identity better than the CLIP image encoder.
• Low-noise gradient steps work better. Backpropagating through final (low-noise) steps gave FaceSim 0.6942 and Dynamic Degree 19.17, versus 0.6456 and 18.98 for high-noise initial steps.
• Truncation step K has diminishing returns. FaceSim was 0.6593 (K=1), 0.6712 (K=2), 0.6835 (K=3), 0.6942 (K=4), 0.6966 (K=5); K=4 was chosen for computational cost.
• Reward hacking is measured and controlled. Using Gemini 2.5 Pro as a video-understanding VLM judge on the evaluation set, the hacking rate was 7% for Wan2.2, 58% without KL regularization, 52% without FSM, and 10% with the full method. Notably, removing these components raised FaceSim (0.7544 without KL, 0.7388 without FSM) but produced stiff, expressionless faces.
• Loss weight ratio matters. Ablating the reward/KL weight ratio gave FaceSim 0.7544 with 58% hacking at 0, 0.7215 with 52% at 1, 0.6942 with 10% at 10, and 0.6371 with 9% at 100. The chosen setting was λ1 = 0.1 and λ2 = 1, a ratio of 10.
• Results hold under a different face recognition model. On Wan 2.2 A14B, FaceSim-Arc went from 0.5780 to 0.6942 and FaceSim-Cur (CurriculumFace) from 0.5805 to 0.6989.
• Local temporal constraints add little. Adding local temporal constraints (LTC) yielded FaceSim 0.6128 standalone and 0.6961 combined with the method, versus 0.6942 for the method alone; the authors argue local rewards can propagate identity errors and that current I2V foundation models already handle frame-to-frame consistency.
• Human study favors the method on identity. With 50 images randomly sampled from the small-face evaluation set and 96 volunteers choosing between paired videos, participants judged the method better at identity preservation while rating it comparably on visual quality, text alignment, and motion amplitude.
• Training face scores rise steadily. The face score on the training set increased monotonically across training steps, indicating the identity reward was effective (reported in the supplementary material).

Methodology in Plain English

The researchers treat identity preservation as a reward-maximization problem rather than a data-fitting problem. Starting from pure Gaussian noise, they run the model's own sampling process to produce a video (this is "on-policy" generation, matching what the model does at inference rather than feeding it ground-truth frames). A frozen VAE decoder visualizes the predicted output, and an ArcFace face-identity model scores how closely the generated faces match the faces in the ground-truth video.

To keep memory and compute manageable, the gradient is backpropagated only through the last K sampling steps (K=4), since later, low-noise steps carry the fine appearance details that matter for identity. The reward gradients update trainable parts of the model — for Wan 2.2 27B-A14B, everything is frozen except the low-noise expert model.

For scoring, instead of comparing a generated frame against one reference image (which encourages a frozen, copy-pasted expression) or against its time-aligned ground-truth frame (which provides weak supervision), the method pools all ground-truth frames and averages the cosine similarity of each generated frame against every frame in that pool. This gives a broader, multi-angle reference covering pose, expression, and lighting variation.

To stop the model from gaming the reward, a KL-divergence term penalizes how far the tuned model's sampling trajectory drifts from the original model at each step, expressed as a weighted squared difference between velocity predictions. Training used the ROLL framework, the Adam optimizer with a learning rate of 2e-5 for 100 steps at batch size 64, with a facial reward weight of 0.1 and a KL loss weight of 1. Wan2.2-Lightning, the distilled version of Wan2.2 requiring only 8 steps without classifier-free guidance, was used to improve efficiency.

Data came from high-quality 960p internet videos, with faces detected by SCRFD. To emphasize the hard small-face case, only clips where the largest face bounding box per frame did not exceed 100×100 pixels were kept; videos with less than 40% of frames containing a detectable face were discarded, and Qwen2.5-VL was used to remove videos with faces occluded by objects such as phones or masks.

Why This Matters

Impact on research. The paper argues that identity preservation in I2V is an under-explored problem compared to T2V, and that the usual instinct — injecting more identity features — is misguided, because the identity is already fully encoded in the first frame. The real problem is preservation over time. It also makes a case that reward-based optimization suits quantifiable perceptual metrics like facial identity better than DPO (which only ranks relative preferences) or GRPO (whose group advantages degrade when samples within one prompt are too similar). Because no architectural modules are added, the approach avoids the single-subject limitation of module-based identity injection and generalizes naturally to multi-person scenes, as demonstrated visually in the supplementary material.

Real-world applications:

• Consumer and social video creation, where a user animates a single photo of themselves and expects the person to remain recognizable throughout.
• Film, advertising, and e-commerce content production, where a presenter or model appears across many generated shots and must stay consistent (supported by Alibaba's Taobao & Tmall Group affiliation).
• Virtual avatars and digital humans that need to stay on-model across long or expressive sequences.
• Lowering the barrier for small teams to produce personalized video without per-person fine-tuning, which the paper notes prior methods such as PersonalVideo and MagicID require.

Industry relevance. The work was developed with the Taobao & Tmall Group of Alibaba and Alibaba Group, supported through the Alibaba Research Intern Program, and evaluated on an in-house 15B model — indicating direct applicability to commercial video generation pipelines. The efficiency choices (K=4 truncation, 8-step distilled Lightning sampling, no CFG) point toward practical deployment constraints.

Future Directions

• Extending beyond faces. The stated limitation is that consistency of non-facial attributes — jewelry, accessories, clothing — remains underexplored. The authors plan a unified identity reward model covering these aspects.
• Open questions about reward design. The ablation shows removing KL or FSM raises FaceSim while producing stiff, unnatural faces, which means raw face-similarity numbers are not a sufficient target. Finding reward formulations that optimize identity without hacking remains open.
• Understanding when local temporal rewards help. The experiments found local temporal constraints gave limited benefit, and the authors attribute this partly to I2V foundation models already handling frame-to-frame consistency — a claim that could be revisited as base models change.
• Transfer to other generators and tasks. Only Wan 2.2 5B, Wan 2.2 27B-A14B, and one in-house 15B model were tested; whether the reward framework transfers to other architectures or to T2V identity preservation is not reported.

Target Audience

Researchers and engineers working on diffusion-based video generation, reinforcement learning for generative models, and identity-preserving or subject-driven synthesis. It is also relevant to practitioners at media, e-commerce, and content-creation companies who need person-consistent generated video at scale. Readers should be comfortable with diffusion sampling, reward-based fine-tuning, and evaluation metrics such as FaceSim and VBench-I2V; the paper is not written for beginners.

Authors’ abstract

Recent advances in image-to-video (I2V) generation have achieved remarkable progress in synthesizing high-quality, temporally coherent videos from static images. Among all the applications of I2V, human-centric video generation includes a large portion. However, existing I2V models encounter difficulties in maintaining identity consistency between the input human image and the generated video, especially when the person in the video exhibits significant expression changes and movements. This issue becomes critical when the human face occupies merely a small fraction of the image. Since humans are highly sensitive to identity variations, this poses a critical yet under-explored challenge in I2V generation. In this paper, we propose Identity-Preserving Reward-guided Optimization (IPRO), a novel video diffusion framework based on reinforcement learning to enhance identity preservation. Instead of introducing auxiliary modules or altering model architectures, our approach introduces a direct and effective tuning algorithm that optimizes diffusion models using a face identity scorer. To improve performance and accelerate convergence, our method backpropagates the reward signal through the last steps of the sampling chain, enabling richer gradient feedback. We also propose a novel facial scoring mechanism that treats faces in ground-truth videos as facial feature pools, providing multi-angle facial information to enhance generalization. A KL-divergence regularization is further incorporated to stabilize training and prevent overfitting to the reward signal. Extensive experiments on Wan 2.2 I2V model and our in-house I2V model demonstrate the effectiveness of our method. Our project and code are available at https://ipro-alimama.github.io/.