EmoReward: Self-Evolving Reward Calibration for Emotional Video Generation RL

AI summary

Overview

• Research area: Computer vision and generative AI — specifically reinforcement-learning (RL) post-training of video generation models, with a focus on emotional expressiveness rather than generic visual quality.
• Technical level: Advanced. The paper assumes familiarity with Group Relative Policy Optimization (GRPO), reward models, diffusion/flow-based video generators, and multimodal evaluators.
• Scope (one sentence): The paper introduces EmoReward, a training mechanism that calibrates how much trust to place in different emotional-evidence "reward providers" by learning from the traces of a completed training run and applying those calibrated weights to the next run, without retraining any reward model.

What This Paper Is About

Reward-driven RL works well when correctness is checkable (code, mathematics), but emotional expression in video has no single correct answer, so existing systems fall back on fixed mixtures of visual-quality, motion-quality, and text-alignment rewards that poorly capture emotional intent and can be gamed by the policy. The authors' goal is to make the reward system decide, per video, which emotional evidence is trustworthy (a visible face, a semantically relevant motion cue, a color palette) and, across training runs, how much each evidence source should count. EmoReward does this by combining a per-video gating mechanism (Reflective Reward Gating) with a persistent memory (RewardMemory) distilled from a completed run's reward–outcome traces.

Key Contributions

1. EmoReward itself — described by the authors as, to their knowledge, the first self-evolving training mechanism for emotional video generation RL that calibrates reward providers across training runs without updating reward-model parameters (providers stay frozen while the generator is updated).
2. RewardMemory — described as, to their knowledge, the first memory designed for multi-reward systems in video generation. It distills provider reliability into bounded fusion weights, enabling iterative calibration of RL reward signals across runs.
3. A cross-run self-evolution paradigm — a completed training run is treated as a unit of reusable experience that informs reward allocation in the next run.
4. Extensive validation — experiments across three video RL algorithms (DanceGRPO, FlowGRPO, SAGE-GRPO) on HunyuanVideo 1.5, plus transfer experiments on two image backbones (Stable Diffusion 3.5 and FLUX), ablations, mechanism analyses, and an ordinal human study.

Main Findings

• VideoAlign gains across all three RL algorithms: under the paper's fair-comparison setting (batch size 2, group size G = 4, fusion α = β = 0.5, evaluated on a 302-prompt subset that does not overlap the training set), all four VideoAlign heads increase for every RL algorithm tested. Overall improves by 0.3218 for DanceGRPO with Reflect, 0.2343 for FlowGRPO with EmoReward, and 0.5645 for SAGE-GRPO with EmoReward.
• Specific Table 1 numbers: SAGE-GRPO goes from Overall −0.8269 to −0.2624, VQ −0.4106 to −0.1710, MQ −0.3275 to −0.0926, and TA −0.0888 to 0.0012 with EmoReward. FlowGRPO goes from −0.7719 to −0.5376 Overall, and its TA moves from −0.1398 to 0.0323. DanceGRPO with Reflect goes from −1.2812 to −0.9594 Overall.
• Preference proxies move less: PickScore and CLIPScore show smaller and mixed changes; the authors explicitly state the VideoAlign gains do not imply uniform improvement on all evaluation axes.
• Best reward policy in the controlled seven-arm ablation: under SAGE-GRPO, EmoReward with RewardMemory achieves the highest reliability selectivity (0.428) and both composite scores (core 0.922, emo 0.884), versus 0.307 / 0.238 / 0.308 for VideoAlign-only, 0.376 / 0.266 / 0.272 for round-robin, and 0.371 / 0.246 / 0.191 for fixed fusion. The variant without RewardMemory leads the standalone preference pillar (EF = 1.000).
• Memory reallocates rather than activates: the largest global multipliers go to "macro" and "motion" (1.833 and 1.510), while three constant channels receive 0.778. The authors caution that macro shares evaluator information with the reference outcome, and that memory cannot activate invalid evidence.
• Emotion-category coverage: the GT-versus-SAGE-GRPO-with-EmoReward comparison across six metrics and eight emotion categories shows the four VideoAlign heads improving in every category, with smaller and nonuniform PickScore/CLIPScore changes.
• Human study favors EmoReward: with 18 participants rating 30 prompts on Bad/Normal/Good, EmoReward gets the highest Good shares for emotion fidelity (60.0%), temporal coherence (54.3%), and visual quality (57.2%), compared with 53.0%, 51.5%, and 45.0% for SAGE-GRPO. It also has the lowest Bad shares (15.4%, 20.2%, 15.9% versus 31.5%, 28.9%, 30.9% for SAGE-GRPO).
• A distribution-level nuance: SAGE-GRPO receives more Bad ratings than the pretrained GT on every criterion, even though it has more Good ratings — the authors argue this is why the full ordinal distribution matters rather than a binary preference.
• Image transfer is objective-dependent: SD3.5–DanceGRPO improves on HPSv3 with calibration, but FLUX's HPSv3-optimized baseline remains strongest on that metric. Unequal SD3.5 training budgets limit causal attribution.

Methodology in Plain English

The system runs a loop with four named stages — Plan, Act, Reflect, Evolve — wrapped around a standard GRPO training procedure.

Plan (turn the prompt into a checklist). A planner converts the emotional prompt into an "EmoScript": scalar valence and arousal targets, target facial action-unit intensities, body cues, and an expression window saying when in the clip the expression should occur. This separates what emotion is intended from which observable cues should be checked. The paper notes that continuous valence–arousal–dominance trajectories and dominance targets are outside this implementation.

Act (score the video with several independent judges). A video is sampled from the policy and scored by a pool of providers. Seven providers enter the fusion policy — a VLM judge, facial action units, body cues, temporal evidence, CLIP-AUTT, R1-Omni, and motion naturalness — while chromatic atmosphere and kinetic intensity serve as auxiliary readouts, giving nine observation channels in total. Crucially, each provider returns both a score and a validity flag, so a missing face channel's neutral default is not treated as evidence.

Reflect (per-video gating). Because a fixed weighting assumes the same trust allocation in every scene, the authors multiply each provider's base weight by four computed (not learned) gates: a phase gate tied to training-step warmup, a face gate limiting channels that need visible faces, a conflict gate that attenuates disagreement between semantic providers, and a subtlety gate that restrains intensity-sensitive cues. Invalid evidence gets zero gate. The remaining active providers are renormalized into a single emotion reward, which is then blended with the original video reward as a weighted average.

Evolve (learn trust from a finished run). This is the distinctive part. During one complete reference training run, the system logs traces: context signature, provider scores, gates, weights, total reward, and the outcome. Afterward, RewardMemory groups traces into emotional "regimes," computes each provider's correlation with the outcome, centers it against the mean correlation across providers, clips it, and exponentiates it into a bounded multiplier. Providers that agreed more with outcomes get more weight next time; constant providers get zero correlation and a reduced multiplier. The resulting table is frozen for the entire next run — only the generator receives gradients, and runtime lookup neither re-estimates reliability nor writes back. With no memory entry, the multipliers default to 1 and the system degrades gracefully to the gating-only variant.

Configuration and evaluation. Memory calibration uses K = 7 providers, temperature τ = 0.5, a log-multiplier cap c = 1, and minimum context support n_min = 20, with a global fallback when a context has too few traces. VideoAlign supplies both the base video reward and the reference outcome y. The paper is careful to note that VideoAlign is also an optimization objective, so agreement with it is distinguished from independent emotional fidelity.

Why This Matters

Impact on research. The paper reframes a familiar problem — reward hacking in subjective domains — as a weight-allocation problem rather than a reward-model-capacity problem. Because providers stay frozen and only a small multiplier table changes, the approach sidesteps the cost and instability of retraining evaluators, and it turns a completed training run into reusable data. It also argues mathematically that constant terms cancel inside GRPO's group-normalized advantage, so calibration matters specifically when provider validity and relative contribution vary across candidates — a useful clarification for anyone designing multi-reward RL.

Real-world applications:

• Advertising and short-form video generation, where matching a requested emotional tone (warmth, suspense, exuberance) is the actual product requirement, not just visual sharpness.
• Virtual avatars and digital humans, where facial and gestural expressiveness must be judged jointly and where face visibility varies shot to shot.
• Assistive and therapeutic media, where restrained sadness or calm should not be rewarded for exaggerated motion — precisely the failure mode the subtlety gate targets.
• Animation and previsualization pipelines, where a studio wants to steer a generator toward a directorial intent and can supply run-level feedback rather than thousands of pairwise annotations.

Industry relevance. The work comes from a collaboration involving Tencent Hunyuan and The Hong Kong University of Science and Technology, ships with a project page and public GitHub repository, and is validated on production-scale backbones (HunyuanVideo 1.5) plus Stable Diffusion 3.5 and FLUX. Its "reuse experience across runs" framing is directly applicable to industrial post-training pipelines where each RL run is expensive and human preference data is the bottleneck.

Future Directions

• Sustained multi-generation evolution. The experiments test a single reference-to-calibrated transition, not improvement across many generations. The authors flag distribution shift, stale evidence, and feedback loops that could reinforce a proxy's own errors as the key obstacles, and suggest retaining diverse historical traces and evaluating held-out outcomes.
• Isolating contextual from global calibration. The authors state that a matched global-memory control remains necessary to isolate the benefit of regime-specific multipliers over a single global table.
• Extending beyond emotion. The framework needs only a task-appropriate provider pool and a meaningful outcome signal; the authors name action fidelity and audio–visual correspondence as natural visual extensions, and language models / tool-use agents as candidates combining semantic judges, execution checks, and trajectory-level assessments.
• Model-level human feedback. The trace representation suggests a way to translate a reviewer's structured assessment of a finished model ("movement is too weak in these contexts") into bounded changes to the next run's allocation. Mapping subjective feedback to providers and regimes, reconciling raters, and preventing compensating failures are left open, and the paper's ordinal study evaluates outputs without implementing this mechanism.

Target Audience

Researchers and engineers working on RL post-training for generative media — particularly those who already use GRPO-style methods and are hitting the limits of hand-tuned multi-reward weights. It is also relevant to practitioners building reward-model ensembles or agentic evaluators for subjective qualities, and to teams in video-generation product organizations who need steerable emotional control without large-scale human preference annotation. Readers without background in RL fine-tuning of diffusion/flow models will find the algorithmic sections demanding, though the Plan/Act/Reflect/Evolve framing is readable at a conceptual level.

Authors’ abstract

Group Relative Policy Optimization (GRPO) has advanced generative model post-training in verifiable domains such as code and mathematics, but evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Existing approaches typically rely on fixed rewards for visual quality, motion quality, and text alignment that poorly capture emotional intent and can invite reward hacking, while RLHF is difficult to scale and subject to individual aesthetic differences that complicate the calibration of facial, body motion, and chromatic cues across narrative contexts. We introduce EmoReward, a new self-evolving training mechanism that distills evidence reliability from reward traces collected during an actual training run to adaptively guide multi-reward weight allocation in subsequent training by determining which emotional evidence to trust and how much to weight it. EmoReward first plans specific grounding cues from the emotional prompt to guide evaluation, then uses heterogeneous signal primitives to score the generated video during the act stage. Reflective Reward Gating calibrates their contributions through instance-level gates for phase, face confidence, conflict, and subtlety, then renormalizes weights over the active set to produce the sample's fused reward. RewardMemory then estimates each primitive's reliability from rollout scores and outcomes within each emotional regime in the finished training run, then fixes bounded multipliers that calibrate fusion weights in the next training run without updating reward models. Experiments on HunyuanVideo 1.5 show improvements across multiple RL algorithms and compare EmoReward with other agentic reward methods under SAGE-GRPO. Ablations and transfer experiments on SD 3.5 and FLUX further support adaptive reward calibration for visual generation.