PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewards

AI summary

Overview

Research area: Computer vision and generative modeling — specifically reward-based post-training of text-to-video (T2V) diffusion models, using Optimal Transport (OT) to align text and video representations.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching denoisers, Vision-Language Model embeddings, Optimal Transport, and reinforcement learning fine-tuning (GRPO).

Scope in one sentence: The paper introduces PISCES, an annotation-free T2V post-training algorithm whose Dual OT-aligned Rewards module (a distributional quality reward and a discrete token-level semantic reward) improves both visual quality and text-video semantic alignment without any human preference labels.

What This Paper Is About

Reward-based post-training can improve T2V generators, but existing methods either require large-scale human preference annotations (expensive, hard to scale) or derive rewards from pre-trained VLM text-video embeddings that are distributionally misaligned. The paper's goal is to close that gap by aligning text embeddings to the real-video embedding space with Optimal Transport before computing rewards, so that an annotation-free method can match or beat annotation-based ones.

Key Contributions

1. PISCES framework. An annotation-free T2V post-training algorithm that identifies misaligned text-video embeddings from pre-trained VLMs as the core bottleneck in existing annotation-free rewards, and addresses it by aligning embeddings with OT before reward computation — stated as the first use of OT for reward supervision in generative post-training.

2. Dual OT-aligned Rewards module. Composed of (i) a Distributional OT-aligned Quality Reward, which uses a learned OT map to project text embeddings into the real-video manifold and scores cosine similarity between [CLS] representations, and (ii) a Discrete Token-level OT-aligned Semantic Reward, which builds a spatio-temporal cost matrix over text and video tokens and solves a partial OT problem with an entropic Sinkhorn solver.

3. State-of-the-art results claimed across both paradigms. PISCES reportedly outperforms annotation-based and annotation-free post-training methods on VBench Quality, Semantic, and Total scores for both short-video (VideoCrafter2) and long-video (HunyuanVideo) generation, corroborated by a human preference study.

4. Optimization-agnostic reward design. The Dual OT-aligned Rewards module is shown to work with direct backpropagation (via consistency distillation) and with reinforcement learning fine-tuning (GRPO).

Main Findings

• VBench gains on VideoCrafter2 (short video). PISCES reaches Total 82.75, Quality 84.05, Semantic 77.54, versus the vanilla VideoCrafter2 baseline at Total 80.44, Quality 82.20, Semantic 73.42. Compared with annotation-free baselines, this exceeds T2V-Turbo (81.01 / 82.57 / 74.76) and T2V-Turbo-v2 (81.87 / 83.26 / 76.30, reproduced without motion guidance), and compared with annotation-based baselines it exceeds VideoReward-DPO (80.75 / 82.11 / 75.29), VideoDPO (81.93 / 83.07 / 77.38), and UnifiedReward (81.43 / 83.26 / 74.12). VCM drops below the vanilla baseline on all three scores (73.97 / 78.54 / 55.66).

• VBench gains on HunyuanVideo (long video). PISCES reaches Total 85.45, Quality 86.73, Semantic 80.33, versus vanilla HunyuanVideo at Total 83.24, Quality 85.09, Semantic 75.82. Baselines include T2V-Turbo (83.86 / 85.57 / 77.00), T2V-Turbo-v2 (84.25 / 85.93 / 77.52), VideoReward-DPO (83.54 / 85.02 / 77.63), VideoDPO (84.13 / 85.71 / 77.83), UnifiedReward (83.80 / 85.46 / 77.15), and VCM (81.77 / 84.60 / 70.49).

• Comparison against a wider field of T2V models. When post-trained with PISCES, VideoCrafter2 scores Quality 83.73, Semantic 77.63, Total 82.51, and HunyuanVideo scores 86.73 / 80.33 / 85.45. The paper reports these as best-in-class against ModelScope (78.05 / 66.54 / 75.75), Show-1 (80.42 / 72.98 / 78.93), Pika-1.0 (82.92 / 71.77 / 80.69), Gen-3 (84.11 / 75.17 / 82.32), Kling (83.39 / 75.68 / 81.85), vanilla VideoCrafter2 (82.20 / 73.42 / 80.44), and vanilla HunyuanVideo (85.09 / 75.82 / 83.24).

• Human preference study validates the automatic metrics. On 400 prompts with 85 participants, PISCES is consistently preferred over HunyuanVideo, T2V-Turbo-v2, and VideoReward-DPO across visual quality, motion quality, and text alignment.

• OT alignment is the critical ingredient. In the ablation on VideoCrafter2, removing OT ("PISCES w/o OT") lowers the Semantic Score from 77.63 to 75.82 and the Quality Score from 83.73 to 83.44; the full model also has the highest Total Score (82.51) versus 81.92 without OT.

• The two rewards are complementary. Using only the Quality Reward raises the Quality Score from 82.20 to 83.77 and the Total Score to 82.21. Using only the Semantic Reward raises the Semantic Score from 73.42 to 76.99 but leaves Quality at 82.87 and Total at 81.70. The full model combining both yields the best Quality, Semantic, and Total scores (83.73 / 77.63 / 82.51).

• OT outperforms alternative alignment mappings. On a Mutual KNN alignment metric and a Spearman correlation structure-preservation metric over 10,000 WebVid10M text-video pairs: Contrastive (0.2135 / not reported), Mapping w/ L2 (0.2318 / 0.4873), Mapping w/ KL (0.2284 / 0.4720), and OT in PISCES (0.2597 / 0.9018). OT also yields the best downstream post-training scores on HunyuanVideo (Quality 86.73, Semantic 80.33).

• The partial OT plan improves matching quality. Constraining the OT cost matrix with temporal and spatial terms produces accurate token-region correspondences, whereas vanilla cross-attention is diffuse and unconstrained OT misaligns tokens. The paper reports that the discrete partial OT improves InternVideo2's video-text matching performance by 8.11%.

• Distorted embeddings destabilize post-training. In a controlled same-seed experiment with the prompt "A robot reaching for a red/green apple on the table," OT-aligned rewards keep background, robot appearance, and motion stable with only the apple color changing, while an L2-loss mapping produces unstable appearance, object placement, and artifacts such as disappearing objects.

Methodology in Plain English

The authors start from the observation that annotation-free rewards reuse embeddings from VLMs like InternVideo2, and those text and video embeddings do not live in the same distribution — so cosine similarities computed on them are unreliable. Their fix is to align the spaces first using Optimal Transport, a mathematical tool for moving one probability distribution onto another at minimum cost.

Quality side. They train a neural transport map that pushes text embeddings toward the distribution of real-video embeddings, using a Monge-Kantorovich formulation with a squared Euclidean transport cost, implemented via iterative optimization of a transport map and a potential function (both 3-layer MLPs with ReLU and LayerNorm). Once trained, the OT-transformed text [CLS] token stands in as a proxy for a real video embedding, and the Quality Reward is simply the cosine similarity between that transformed text [CLS] token and the generated video's [CLS] token — a measure of whether the generated video points in the same global "quality direction" as real videos.

Semantic side. They augment the cross-attention layers of InternVideo2 with a partial Optimal Transport plan. For each attention head, they build a cost matrix between text tokens and video patch tokens from three terms: semantic similarity (1 minus cosine similarity), a temporal constraint comparing each text token's expected frame index under attention with a patch's frame index, and a spatial constraint comparing the text token's expected 2D position with the patch's coordinate, with weights balancing the temporal and spatial penalties. A partial OT problem is solved on this matrix with an entropic Sinkhorn solver using a fraction-of-mass of 0.9, allowing text tokens to match only a relevant subset of video tokens rather than forcing full mass transport. The resulting transport plan is fused into the attention map in log-space, and the refined features go through InternVideo2's pre-trained Video-Text Matching classifier; the positive-class logit after softmax becomes the Semantic Reward.

Post-training. The two rewards are plugged into two optimization schemes. For direct backpropagation, they add the negative rewards to a consistency distillation loss and optimize the denoiser with LoRA (all other parameters frozen). For reinforcement learning, they use GRPO — sampling a group of videos per prompt, computing the rewards, normalizing them into advantages, and applying a clipped policy-gradient objective. Training uses Euler ODE sampling for direct backprop and SDE sampling for GRPO.

Setup. Experiments post-train VideoCrafter2 (2 seconds at 8 FPS) and HunyuanVideo (5 seconds at 25 FPS) on a balanced mix of WebVid10M and VidGen-1M, resizing frames to 512×320 and 848×480 respectively. Training runs on 8×A100 80GB GPUs for 2 days at a learning rate of 1e-6 with batch size 1 and accumulation 32 (direct backprop), or 4 days for GRPO, which the paper breaks down as roughly 30 hours of actual training for direct backpropagation and roughly 78 hours for GRPO. Training the neural OT map and critic took 1 A100 GPU for one day, equivalent to 24 A100 GPU-hours, using 8-frame clips encoded by frozen InternVideo2. Evaluation is VBench, which aggregates 16 dimensions into Quality and Semantic Scores and a weighted Total Score.

Why This Matters

PISCES attacks the scalability ceiling of reward-based T2V post-training. Annotation-based reward models are effective but require large human preference datasets; annotation-free ones have so far lagged behind. If aligning embeddings with OT really lets an annotation-free method surpass annotation-based ones, it changes the economics of improving video generators.

Impact on research. The paper reframes reward misalignment as a distributional alignment problem, which is a reusable insight: any reward defined on pre-trained cross-modal embeddings (image, video, or multimodal) could be reformulated this way. It also introduces OT-based reward design as a complement to the more common DPO-style and preference-model approaches.

Real-world applications (as identified by the authors):

• Multimedia content creation, where better prompt adherence and temporal coherence reduce manual retakes.
• Education, producing visual explanations and instructional content.
• Robotics and embodied AI, enabling rapid prototyping of scenarios and simulations.
• Scientific visualization, communicating complex processes through controllable video synthesis.

Industry relevance. The method requires only a frozen off-the-shelf VLM and a modest alignment training run (24 A100 GPU-hours for the OT map), plus LoRA post-training of the denoiser. For organizations already building video generation products, this is a comparatively low-cost path to quality and prompt-adherence improvements without building an annotation pipeline, and it applies to both short-video and long-video generators.

Future Directions

• How does this scale? PISCES is validated on VideoCrafter2 and HunyuanVideo; whether the same reward design holds for substantially larger generators and longer durations than the 5-second HunyuanVideo setting is not established here.
• Sensitivity to the partial OT and cost-matrix hyperparameters. The cost matrix balances temporal and spatial penalties with weights whose values are not reported in the provided content, and the Sinkhorn solver runs at a fixed fraction-of-mass of 0.9. How sensitive results are to these choices is an open question.
• Beyond annotation-free rewards. The paper's OT alignment could be combined with human-preference reward models rather than replacing them; whether the two are additive is untested.
• Generalization of the reward signal. A human study on 400 prompts with 85 participants is encouraging but limited; broader cultural, stylistic, and rare-prompt coverage, plus generalization beyond the WebVid10M and VidGen-1M training mixture, remains to be checked.

Target Audience

Researchers and engineers working on video generation, diffusion/flow-matching post-training, reward modeling, and multimodal alignment will get the most from this paper. It is also relevant to practitioners interested in Optimal Transport applied to vision-language representations, and to teams evaluating whether annotation-free post-training is a viable substitute for human preference data pipelines. Readers need a working understanding of embedding-space alignment and reinforcement learning fine-tuning to follow the method sections.

Authors’ abstract

Text-to-video (T2V) generation aims to synthesize videos with high visual quality and temporal consistency that are semantically aligned with input text. Reward-based post-training has emerged as a promising direction to improve the quality and semantic alignment of generated videos. However, recent methods either rely on large-scale human preference annotations or operate on misaligned embeddings from pre-trained vision-language models, leading to limited scalability or suboptimal supervision. We present $\texttt{PISCES}$, an annotation-free post-training algorithm that addresses these limitations via a novel Dual Optimal Transport (OT)-aligned Rewards module. To align reward signals with human judgment, $\texttt{PISCES}$ uses OT to bridge text and video embeddings at both distributional and discrete token levels, enabling reward supervision to fulfill two objectives: (i) a Distributional OT-aligned Quality Reward that captures overall visual quality and temporal coherence; and (ii) a Discrete Token-level OT-aligned Semantic Reward that enforces semantic, spatio-temporal correspondence between text and video tokens. To our knowledge, $\texttt{PISCES}$ is the first to improve annotation-free reward supervision in generative post-training through the lens of OT. Experiments on both short- and long-video generation show that $\texttt{PISCES}$ outperforms both annotation-based and annotation-free methods on VBench across Quality and Semantic scores, with human preference studies further validating its effectiveness. We show that the Dual OT-aligned Rewards module is compatible with multiple optimization paradigms, including direct backpropagation and reinforcement learning fine-tuning. Project page: https://roar-ai.github.io/pisces