RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

AI summary

Overview

Research area: Computer vision and generative video modeling, specifically reward modeling for reinforcement learning (RL) post-training of video generation models.

Technical level: Intermediate to Advanced. The paper assumes familiarity with reinforcement learning from human feedback, group-relative policy optimization, multimodal LLMs, and correlation-based evaluation metrics, though the core intuition (giving a scorer explicit criteria instead of asking for a bare number) is accessible to a general reader.

Scope in one sentence: The paper proposes RewardVerse, a video reward framework that inserts a learned, query-specific "dynamic rubric" between the evaluation query and the scorer to reduce unstable scalar scoring, plus a two-stage training algorithm (RGPO) that jointly optimizes the rubric generator and the scorer.

Note: the supplied paper content is truncated (the appendix stops mid-sentence in Section A.1.3), so several appendix items referenced in the main text — including A.1.5 protocol comparisons, A.1.6 pairwise position-bias analysis, and A.1.7 full alignment analysis — are not fully reported here and are not reproduced below.

What This Paper Is About

Video reward models are meant to score generated videos so that reinforcement learning can push video generators toward human-preferred outputs, but most existing models map a video directly to a single number without stating what that number means. Because video quality is subjective and multi-dimensional, this unconstrained mapping produces what the authors call scalar drift: scores collapse into a narrow band or shift wildly across prompts, leaving RL with weak or unreliable signals. The paper's goal is to make the scoring process explicit and stable by generating a rubric first, then scoring against it, and by training both steps together.

Key Contributions

1. Rubric-as-reward for video reward modeling. The authors state they are the first to introduce the rubric-as-reward paradigm to video reward modeling, using a dynamic rubric (themes, weights, and scoring tips) as an intermediate representation between query and scorer to stabilize pointwise scores and mitigate scalar drift.

2. Data-efficient joint policy optimization (RGPO). A two-stage training algorithm under Group Relative Policy Optimization: Stage 1 warms up the scorer with self-evolving seed rubrics; Stage 2 jointly optimizes a rubric generator to produce query-adaptive criteria while continuously calibrating the scorer against human ratings — using only 30 preference pairs per dimension and no prior supervised fine-tuning.

3. Systematic empirical validation. Evaluations on a 16-dimension EvalVerse pointwise test set and on the external VGRB benchmark, plus component ablations and a downstream RL experiment in which RewardVerse serves as the reward for GRPO fine-tuning of a video generator.

4. Diagnostic study of scalar drift. A 2×2 ablation (rubric present/absent × natural-language float vs. soft-logits decoding) over a 240-video pointwise hold-out pool that isolates which design choice fixes which symptom.

Main Findings

• Rubrics expand score resolution. Holding the soft-logits decoding format fixed, adding a rubric (V2 → V3) expands the scoring standard deviation by +82.6% (σ: 0.258 → 0.471) for Qwen2.5-VL-7B. Under natural text generation, the rubric expanded the collapsed standard deviation from 0.203 → 0.275 for Qwen2.5-VL-7B, while reducing the noisier variance of Gemini-3.1-Pro from 1.500 → 1.158.

• Rubric alone is not sufficient. In variant V4 (rubric + natural-language float), two-thirds of comparison pairs still tied because natural-language outputs remain anchored to the model's high-band linguistic prior. The authors therefore adopt a soft-logits readout (expected value over rating tokens "1"–"5") as their deployed configuration (V3).

• Best pointwise correlation on EvalVerse. The full model ("Joint (Ours)") achieves the highest PLCC in 14 of the 16 dimensions, reaching 0.554 PLCC and 0.446 SRCC on the table's Avg. column (defined in the table caption as the macro-average over the 10 dimensions supported by all compared reward models). Gains are largest on cognitive and temporal dimensions: Logic PLCC of 0.750 versus a next-best 0.593, and Action PLCC of 0.566 versus 0.390. VideoScore-v1.1 reaches a PLCC of −0.245 on Logic.

• Strong pairwise transfer on VGRB. On the seen Visual Quality split, RewardVerse reaches 0.545 accuracy with ties and 0.660 without ties, outperforming the strongest non-oracle baseline, VisionReward (0.474 / 0.590), by 7.1 and 7.0 percentage points. On the unseen Text Alignment split it reaches 0.471 with ties and 0.623 without ties, the highest among non-oracle baselines. The oracle upper bound, VideoReward, reaches 0.538 / 0.722 (Text Alignment) and 0.596 / 0.756 (Visual Quality).

• Both training stages matter. Averaged over 16 EvalVerse dimensions, full RGPO scores 0.530 PLCC and 0.416 SRCC. Removing Stage-1 scorer warm-up drops PLCC by 0.099 and SRCC by 0.061 (to 0.431 / 0.355); removing Stage-2 joint optimization yields a 0.074 PLCC gap (0.456 / 0.364); removing hierarchical tips (flat rubric) gives 0.497 / 0.398.

• Rubric-free training is far less data-efficient. Under the same 480-pair budget, zero-shot rubric-free soft-logits scoring scores 0.361 PLCC / 0.234 SRCC and direct rubric-free pointwise training scores 0.439 / 0.351, both well below full RGPO.

• Base-model biases differ. Gemini-3.1-Pro systematically underestimates (bias of −0.398 in V1 and −0.584 in V4), whereas Qwen2.5-VL shows systematic overestimation. On Gemini-3.1-Pro over the 1K pointwise test set, adding the rubric raises PLCC by +0.103 (a 21.0% relative improvement) and SRCC by +0.084 (17.5% relative), but worsens calibration — motivating explicit RGPO training.

• Better downstream RL behavior. Using RewardVerse as the reward for GRPO fine-tuning of Wan-2.2-A14B on Visual Quality improves Imaging Quality from 0.640 → 0.653, keeps VBench-Quality essentially stable (0.808 → 0.809), and improves VBench-Text from 0.428 → 0.446. Using VideoReward instead raises Imaging Quality to 0.648 but reduces VBench-Quality to 0.804 and VBench-Text to 0.392.

Methodology in Plain English

The framework splits evaluation into two roles that share one model backbone but use different prompts.

The rubric generator sees only the query — the target dimension plus the text prompt used to synthesize the video — and never the video itself. This deliberate blindness prevents it from writing lenient criteria for bad videos or harsh ones for good videos. It outputs a structured rubric: a set of themes, a weight per theme that sums to one, and execution tips for each theme.

The scorer then rates the video against each theme. Instead of reading a generated digit, it reads the model's logits over the five rating tokens ("1" through "5") and computes a weighted average — the soft-logits expected score. This sidesteps the model's tendency to output high, safe numbers in natural language. The theme scores are combined using the rubric weights into one pointwise reward, and every video is scored independently to avoid position bias in pairwise comparisons.

Stage 1 (seed-guided scorer warm-up) establishes a human-aligned sense of score margins before any dynamic rubric generation is attempted. For each dimension, a frontier MLLM (Gemini-3.1-Pro) runs an offline propose–verify–revise loop over 30 preference pairs: it drafts a rubric explaining why the preferred video wins, scores both videos with it, and revises on failure. Verified rubrics are deduplicated with a two-level Jaccard filter and reduced to five representative entries by an MCR² sampler. The scorer is then optimized with GRPO using a pairwise preference reward (a sigmoid over the score difference), a binary format reward, and a calibration loss that pulls the predicted score difference toward the human-annotated margin. Calibration is on relative margins rather than absolute scores, to avoid overfitting in the low-data setting.

Stage 2 (joint policy optimization) samples several rubrics per query, scores them by how well each separates the preferred video from the non-preferred one, and adds format and alignment rewards (alignment measured as cosine similarity between BGE-M3 embeddings of generated and seed themes). The generator is updated with the same clipped GRPO objective. Crucially, the two roles receive asymmetric signals: the scorer gets no direct policy-gradient update from the rubric-level reward — otherwise it could inflate reward by simply widening score gaps — and is instead trained only on the margin calibration loss, with the generated rubric treated as fixed text so gradients do not flow back through it. A scheduled coefficient activates scorer calibration after an initial generator warm-up.

Evaluation uses a pointwise set curated from EvalVerse (16 fine-grained secondary dimensions) and VideoGen-RewardBench (VGRB), described as a large-scale external pairwise preference benchmark of over 26.5K video pairs, with the Visual Quality split treated as seen and Text Alignment as unseen. Training runs on Qwen2.5-VL-7B.

Why This Matters

Impact on research: The paper reframes unstable scalar rewards as a representation problem rather than only a data or loss-function problem, and shows that a learned intermediate criterion can improve both pointwise correlation and pairwise agreement. The diagnostic vocabulary of "score-range collapse" and "high variance under context shift" gives a concrete way to talk about reward reliability for RL on generative media. The asymmetric optimization design — deliberately not backpropagating the rubric-level reward into the scorer — is a transferable trick for anyone building multi-role reward pipelines.

Real-world applications:

• Post-training pipelines for generative video products, where a stable reward is needed for RL fine-tuning without degrading unrelated capabilities.
• Automated quality triage for AI video generation platforms, replacing ad hoc scalar scoring with an interpretable per-theme breakdown.
• Preference-based A/B evaluation of video generation models, where pairwise ties and score drift currently obscure real differences.
• Interpretability and audit tooling, since the generated rubrics expose why a score was assigned rather than only what it was.

Industry relevance: The method's headline claim is extreme data efficiency — 30 preference pairs per dimension, 480 pairs total, versus large-scale annotation efforts — which matters for labs that have compute and models but limited high-quality human labels. The fact that the paradigm also improves a proprietary frontier model (Gemini-3.1-Pro) suggests it is not tied to one backbone.

Future Directions

• Closing the remaining calibration gap. Even the deployed rubric-plus-soft-logits configuration still deviates from human rating distributions, and adding rubrics to Gemini-3.1-Pro improved correlation while worsening bias — the paper explicitly leaves how to remove these intrinsic score-scale biases as an open problem.
• Beyond 30 pairs and 16 dimensions. Whether the approach holds at larger annotation budgets, and whether the 10-dimension subset supported by all compared reward models can be extended, is not established here.
• Extending the paradigm beyond video. The related work cited covers rubrics in LLM/agent domains and image generation, suggesting transfer to audio or other open-ended modalities where quality is subjective.
• Downstream RL robustness at scale. The downstream experiment covers one generator (Wan-2.2-A14B), one dimension (Visual Quality), and three metrics; the paper's claim of being "less prone to reward hacking" rests on this limited setting and invites broader reward-hacking stress tests.

Target Audience

Researchers and engineers working on reward modeling, RLHF/GRPO post-training, and multimodal evaluation — particularly those building reward signals for video generation. It also suits practitioners who need interpretable, data-efficient scoring pipelines, and readers interested in the general question of how explicit criteria change the behavior of large multimodal models on subjective judgment tasks. Beginners will follow the high-level argument but may find the training-objective and ablation details dense.

Authors’ abstract

Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.