Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

AI summary

Overview

Research area: Multimodal AI alignment and reward modeling (Natural Language Processing / Multimodal Large Language Models).

Technical level: Intermediate to Advanced. The paper assumes familiarity with RLHF, preference optimization, and multimodal architectures, though the core ideas are explained with reasonable clarity.

Scope: Omni-RRM is a single reward model that evaluates and compares model responses across text, image, video, and audio using a shared five-criterion rubric, trained on a fully automated, teacher-annotated preference dataset rather than human labels.

What This Paper Is About

Multimodal large language models (MLLMs) need reward models to steer their outputs toward what humans actually want, but existing reward models are mostly built for images and video, depend on expensive human annotation, and collapse their judgment into a single opaque number. Omni-RRM addresses all three limitations at once: it covers four modalities, generates its own training data automatically, and produces structured, criterion-by-criterion justifications instead of a black-box score.

Key Contributions

1. Automatic rubric-grounded preference synthesis pipeline. A fully automated method that generates candidate response pairs by contrasting a strong model against a weaker one, then has two heterogeneous teacher models (GPT-4o-mini and Gemini-2.0-Flash) annotate each pair with scores, a verdict, and five criterion-level justifications, keeping only pairs where both teachers independently agree. This produces Omni-Preference, a 41.0k-sample dataset spanning image (17.0k), video (12.2k), and audio (11.8k).

2. Omni-RRM, an omni-modal reward model. A single 3B/7B model that supports text, image, video, and audio under a shared five-criterion rubric (Fluency & Coherence, Relevance, Accuracy & Completeness, Reasoning Quality, Safety & Ethical Alignment). Rather than emitting a scalar, it generates a structured record containing two scores in [0,10], a categorical verdict (A, B, or tie), and five comparative justifications before the final label.

3. Progressive SFT + GRPO training regimen. A two-stage recipe where supervised fine-tuning (via LoRA) first internalizes the rubric output schema, followed by Group Relative Policy Optimization using a composite reward that balances schema validity, preference correctness, and justification quality. This second stage is specifically designed to sharpen discrimination on low-margin "hard" preference pairs.

4. Extensive validation including inference-time alignment. Demonstrated state-of-the-art accuracy on video and audio benchmarks, Best-of-N reranking gains across three modalities, cross-modal transfer analysis, and text-only generalization.

Main Findings

• State-of-the-art omni-modal accuracy for open-source reward models. Omni-RRM-7B reaches 80.2% on ShareGPT-Video, 66.8% on Audio-HH-RLHF, and 65.0% on TA2T, with a five-benchmark Overall accuracy of 70.4%. This is a +17.0% relative improvement over its Qwen2.5-Omni-7B backbone and nearly matches Gemini-2.5-Pro (70.6%) despite being an order of magnitude smaller.

• Rubric-grounded rationales act as structural regularization, not decoration. Removing the five-criterion justifications during training drops the 7B model's four-benchmark Overall from 71.8% to 64.4%. Ablating the rubric reward term during GRPO (keeping the rationales but not optimizing for them) still lowers performance to 68.1%, showing the rubric must shape both the data and the optimization.

• RL refinement after SFT drives the largest gains. For the 7B model, Overall accuracy jumps from 62.6% (SFT only) to 70.4% (SFT+RL), with particularly steep improvements on MM-RewardBench (61.0% to 72.9%) and ShareGPT-Video (70.5% to 80.2%).

• Hard pairs benefit most. Difficulty-stratified evaluation shows average gains of +12.0 percentage points on low-margin pairs versus +6.6 points on easy pairs, confirming the design focus on nuanced discrimination works as intended.

• Specialized reward modeling beats raw scale. On MM-RewardBench, Gemini-2.5-Pro scores 63.3% while Omni-RRM-7B scores 72.9%, R1-Reward-7B scores 72.3%, and UnifiedReward-Think-7B scores 71.4%. Targeted supervision matters more than parameter count.

• Cross-modal training yields mutual reinforcement. Removing any single modality from training degrades performance on the others. Audio-only training reaches 63.8% on Audio-HH-RLHF, while training with all three modalities reaches 65.1% under an identical budget, evidence that vision and video preference structure provides a scaffold for the data-scarce audio domain.

• Effective as a plug-and-play reranker. Using Qwen2.5-Omni-7B as a fixed generator at N=10, Omni-RRM's Best-of-N selection beats self-consistency by +1.9 on MMMU (54.5 to 56.4), +1.7 on Video-MME (52.4 to 54.1), and +2.6 on AVQA (77.7 to 80.3).

Methodology in Plain English

The researchers build a reward model — a system that looks at a prompt and two candidate responses and decides which is better. Their approach has two halves: making training data, and training the model.

Making the data. Instead of paying humans to compare responses, they sample answers from a deliberately mismatched pair of models: a strong generator and a weak one. This "capability gap" strategy produces pairs that actually differ in quality, whereas sampling twice from the same model mostly produces unusable ties. Two different teacher models then score each pair across five shared criteria and give a verdict. Crucially, a pair only survives if both teachers independently agree on a non-tie winner and their numeric scores are consistent with their written reasoning. Pairs where teachers disagree or dilute too much get discarded. The score gap also lets them tag which examples are "hard" (margin under 2 points), which they later target during reinforcement learning.

Training the model. Stage one is standard supervised fine-tuning: the model is taught to produce the structured output — two numeric scores, five written justifications, and a final verdict — using LoRA adapters for efficiency. This gets the format locked in. Stage two uses GRPO, a reinforcement learning algorithm that samples several outputs per input and rewards the ones that do better relative to the group. Three reward components combine: a format reward (is the output valid structured data?), a preference reward (did it pick the right winner, and is the verdict consistent with the scores?), and a rubric reward (are the justifications substantive and consistent?). The rubric reward is the key piece — it forces the model to arrive at its verdict through disciplined criterion-by-criterion analysis rather than pattern-matching.

Evaluation. They test on established preference benchmarks covering image, video, and audio, plus a Best-of-N experiment where the reward model reranks candidate answers from a fixed generator, isolating the alignment effect from any change to the generator itself.

Why This Matters

Impact on research. This work reframes reward modeling from scalar regression to structured, auditable generation. The finding that a shared rubric induces positive cross-modal transfer — where video reasoning patterns help audio judgment — suggests that modality-specific reward models are leaving performance on the table. It also provides a reproducible, fully automated data pipeline that removes human annotation as the bottleneck for under-explored modalities. The public release of dataset, code, and checkpoints lowers the barrier for further work.

Real-world applications:

• Multimodal AI product quality control. Any system generating image captions, video summaries, or audio descriptions can use this as an automated grader that explains why one output beats another, which is far more actionable for engineers debugging failures than a single number.
• Inference-time response selection. The Best-of-N results show direct accuracy gains at deployment time without retraining the underlying generator — useful for applications where latency budget allows sampling multiple candidates.
• Low-resource modality bootstrapping. The audio results demonstrate that borrowing rubric structure from vision-heavy data can compensate for scarce audio preference data, relevant for teams working in speech or acoustic domains.
• Safety and compliance auditing. Because the rubric includes an explicit Safety & Ethical Alignment criterion with written justifications, the model's judgments can be inspected and challenged, which matters for regulated deployments requiring explainable decision-making.

Industry relevance. The efficiency angle is significant: a 7B open-source model matching a frontier proprietary model on preference judgment means teams can run reward modeling on-premises rather than paying per-call API costs, which matters for high-volume post-training loops and for organizations with data residency requirements.

Future Directions

• Extending beyond the fixed five-criterion rubric. The rubric is hand-designed and static. Whether criteria should be learned, task-adaptive, or domain-specific (e.g., medical or legal reasoning) remains open, as does the question of how sensitive performance is to rubric misspecification.

• Reducing inference overhead of structured rationales. The paper acknowledges that generating five justifications adds decoding latency versus scalar scoring. Whether distilled or shortened rationale formats can retain the discrimination benefit at lower cost is unexplored.

• Broadening real-world audio coverage. The primary audio benchmark (Audio-HH-RLHF) is constructed by applying text-to-speech to text prompts, so it lacks acoustic noise, speaker variability, and genuine environmental audio. A stronger claim about real-world audio would require evaluation on more naturalistic data.

• Tighter integration into the post-training loop. The work validates Omni-RRM for evaluation and reranking, but does not demonstrate its use as the reward signal inside a full RLHF or DPO pipeline for the generator itself. Whether the rubric-grounded signals translate into better policy optimization is an obvious next experiment.

Target Audience

Researchers and engineers working on multimodal alignment, RLHF/RLAIF pipelines, and reward modeling will find the most value here — particularly those interested in interpretable reward signals or in extending preference learning to under-resourced modalities like audio. Practitioners building evaluation and reranking systems for multimodal products will benefit from the practical Best-of-N results and the released checkpoints. Readers looking for a general introduction to reward modeling should first acquire background in preference optimization and RLHF, since the paper moves quickly past foundational concepts.

Authors’ abstract

Multimodal large language models (MLLMs) struggle with alignment due to the limitations of existing reward models (RMs), which are predominantly vision-centric, dependent on costly human labels, and provide opaque scalar scores that fail to capture nuanced reasoning, leading to brittle alignment. We present Omni-RRM, an \textbf{Omni}-modal \textbf{R}ubric-grounded \textbf{R}eward \textbf{M}odel that generates multi-dimensional reward signals across text, image, video, and audio. To overcome the high cost and inherent inconsistency of human-centric evaluation in multi-dimensional reasoning, we introduce \textbf{Omni-Preference}, a high-quality dataset constructed via automatic rubric-grounded preference synthesis. In this pipeline, teacher models reconcile raw preferences into explicit justifications, ensuring that the synthesized supervision is both high-fidelity and interpretable. Omni-RRM is trained using a progressive SFT + GRPO regimen, specifically optimized to sharpen reward discrimination on low-margin, hard preference pairs. It achieves state-of-the-art accuracy on video (80.2\% on ShareGPT-Video) and audio benchmarks (66.8\% on Audio-HH-RLHF and 65.0\% on TA2T), yielding a five-benchmark Overall accuracy of 70.4\% and a +17.0\% relative gain over its backbone. Furthermore, Omni-RRM effectively guides Best-of-$N$ selection and exhibits robust transfer to text-only alignment. All resources, including the dataset, training and inference code, and model checkpoints are available at https://tmfk418.github.io/Omni-RRM.