RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

AI summary

Overview

Research area: Computer Vision / Embodied AI — specifically, the reliability of multimodal large language models (MLLMs) used as automatic judges of robot-manipulation video generation.

Technical level: Advanced. The paper assumes familiarity with video generation models, MLLM-as-a-judge evaluation, Pearson correlation as an agreement metric, and reinforcement learning from verifiable rewards (GRPO).

Scope in one sentence: The paper builds a human-annotated benchmark (RoboJudge-Bench) that measures how well 29 model judges and 8 non-MLLM metrics agree with human ratings of embodied videos, then trains a dedicated 9B judge (RoboJudge-9B) that outperforms all evaluated baselines.

What This Paper Is About

Video generators are being proposed as "world models" for robots, but their outputs are only useful if the motion obeys physics and the instructions are actually carried out. Researchers increasingly use MLLMs to score such videos automatically, yet nobody had systematically checked whether those MLLM judges agree with human raters on embodied manipulation footage.

This paper fills that gap: it measures judge–human agreement on 800 human-annotated embodied videos along two axes (Physical Adherence and Instruction Alignment), shows that existing judges are uneven and often miss physical failures, and then trains a specialized 9B judge on 12K videos to close the gap.

Key Contributions

1. RoboJudge-Bench: A benchmark of 800 human-annotated generated rollouts drawn from 8 source corpora and 10 video generators, spanning robot grippers, dexterous humanoid hands, and human hands in both simulated (131) and real (669) settings, rated on 1–5 scales for Physical Adherence and Instruction Alignment.

2. A systematic audit of existing judges: Comparison against 29 model-judge baselines (10 closed-source MLLMs, 7 open-weight MLLMs, 7 specialized embodied MLLMs, 5 specialized judge models) plus 8 non-MLLM metrics, with failure analysis showing errors concentrated on fine-grained interaction errors and physical hallucinations.

3. RoboJudge-Train: A training corpus of 12,351 videos (11,520 generated from 14 generators plus 831 verified successful real demonstrations), annotated with 24,702 axis-level samples carrying human-verified scores and model-assisted rationales.

4. RoboJudge-9B: A 9B judge fine-tuned from Qwen3.5-9B via supervised fine-tuning followed by reinforcement learning (GRPO), which achieves the highest overall agreement among all evaluated model judges and transfers to external benchmarks.

Main Findings

• Existing judges are much better at task alignment than at physics: Before task-specific training, the best Instruction Alignment correlation was r = 0.733 (GPT-5.5), versus only r = 0.611 for the best Physical Adherence correlation (Gemini-3.7-Flash).

• Frontier proprietary models lead but still fall short: GPT-5.5 reaches r = 0.49 on Physical Adherence; Gemini-3.7-Flash is the strongest proprietary baseline overall at r = 0.630.

• Specialized video judges do not transfer to embodied scenarios: They reach at most r = 0.42 on Physical Adherence and 0.59 on Instruction Alignment. PhyJudge is the strongest of this group at r = 0.417 on PA, still well below RoboJudge-9B's 0.645.

• Embodied VLMs are weak judges: Specialized embodied MLLMs stay at r ≤ 0.261 on Physical Adherence, and RoboBrain2.5-8B and RynnBrain-1.1-9B are near random (r ≤ 0.07).

• Non-MLLM metrics are poor proxies for physical plausibility: MUSIQ correlates at r = 0.078 and pixel consistency at 0.155; DINO and V-JEPA Paired Similarity reach r = 0.392 and 0.370, both below RoboJudge-9B.

• RoboJudge-9B leads the field: Overall Pearson r = 0.719 (PA 0.645, IA 0.775), overall exact-match accuracy 0.473 and MSE 0.747, compared with 0.401 overall for the Qwen3.5-9B base model. The 0.089 gain over Gemini-3.7-Flash is reported as statistically significant via paired cluster bootstrap.

• Both training stages contribute: SFT alone lifts overall r from 0.401 to 0.700; adding RL raises it to 0.719 (PA 0.624 → 0.645, IA 0.766 → 0.775, overall accuracy 0.438 → 0.473).

• Gains transfer out of distribution: On VideoPhy2, SFT+RL reaches r = 0.346 (PC) and 0.572 (SA), versus 0.179 and 0.412 for the base model. On Physion-Eval, Youden's J rises from 0.056 (base) to 0.105 (SFT) and 0.093 (SFT+RL).

• Frame rate is not the bottleneck: Varying sampling at 2, 4, and 8 FPS leaves GPT-5.5 at 0.620, 0.618, 0.620 overall; Qwen3.5-9B improves from 0.363 at 2 FPS to 0.401 at 4 FPS with no further gain at 8 FPS, so 4 FPS is the default.

• Physical and instruction quality dissociate across generators: In a downstream ranking of 17 video generation models on 50 shared prompts, LingBot Video MoE Base, LongCat, and SVD-XT score relatively high on PA but much lower on IA, while HappyHorse and Wan2.6 Flash score highly on both.

Methodology in Plain English

Building the test set. The team started from real and simulated manipulation recordings — robot corpora such as AgiBot-World, DROID, GR1-InLab and Open X-Embodiment, human-hand corpora such as EgoDex, EgoScale and EPIC-KITCHENS, and the RoboTwin simulator. From each clip they took an initial frame and a task instruction, refined under-specified instructions with GPT-5.5 and had humans verify them, then generated ten prompt variants per clip using five prefix templates and five rewriting templates. Fourteen generators produced the pool; 800 videos from 10 of them form the benchmark.

Defining what "good" means. Each video is scored 1–5 on two axes. Physical Adherence is judged without seeing the instruction and breaks into agent consistency, scene consistency, and interaction realism. Instruction Alignment is judged with the instruction and first frame, and breaks into agent match, object correctness, and goal completion. This separation means a video can be physically flawless while completing the wrong task (PA 4, IA 2 in the paper's example) or vice versa (PA 3, IA 5).

Human labeling. Twenty graduate-level annotators went through four calibration rounds on 100 practice videos each; exact agreement climbed from 32.8% to 52.9% and within-one-point agreement from 76.2% to 92.1%. For the test set, one annotator labels an item and a second cross-reviews it, with an expert adjudicating the final ground truth. Generator identity is hidden during labeling.

Training data at scale. Because expert-adjudicated labels are expensive, the 12,351-video training set uses a human-in-the-loop shortcut: three closed-source MLLMs (Gemini-3.5-Flash, GPT-5.5, Seed-2.1-Lite) propose scores, humans review and correct them and write evidence notes, and GPT-5.5 rewrites those notes into structured reasoning traces conditioned on the corrected scores. Training clips are deliberately disjoint from benchmark clips to avoid leakage.

Training the judge. Starting from Qwen3.5-9B, they do one epoch of full-parameter supervised fine-tuning, then 200 steps of GRPO with a simple binary reward: 1 if the predicted integer score exactly matches the reference, 0 otherwise.

Evaluation. Judges are scored by Pearson correlation with human ratings (primary), exact-match accuracy, and mean squared error. General and embodied MLLMs share a unified prompt with 4 FPS sampling and at most 32 frames; specialized judges run with their own native prompts and score ranges.

Why This Matters

Impact on research. The paper reframes embodied video evaluation as a question about the judge, not just the generator. Its evidence that general MLLMs, embodied VLMs, and general-domain physical judges all mis-score manipulation videos means results reported by prior automatic evaluators on embodied benchmarks should be read with caution. The PA/IA split also gives the field a way to report two genuinely independent failure modes rather than one blended score.

Real-world applications:

• Robot policy and world-model evaluation: Using a human-aligned judge to rank or filter generated rollouts without collecting fresh human ratings each time.
• Planning via Best-of-N selection: Scoring candidate imagined rollouts online and picking the most physically plausible and instruction-consistent one before acting.
• Training-data curation for video generators: Automatically filtering generated manipulation clips for physical violations such as interpenetration, vanishing objects, or extra hands.
• Quality assurance for embodied video products: Auditing generators on separate physical-realism and task-completion axes, since models like LingBot Video MoE Base, LongCat, and SVD-XT look strong on one and weak on the other.

Industry relevance. Teams building robot foundation models, world models, or manipulation video generators need cheap, trustworthy automatic evaluation; a 9B open-weight judge that beats Gemini-3.7-Flash on this task (r = 0.719 vs 0.630) is a practical substitute for both expensive proprietary calls and human labeling.

Future Directions

• Online rollout evaluation: The paper suggests RoboJudge-9B's scores could guide Best-of-N physical rollout selection during embodied planning, but this is framed as potential rather than demonstrated; testing it inside a closed planning loop is an open step.

• Agentic embodied video generation: The authors point to using the judge as a quality signal inside agentic generation pipelines, which would require the judge to remain calibrated under distribution shift from its training pool.

• Broadening embodiment and generator coverage: The benchmark covers 8 corpora and 10 generators, and training adds LIBERO and DreamDojo-HV; whether agreement holds for embodiments and generators outside this pool is not established.

• Closing the residual physics gap: Even the best model reaches PA r = 0.645 versus IA r = 0.775, and the failure analysis shows judges still overlook duplicated entities, extra hands, and incorrect final states — the specific error types that need targeted supervision next.

Target Audience

Researchers and engineers working on embodied world models, robot manipulation video generation, and MLLM-as-a-judge evaluation. It is most useful for those who need to select or build an automatic evaluator for generated robot video, and for anyone who has reported generator rankings produced by an unvalidated MLLM judge. Readers without background in video generation or correlation-based agreement metrics will find the experimental sections dense, though the dimension definitions and failure figures are accessible on their own.

Authors’ abstract

Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.