PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?

AI summary

Overview

Research area: Computer vision / multimodal evaluation — specifically video-language model judges (reward models and LLM-as-a-judge) applied to ultra-long video.

Technical level: Intermediate. The paper is readable without specialized background, but it assumes familiarity with multimodal LLMs, retrieval, and preference-pair benchmarks.

Scope: The paper introduces PlaylistEval, an agentic pipeline that automatically builds a video-language judge benchmark over a roughly 100-hour playlist collection without human annotation, producing PlaylistBench (630 preference pairs across seven domains) and evaluating 17 models from eight families on it.

What This Paper Is About

Video-language models are now used as "judges" that score candidate answers against video evidence, both for evaluating systems and for training them via reward models. Existing judge benchmarks use clips of only a few minutes, let judges score from transcripts or text descriptions alone, and depend on costly human annotation that does not scale to long video. This paper asks whether judges stay reliable when the evidence is scattered across day-long, multi-video playlists, and builds an automated benchmark to test that.

Key Contributions

1. PlaylistEval, an agentic benchmark curator that builds video-language judge benchmarks from native video collections over a 100-hour playlist, with no human annotation, at roughly 1 USD per accepted question (352 USD total).
2. PlaylistBench, a benchmark of 630 preference pairs over 327 unique questions across seven domains (Education, Drama, Life, Art, History, Documentary, Podcasts), spanning both static factual and dynamic narrative knowledge, dominated by rating gaps of 1 and 2. On a stratified subset of 152 pairs it agrees with human judgments 93.0% of the time (inter-annotator agreement 0.781).
3. A controlled testbed with three explicit knobs — playlist size, input modality, and pair difficulty (set by the rating gap between answers) — that let the authors isolate where judges fail.
4. A systematic evaluation of 17 omnimodal and multimodal models from eight families, paired with four retrievers, revealing failures in length scaling, modality grounding, retrieval, and answer-order robustness.

Main Findings

• Judges fall far short of humans. The best judge reaches 75.4% pairwise accuracy, against 93.0% for humans. Small models sit near chance, and judges fine-tuned on short video fall to or below chance.
• Accuracy degrades with playlist length. Judge accuracy drops steadily from 1 hour to 100 hours of video.
• Retrieval helps only a little. Retrieval recovers up to +10.5 points, but the best retriever finds the right video segments only 37.9% of the time.
• Neither frames nor transcript alone suffice. Using frames or transcript alone costs 2–7 points against using both. More thinking or higher resolution barely helps, so the bottleneck is finding the evidence, not seeing it.
• Judges are order-sensitive. When the two answers swap sides, weaker judges (for example Gemma-4-26B-A4B and Gemini-3.5-Flash-Lite) reverse their verdict on roughly half of pairs, whereas stronger judges (for example Qwen-3.8-Max and Gemini-3.7-Flash) stay largely consistent.
• Retrieved frames beat uniform sampling for several models. In the paper's per-domain table, Gemini-3.7-Flash scores 75.4% overall with retrieved frames versus 71.0% with uniform sampling (+4.4), Gemini-3.1-Pro scores 70.2% versus 68.1% (+2.1), and Gemini-3.5-Flash-Lite scores 59.5% versus 54.9% (+4.6). Qwen-3.8-Max is 2.4T-A95B and Kimi-K2.6 is 1T-A32B.
• Difficulty is controllable. The two open-weight gate judges (Qwen3-VL-30B-A3B and InternVL3.5-8B) show accuracy rising monotonically with the rating gap between paired answers, confirming the framework generates pairs of graded difficulty.
• Automation is cheaper than annotation. Verifying the 152 human-checked pairs cost about 4.7 USD each, whereas the pipeline verified every pair at about 0.5 USD each — roughly 8 times cheaper.

Methodology in Plain English

Manual playlist collection (the only human step). The authors chose seven domains — Education, Drama, Life, Art, History, Documentary, and Podcasts — covering static knowledge, dynamic narrative, and mixtures. They searched YouTube playlists and curated a final collection of 29 playlists and 457 videos, about 4 playlists per domain on average, trimmed or extended until each domain covered about 100 hours. They removed private or deleted links, re-sorted videos whose order disagreed with episode keywords in their titles, and removed near-duplicate content.

Automatic indexing. Each video is split into 30-second chunks, transcribed with Qwen3-ASR-1.7B, and embedded into a single vector with Qwen3-VL-Embedding-8B using both sampled frames and transcript. Chunk embeddings are averaged over the 10- to 30-minute segment, the granularity at which evidence is grounded. Segment pairs with similarity above 0.95 are flagged as duplicates, and the gap is filled with newer videos until each domain again covers 100 hours.

Phase I: generating questions over scattered evidence. Same-domain segment pairs with embedding similarity in the range [0.40, 0.90] seed each question — close enough to share a topic, distinct enough to require both. Gemini-3-Flash sees both segments as native video at 0.5 fps and 720p with audio narration and produces a question, a 3- to 5-paragraph gold answer, and verification metadata in one call. The question names entities only by their visual surroundings rather than by name, and the gold answer cites every claim with its supporting span as (video-id @ MM:SS–MM:SS). Three gates verify each item, run cheapest first: a rule-based structural check (at least 3 paragraphs, citations covering 2–15 of the 30-second chunks per segment, i.e. 1.0 to 7.5 minutes of evidence); a video-necessity check that rejects questions answerable from transcript alone (tested by Gemini-3-Flash against shuffled transcript windows from another video, rejected by GPT-5.4-mini) or from model memory with no input (two generator–verifier pairs, Gemini-3-Flash with GPT-5.4 and GPT-5.4-mini with Gemini-3.1-Pro, rejecting only if both recover the gold); and a video-sufficiency check where Qwen-3.7-Plus receives each segment's full video at 0.5 fps with its transcript.

Phase II: generating distractors beyond the transcript. Gemini-3-Flash rewrites the gold answer into four wrong answers rated 4 down to 1, with the gold at 5, injecting visual-only errors (color, spatial layout, gesture, props, on-screen graphics) while preserving length, tone, and structure. Each degraded answer records which question-specific attributes it degrades and how, fixing an intended order of gold ≻ 4 ≻ 3 ≻ 2 ≻ 1. Three more gates apply: a structural check for a well-formed causal record; a textual-undetectability check where Gemini-3.1-Pro and GPT-5.4 each try to rank the shuffled answers from text alone and the set is rejected only if both recover the intended order; and a visual-detectability check where Qwen-3.7-Plus ranks the answers against the actual frames and the set passes only if it reproduces the order exactly. Generator and verifier always come from different model families to mitigate self-preference bias.

Self-correcting loop. Every rejection reason is appended to the generator's prompt as an instruction for the next attempt, so the pipeline corrects itself without human intervention and can be rerun on any new playlist. A Phase II initial rejection regenerates only the distractors, up to two more times, keeping the verified question fixed to avoid repeating costly Phase I video checks; after three Phase II rejections the cycle restarts from Phase I with the same segment pair. At most T = 6 attempts are allowed per seed and phase, after which the segment pair is discarded.

Pair selection and cost. Any two of the five answers form a preference pair. A difficulty gate using two deliberately weaker open-weight judges (Qwen3-VL-30B-A3B and InternVL3.5-8B) keeps only pairs that at least one of them fails, removing trivially easy pairs. Sampling 90 pairs per domain yields PlaylistBench. Free rule-based and cent-level text-only checks run first, so only survivors reach the two native-video models that account for over 90% of the bill.

Why This Matters

Impact on research. Prior video judge benchmarks — VideoJudge, VideoRewardBench, VURB — span clips of only a few minutes and rely on costly human annotation. In contrast to text benchmarks such as RewardBench, RewardBench 2, and JudgeBench, and image-text benchmarks such as VL-RewardBench, Multimodal RewardBench, and Multimodal RewardBench 2, PlaylistEval is described as the first fully automated, native-video judge benchmark. It shows that judge reliability, which is taken for granted at clip scale, is not established at day scale.

Real-world applications:

• Reward modeling for long-video training. Video MLLMs optimized against judge-provided rewards inherit the judge's errors; this work shows where those errors concentrate at long lengths.
• Evaluation of long-video question answering. Long-form answers grounded in hours of video need automated graders, and this benchmark measures how much those graders can be trusted.
• Retrieval system design for long video. The finding that the best retriever finds the right segments only 37.9% of the time is a direct signal for teams building video search and RAG pipelines.
• Cheap benchmark construction. The roughly 8-times cost reduction versus human annotation (about 0.5 USD versus 4.7 USD per pair) suggests a practical route to rebuilding benchmarks over new content collections.

Industry relevance. Because judge models are used both for ranking systems and for generating training signal, a judge that reverses its verdict on roughly half of pairs when answers swap sides translates into misranked systems and misguided updates. The paper's diagnosis — that the bottleneck is finding evidence rather than seeing it — points industry effort toward retrieval and multi-modal grounding rather than toward more thinking tokens or higher input resolution.

Future Directions

• Closing the retrieval gap. The best retriever locates the correct video segments only 37.9% of the time over 100-hour playlists, while retrieval itself recovers up to +10.5 points. Improving long-horizon retrieval appears to be the highest-leverage fix.
• Reducing position and order bias. Weaker judges reverse verdicts on roughly half of pairs under answer swapping; whether training, prompting, or architectural changes can make judges as order-stable as the strongest models is an open question.
• Extending beyond 100 hours. The paper measures degradation from 1 hour to 100 hours and beyond; whether the trend continues indefinitely or flattens is not established.
• Improving retriever–judge coupling. Only four retrievers were tested alongside the 17 judges; whether jointly trained or retrieval-aware judges outperform the current pairing is left open.

Target Audience

Researchers and engineers working on multimodal evaluation, video-language models, video reward modeling, and long-video retrieval. It is also relevant to practitioners building automated grading or ranking systems for video content, and to benchmark designers interested in fully automated, self-correcting data generation pipelines.

Authors’ abstract

Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether their judgments remain reliable when the evidence is buried in day-long videos has yet to be established. Existing benchmarks cannot answer this. Their videos are typically only a few minutes long, many answer pairs can be separated from the transcript alone, and collecting human judgments does not scale to ultra-long videos. We introduce PlaylistEval, an agentic framework that builds video-language judge benchmarks over 100-hour playlist collection without human annotation. It automatically generates questions with paired answers whose differences are controlled by causal degradation, so that every pair demands retrieval across the collection. The resulting benchmark contains 630 pairs across seven domains spanning both static and dynamic knowledge, and on a stratified subset of 152 pairs it agrees with human judgments 93.0% of the time (IAA 0.781). Evaluating 17 omnimodal and multimodal models from eight families reveals that frontier judges reach only 75.4% pairwise accuracy, while open-source judge models perform far behind. We further show that both retrieval and final judgment depend on using multiple modalities, and that judge accuracy degrades as the playlist set grows. We release our pipeline, benchmark, and evaluation code at https://playlisteval.github.io.