ReactionBench: A Human-Anchored Benchmark for Modeling Reactions to Video

AI summary

Overview

Research area: Multimodal video understanding and benchmark design, specifically whether vision-language models can connect an observed viewer's reaction to the video content that triggered it. Technical level: Intermediate — readers should be comfortable with standard evaluation vocabulary (accuracy, Spearman correlation, confidence intervals, LoRA adaptation), but no specialized mathematics is required. Scope (one sentence): The paper introduces ReactionBench, a human-anchored benchmark built from 31,834 synchronized stimulus–reaction clips (433:42:49 hours, eight domains), and uses it to measure how seven multimodal models rank reaction intensity, match a reaction to its stimulus, and explain the cues behind a reaction.

What This Paper Is About

Most video benchmarks test whether a model can describe events, reason about time, or read the emotions of the people shown on screen. ReactionBench asks a different question: given a recording of someone watching a video, can a model tell how strongly that person reacted at different moments, identify which part of the watched content they were reacting to, and explain which concrete cue drove the response? The benchmark pairs each recorded viewer reaction with the stimulus visible on screen at that moment, so the correct answer is fixed by the recording itself rather than by opinion.

Key Contributions

1. A three-task benchmark for reaction-specific video understanding. ReactionBench operationalizes the stimulus–reaction relation through Intensity Profiling (IP, ranking moments within one reaction video by intensity), Stimulus–Reaction Matching (SRM, choosing the temporally aligned stimulus from four same-video candidates), and Grounded Rationale Generation (GRG, writing a cue-based explanation scored against blind-written human references).
2. A human-anchored data pipeline. Construction starts from 31,834 synchronized stimulus–reaction clips spanning 433:42:49 hours across games, music, sports, live streams, movies and television, expert commentary, pop culture, and comedy. Human raters produced 14,894 ratings over 3,257 moments from 1,999 videos, and the frozen evaluation comprises 250 intensity profiles (642 moments), 112 matching items, and 200 rationale items carrying 447 human references.
3. A seven-model evaluation. GPT-5, Gemini 3.7 Flash, Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B, Qwen3.5-397B-A17B, and Kimi-K2.6 are evaluated zero-shot under a documented input protocol (1 FPS, maximum side of 512 pixels).
4. Diagnostic analyses of measurement itself. The paper reports how scoring organization (joint versus independent versus grouped-window intensity scoring), supervised LoRA adaptation of Qwen3.5-9B, and judge choice change the numbers that get reported.

Main Findings

• Matching is barely above chance for most models. Four-way SRM accuracy spans 0.259 to 0.509 against uniform-choice chance of 0.250. The best model, GPT-5, reaches 0.509; Gemini 3.7 Flash reaches 0.500; Qwen3.5-9B sits lowest at 0.259.
• GPT-5 shows no detectable sensitivity to which reaction it is shown. On 109 shared SRM items, accuracy is 0.422 with the normal input, 0.376 with stimulus options alone, and 0.385 when another viewer's reaction replaces the aligned one; both paired intervals include zero. On GRG (197 common items), the composite is 2.690 with the normal input, 1.959 without the reaction, and 2.746 with a different viewer's reaction — removing the reaction matters, substituting it does not.
• Rationale generation leaves headroom and is uneven across dimensions. GRG composites span 2.610 to 2.910 out of 5 per model (2.644 to 2.948 on the 191 items scored for all seven). No model leads every column: Gemini 3.7 Flash leads IP joint and reference consistency, GPT-5 leads SRM and cue, Qwen3.6-27B leads IP independent, and Kimi-K2.6 leads the GRG composite.
• Intensity ranking is weak and protocol-dependent. Against full human consensus, joint-scoring ρ₀ ranges from 0.026 to 0.176, while independent per-moment scoring ranges from 0.177 to 0.329. Independent scoring improves every one of the seven models by 0.139 to 0.226 in mean ρ₀, and every paired bootstrap interval excludes zero (largest gain: Qwen3.5-9B at +0.226).
• Score organization, not just context, carries part of that gap. With Qwen3.5-9B on 246 common videos, full-video joint scoring gives ρ₀ = 0.039, grouped-window joint scoring 0.172, and independent scoring 0.278. Grouped-window scoring also produces 0 of 246 constant profiles, versus 56 of 250 for independent scoring.
• Human agreement on intensity is only moderate. Ordinal Krippendorff's α = 0.357. Across 27,455 within-item rater pairs, 57.1% receive exactly the same score and 25.3% of items are unanimous; of one-point disagreements, 69% occur at 1–2, 24% at 2–3, and 7% at 0–1. The video-weighted individual-rater reference on pairwise ordering accuracy is 0.607, which the authors describe as agreement with a fixed rater pool rather than a human ceiling.
• Human–model gaps are not universal. Under a matched leave-one-rater-out protocol, Gemini 3.7 Flash reaches 0.585 joint pairwise accuracy against 0.607 for the held-out rater (−0.022, 95% CI [−0.081, 0.036]), and independent-input models span 0.587 to 0.661 on their matched sets. After adaptation, intensity-mixture seed 0 reaches 0.681 independent pairwise accuracy (0.074 above the reference, CI [0.031, 0.116]) and multi-task adaptation reaches 0.692 (0.085 above, CI [0.040, 0.128]).
• Supervised adaptation helps ranking but can collapse outputs. Multi-task LoRA on Qwen3.5-9B raises joint ρ₀ by 0.134 and independent ρ₀ by 0.140, yet produces 153/250 constant joint profiles; of the joint gain, 0.030 comes from replacing predictions with constants and 0.104 from better ranking on 97 nonconstant videos. Matching-only adaptation raises SRM accuracy from 0.304 to 0.411 (McNemar p = 0.036) mainly by removing 17 parse failures — the gain is 0.011 on the 95 base-parseable items (p = 1.000).
• The judge is validated on the target rubric. The shared text-only DeepSeek-V3.1 judge correlates with human means at Spearman ρ = 0.593 (95% bootstrap CI [0.43, 0.72]) on 84 anonymized generator–item pairs graded by three Prolific raters; mean judge–individual agreement is 0.447, close to mean inter-rater agreement of 0.441.
• An audio-inclusive control points to shortcut risk. In a Gemini 3.7 Flash native-video control with audio, replacing same-video distractors with other-video windows raises accuracy by +0.232 (McNemar p = 6×10⁻⁶), consistent with an audio rejection cue, though audio is not isolated from other changes in distractor source.

Methodology in Plain English

The researchers started from real online reaction videos, where a creator watches something and their face and voice are recorded alongside the content itself, usually in a picture-in-picture layout. Automated preprocessing inventories the files, finds usable viewing intervals, splits the stimulus region from the reactor region while keeping them time-aligned, and flags broken crops, frozen stimulus areas, or missing faces.

To avoid paying humans to rate every second of 433 hours of video, a model-based pre-screen nominates promising reaction moments. In a blind 50-clip pilot, 92.0% of nominations from the selected Gemini 3.6 Flash configuration received a human score of at least 2, versus 68.0% for a LibreFace-based selector; the selected configuration was then run over 14,582 non-pilot clips and produced 21,574 nominations. Model outputs are used only for nomination and never as labels or evaluation references.

Humans then did the parts that require judgment. Raters scored nominated moments on a behaviorally anchored 0–3 scale, with at least three ratings per moment. Separately, three annotators watched the stimulus with audio and a muted reaction view and wrote one-sentence explanations from a blank text box, producing 447 accepted references after quality review. Matching labels required no annotation at all: the correct option is whatever stimulus window was on screen at that moment, so the ground truth comes from the synchronization itself.

The three tasks are then scored in deliberately different ways. Intensity asks for 0–100 scores on marked moments and uses an extended Spearman correlation that assigns zero when a model returns a constant, since standard Spearman is undefined there. Matching is plain four-way accuracy, with incorrect selections also described by how far their start time sits from the aligned window. Rationale generation is graded by a text-only judge on cue quality, cue–reaction linking, and consistency with the human references, for a 0–5 composite.

Throughout, the paper reports Wilson intervals for accuracy and paired bootstrap intervals for mean differences, keeps the original Wilcoxon tests as secondary checks, and validates its LLM judge against human raters rather than assuming the judge is correct.

Why This Matters

Impact on research. The paper argues that benchmarks measuring what is in a video miss a distinct capability: relating what a person sees to how that person responds. It also shows that the reported number for a subjective target depends heavily on how the question is asked — a within-model gain of 0.139 to 0.226 in correlation from changing only the scoring organization — which suggests benchmark results in affective and preference-style evaluation should report elicitation format and constant-prediction rates.

Real-world applications:

• Content testing for film, streaming, games, and advertising, where creators want to know which moment in a trailer or clip actually produced a reaction rather than just whether the clip was liked.
• Automated editing and highlight selection, where the system needs to locate the stimulus moment responsible for a visible response rather than the loudest audio peak.
• Accessibility and media description, where an explanation linking a specific on-screen cue to a viewer response is more useful than an emotion label.
• Responsible moderation and platform analytics, provided outputs are treated as aggregate signals under the paper's stated limits; the authors explicitly exclude inferring private mental states, diagnosing emotion or health, deception detection, ranking people by expressiveness, and surveillance.

Industry relevance. Any team building or procuring video-capable models now has a public, frozen evaluation with released task files, prompts, predictions, and code, plus evidence that parameter count alone does not predict performance — Qwen3.6-27B exceeds Qwen3.5-397B-A17B on both IP and SRM. The finding that matching-only fine-tuning gains concentrate on format-failure items is directly actionable for teams interpreting their own fine-tuning gains.

Future Directions

1. Separate protocol effects from context effects. Grouped-window scoring recovers part of the joint-versus-independent gap for Qwen3.5-9B, but frame selection, context, instructions, and output organization still co-vary; a design that varies these one at a time would identify what actually drives the 0.139–0.226 improvement.
2. Address output collapse under supervision. Every intensity-trained adapter except joint-only keeps the independent-minus-joint gap at or above the base model's 0.230, and multi-task adaptation yields 153/250 constant joint profiles. Training objectives that improve ranking without erasing score distinctions are an open problem.
3. Remove modality shortcuts. The +0.232 accuracy jump when distractors come from other videos in the audio-inclusive control suggests models may use audio as a rejection cue rather than grounding the reaction; modality-matched controls and audio-ablated variants would isolate this.
4. Broaden the evaluated population and reaction types. The frozen intensity set contains seven creators from five channels, with 200 of 250 videos from a source category labeled Patreon and a cap of 50 videos per creator — narrower than the eight-domain collection. The pre-screen also favors strong visible or audible responses, so the benchmark characterizes selected creator reactions rather than ordinary viewing behavior or reactions the selector missed.

Target Audience

Researchers and practitioners building or evaluating multimodal video models, especially those working on affective computing, social signal processing, and human-centered video understanding. It is also valuable for benchmark designers interested in annotation pipelines, LLM-judge validation, and reporting standards for subjective targets, and for product teams in media, streaming, and advertising who need to measure viewer response to specific moments in content.

Authors’ abstract

Understanding what happens in a video is not sufficient for socially responsive multimodal systems: such systems also aim to relate what a person observes to how that person responds. Existing video benchmarks largely evaluate events, temporal reasoning, or people depicted in the video, while affective and viewer-response datasets often focus on emotion labels, aggregate responses, or prediction from the stimulus alone. This leaves an underexplored question: can current…