RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomy

AI summary

Overview

Research area: Computer vision and embodied AI — specifically benchmarking Social Proactive Intelligence (SPI), the ability of an embodied agent to decide whether a proactive action is socially appropriate, not just task-complete.

Technical level: Intermediate (readers should be comfortable with video-language models, benchmark design, classification metrics like Macro-F1, and the idea of world models).

Scope: The paper introduces RobotEQ-Video, a video-centric benchmark built on a four-level world-state taxonomy, and evaluates dozens of vision-language models against human annotators on the task of judging whether a candidate robot action is proper in a given video.

What This Paper Is About

Prior work on Social Proactive Intelligence judged whether a robot's proactive behavior was appropriate using static images, even though videos carry temporal and acoustic cues that images cannot. Existing benchmarks also collected data in a free-form way, so they could not guarantee coverage of the many contextual factors — age, place, time, relationship, emotional state — that determine social appropriateness.

This paper shifts SPI evaluation from images to videos and builds a hierarchical world-state taxonomy (6 domains, 20 dimensions, 142 level-1 attributes, 816 level-2 attributes) to systematically control and cover scenario diversity. Using that taxonomy, the authors assemble a benchmark of 2K+ videos with 16K+ labels for action properness and benchmark 48 representative models (12 closed-source, 36 open-source, as stated in the abstract and introduction; Section 4 describes "40+ representative models, including 10+ closed-source and 30+ open-source").

Key Contributions

1. Task extension: Extends Social Proactive Intelligence from static images to dynamic video, leveraging temporal and acoustic information for behavior judgment.
2. Taxonomy and dataset: Introduces a four-level world-state taxonomy of 6 domains, 20 dimensions, 142 level-1 attributes, and 816 level-2 attributes, yielding a dataset of 2K+ videos and 16K+ labels for action judgment.
3. Benchmark evaluation: Evaluates 48 representative models (12 closed-source, 36 open-source) and reports that models still lag behind human performance on video-centric SPI.
4. World-model exploration: Provides an initial attempt at using world models to predict the consequences of candidate actions and thereby assist social behavior judgment.

Main Findings

• Models trail humans: On the human baseline, the average across both subsets (A-Avg) is 86.9; the best model, GPT-5.6-Luna, reaches an A-Avg of 79.5. The gap holds on both subsets (human synthetic Avg 85.6 vs. GPT-5.6-Luna 79.1; human real Avg 88.3 vs. 79.9).
• Closed-source leads open-source: Closed-source models generally outperform open-source models, with GPT-5.6-Luna achieving the best overall performance, which the authors attribute to stronger reasoning and visual understanding capabilities.
• Metrics agree: Pairwise Pearson correlations among metrics (reported in Figure 5) show subjective-aware and subjective-unaware metrics are correlated, and synthetic and real subsets exhibit similar trends — taken as evidence of high synthetic-data quality.
• Video beats image beats text: Comparing human performance under Video+Action, Image+Action, and Action-only conditions on two disjoint subsets (Set A and Set B), the authors find images provide essential visual cues, while videos offer richer information than static images. (The specific scores appear in Figure 6 and are not stated numerically in the paper text.)
• Prompt-video consistency is strong: Synthetic videos generated from structured prompts (viewpoint constraints plus two-second storyboard content) show strong consistency with their prompts when judged by both models and human annotators.
• Fine-grained cues help: Qualitative attention visualizations (using Qwen3-VL-30B-A3B-Instruct as backbone) concentrate on decision-relevant regions such as a woman's hands interacting with a ladder, and a worker's facial expression while using a wrench on a pipe. Quantitative experiments show that adding fine-grained visual cues (object risk, event, place, expression) during inference improves performance.
• World models help: An initial experiment on a randomly sampled subset of 400 videos shows that a world-model-augmented solution — which predicts a continuation video from the last frame plus the candidate action, then judges properness using both observed and predicted video — improves accuracy over the baseline.
• Dataset composition: 2,036 videos total (1,218 synthetic, 818 real) and 16,850 actions (11,225 synthetic: 6,742 proper / 4,483 improper; 5,625 real: 3,342 proper / 2,283 improper). Videos concentrate around 6–10 seconds, and place distributions follow a long-tailed pattern dominated by daily locations such as residential, commercial, and transit spaces.

Methodology in Plain English

The authors first build a taxonomy of the contextual factors that could affect whether a robot action is socially appropriate, organized as a tree: 6 domains (event, time, place, relationship, human, object), refined into 20 dimensions, 142 level-1 attributes, and 816 level-2 attributes. Refinement used an AI-driven process with human review.

Video collection has two branches. Synthetic videos are generated with AIGC tools: the place dimension is weighted by its commonness profile, other dimensions are uniformly sampled, and an LLM selects five to nine naturally co-occurring dimensions and writes descriptions that are then rendered into robot-centric, naturally lit videos. A "self-evolving" loop has a VLM analyze why videos were discarded and feed lessons back into the description-generation prompts. Real videos come from source datasets chosen for place, event, and character diversity, segmented into clips matching synthetic durations, then filtered by VLMs to remove clips with frequent viewpoint changes or camera shake, clips lacking meaningful social interaction or context, and clips where the camera wearer's hands are visible (which breaks the robot-centric viewpoint). Manual checks yield 818 real videos.

Candidate actions are built by captioning each video, extracting dimensions from the caption, altering one or two of them to produce closely related but distinct captions, generating candidate actions from both captions, then filtering out irrelevant, semantically overlapping, or trivially judged actions.

Human annotation uses majority voting. Annotators are bachelor's and master's students familiar with embodied agents; to mitigate cultural bias the authors focus on videos featuring Chinese roles and use Chinese annotators. Each candidate annotator is screened on 20 randomly selected samples and dropped if their agreement falls below the group average, leaving a pool of eight annotators. Each video goes to four annotators, who independently decide whether the action is appropriate from the robot's perspective. Strong-consensus appropriate actions become positive labels, strong-consensus inappropriate actions become negative labels, and insufficient-agreement cases are excluded.

Evaluation uses two metric families. Subjective-unaware metrics treat the majority-voted label as ground truth and use Macro-F1 (Mac) as the primary metric with Accuracy (Acc) as secondary, because of class imbalance. Subjective-aware metrics compare the model against each individual annotator: Probability of Agreement (PA) measures expected agreement with annotators, and Consensus-Weighted Agreement (CW) normalizes PA by the fraction of annotators who agreed, penalizing errors more on high-consensus samples. Each model receives a video plus candidate actions and outputs a properness judgment.

Why This Matters

Impact on research: The paper argues that social properness is tightly coupled to contextual clues (human age, occupation, emotional state, place, time, event), and that free-form data collection cannot systematically cover them. Providing a structured taxonomy with 142 level-1 and 816 level-2 attributes gives future work a controllable way to generate and audit coverage, and the move from images to video establishes temporal and acoustic evidence as part of the SPI task definition.

Real-world applications:

• Home and eldercare robots deciding whether to intervene when a person is struggling with a task, versus staying out of the way.
• Service and transit robots in crowded public spaces such as subway stations, airports, and elevators, where proximity, queueing, and privacy norms govern whether an approach is welcome.
• Workplace and industrial assistants on construction sites, farms, or maintenance areas, where safety cues and the worker's ongoing task determine whether proactive help is appropriate.
• Wearable and in-ear assistants that must decide when to speak up, a direction the paper notes existing systems such as LLAMAPIE and PROASSIST pursue from a human-centric rather than a robot-centric viewpoint.

Industry relevance: The benchmark gives a common yardstick for comparing vision-language models and embodied agent stacks on a socially grounded judgment task. The result that closed-source reasoning models outperform open-source ones, and that no model reaches the human baseline, sets a clear performance target for teams building proactive robotics products.

Future Directions

• Broaden model coverage: The authors note the benchmark may not include recent, more advanced models given how quickly the field moves, and plan to widen evaluation scope.
• Refine the taxonomy: Despite 816 level-2 attributes, some attributes may remain uncovered; the authors plan further refinement.
• Handle class imbalance and task difficulty: Proper and improper actions are imbalanced in the dataset, which is why Macro-F1 is the primary metric — but improving calibration on minority classes remains open.
• Cross-cultural generalization: The current benchmark only includes videos with Chinese roles and Chinese annotators to reduce cultural bias, so the authors plan to study cross-cultural impacts on SPI.
• Longer videos: The dataset centers on mid-length videos of roughly 6–10 seconds, and the authors leave long-duration video to future research.
• Scale up world-model augmentation: The world-model experiment covers only 400 randomly sampled videos because of the high cost of calling the world-model API for video generation, leaving scaling and deeper integration unexplored.

Target Audience

Researchers working on embodied AI, proactive assistance, and robot social intelligence; benchmark builders in computer vision who need a model for taxonomy-driven, coverage-controlled dataset construction; and practitioners evaluating or fine-tuning vision-language and video-language models for socially aware decision-making. Readers interested in annotation methodology — majority voting, per-annotator agreement metrics, and agreement-weighted evaluation — will also find the setup instructive.

Authors’ abstract

Social Proactive Intelligence (SPI) extends proactive assistance beyond task completeness to consider social appropriateness in diverse embodied scenarios. However, prior SPI research faces two key limitations. First, existing work focuses on static images, whereas dynamic videos provide crucial cues for inferring human states and needs, offering richer information than isolated images. Second, prior work often relies on free-form data collection pipelines, which fail to guarantee comprehensive coverage of diverse scenarios. To address these gaps, we introduce RobotEQ-Video, shifting the focus from image-centric to video-centric analysis. To ensure comprehensive video coverage, we construct a hierarchical world-state taxonomy organized into a four-level coarse-to-fine structure, comprising 6 domains, 20 dimensions, 142 level-1 attributes, and 816 level-2 attributes. The resulting benchmark comprises 2K+ videos with 100K+ human annotations and 16K+ labels for assessing behavior properness. Benchmark evaluation reveals that current systems remain unreliable and fall short of human performance. We further explore how world models can help tackle this task. This work advances SPI research from static images to dynamic videos and ensures more comprehensive scenario coverage during benchmarking.