Social Caption: Evaluating Social Understanding in Multimodal Models

AI summary

Overview

• Research area: Multimodal large language models (MLLMs), social intelligence, and evaluation methodology in natural language processing.
• Technical level: Advanced. The paper combines an interaction-theory-grounded rubric design (APRACE taxonomy), mixed-effects statistical modeling of human annotations, paired significance testing with Holm correction, and human–judge agreement analyses.
• One-sentence scope: The paper introduces Social Caption, a framework for evaluating MLLMs along three social understanding dimensions—Social Inference (SI), Holistic Social Analysis (HSA), and Directed Social Analysis (DSA)—and uses it to benchmark 12 open- and closed-source models while testing whether MLLM judges can replace human annotators.

What This Paper Is About

Existing evaluations of social understanding in MLLMs are dominated by video question answering (QA), which only measures a model's ability to pick a correct answer from given options. The authors argue that real social understanding also requires generating holistic interpretations of a scene (inferring dynamics from body language, setting, and relationships) and directed interpretations that distill only the social information relevant to a specific question. Social Caption is their proposed framework for measuring all three capabilities and for testing whether MLLM judges can scale that evaluation.

Key Contributions

1. A new three-dimensional evaluation framework. Social Caption separates Social Inference (SI), Holistic Social Analysis (HSA), and Directed Social Analysis (DSA), moving beyond accuracy-on-multiple-choice as the sole proxy for social reasoning.
2. Theory-grounded rubrics. The HSA and DSA criteria are built on the APRACE taxonomy, which the paper describes as derived from grounded theory analysis of 5,000+ open-ended interaction descriptions, mapping to Actors and Partners, Relations, Activities, Context, and Evaluation. Each dimension uses six sub-dimensions rated on a 1–5 Likert scale for a maximum score of 30.
3. Analysis of what drives MLLM social understanding. The paper studies scale, architectural design, and the presence of spoken context (WhisperX transcriptions) as factors influencing performance, including a controlled size analysis for InternVL3 (up to 78B parameters) and Qwen2-VL (up to 72B parameters).
4. First study of MLLM judges for multimodal social understanding. Three judges—Gemini-2.5-Pro, InternVL3-8B, and InternVL3-78B—are compared against human ratings using binary F1 agreement, alongside a mismatch test and a self-preference analysis.

Main Findings

• Top closed-source model leads on all three dimensions. Gemini-2.5-Pro achieved the highest SI accuracy (82.08% with transcriptions), the highest HSA score (26.14/30), and the highest DSA score (25.94/30). It significantly outperformed all models on SI except GPT-4o (p_adj = 0.19), and significantly outperformed Gemini-1.5-Pro (p = 0.020) and GPT-4o (p < 0.001) on overall HSA.
• No model beat the human layperson on SI. Human layperson SI accuracy is reported as approximately 85%, above every model tested.
• Spoken context consistently helps. Providing transcriptions improved SI for all models, with accuracy gains up to 13%. Gemini-1.5-Pro and Gemini-2.5-Pro showed only marginal increases with transcripts, suggesting stronger use of non-verbal visual cues. A text-only Qwen2-7B-Instruct baseline reached 64.64%.
• Video pre-training was not a prerequisite. InternVL3 and Qwen2-VL, the highest-performing standard-scale models on SI, both perform video fine-tuning without video pre-training. LongVA (65.43% with transcriptions) did not significantly outperform the text-only baseline sharing its backbone (64.64%, p_adj = 0.25), while Qwen2-VL significantly outperformed both the baseline (p_adj < 0.05) and LongVA (p_adj < 0.01).
• Scale helps, but architecture and backbone priors matter too. InternVL3-1B and Qwen2-VL-2B scored roughly 20% lower than the largest variants in their series. InternVL3-9B, which is larger than InternVL3-8B, scored lower on SI; the paper notes it is the only InternVL3 model using an InternLM3 backbone rather than Qwen2.5. No significant difference was found between InternVL3-78B and GPT-4o (p_adj ≥ 1.00) or Gemini-1.5-Pro (p_adj ≥ 0.56), nor between Qwen2-VL-72B and GPT-4o (p_adj ≥ 1.00) or Gemini-1.5-Pro (p_adj ≥ 0.25). Both open models came within roughly 3% of Gemini-2.5-Pro on SI.
• Open-source models match closed-source models on HSA and DSA. Qwen2.5-VL scored the highest open-source HSA (23.24/30), followed closely by InternVL2 (23.13/30), and Qwen2.5-VL was statistically indistinguishable from GPT-4o (22.85/30). For DSA, MiniCPM-V2.6 (23.51/30), Qwen2.5-VL (23.49/30), and InternVL3-8B (23.01/30) were strongest open-source models and statistically indistinguishable from one another, with no detected significant differences against GPT-4o or Gemini-1.5-Pro overall.
• Strong SI does not imply strong HSA or DSA. Qwen2.5-Omni scored 68.42% on SI and shares the Qwen2.5-7B backbone with Qwen2.5-VL, yet scored far lower on HSA (14.84/30 vs 23.24/30) and DSA (8.91/30 vs 23.49/30). The authors link this to its low-latency streaming architecture.
• Sub-dimension profiles differ by model. Gemini-2.5-Pro achieved the highest human ratings on all six HSA sub-dimensions and all six DSA sub-dimensions. Among open-source models, Qwen2.5-VL was strongest on Scene Description, Topic and Context, and Answer Detail in HSA, and on Relevant Scene Details, Answer Detail, and Adherence to Prompt in DSA; InternVL2 was strongest on Individuals, Topic and Context (tied), and Adherence to Prompt in HSA; MiniCPM-V2.6 was strongest on Socio-Emotional in HSA and on Relevant Individuals, Relevant Interactions, and Relevant Context in DSA.
• Human annotation was reliable. Inter-annotator agreement using binary F1 was 86.60% for HSA and 84.87% for DSA. The study collected 12,000 total human ratings.
• MLLM judges align well with humans but show bias. Human–judge F1 ranged from 78.81–94.12% across HSA sub-dimensions and 75.24–92.23% across DSA sub-dimensions. For every sub-dimension, at least one MLLM judge aligned with human ratings better than human–human agreement did. Most judges assigned higher scores than humans and showed mild self-preference; the authors recommend treating judge scores as relative rather than absolute signals. Mismatch scores stayed low across all judges and both dimensions.
• Judge choice affects which sub-dimension is measured best. InternVL3-78B aligned most strongly with humans on HSA Scene Description, Individuals, Topic and Context, Socio-Emotional Analysis, and Answer Detail, and on DSA Relevant Individuals, Relevant Interactions, and Relevant Context. Gemini-2.5-Pro aligned most strongly on Adherence to Prompt in both dimensions plus DSA Relevant Scene Details. InternVL3-8B aligned most strongly on DSA Answer Detail.

Methodology in Plain English

The researchers instantiate Social Caption on an existing testbed, the Social-IQ 2.0 dataset (about 1 minute per video, 145 videos, 943 questions, 4 answer options per question in the validation set). They run 12 MLLMs capable of processing video: nine open-source models (LongVA, Oryx-1.5, LLaVA-NeXT-Video, MiniCPM-V2.6, Qwen2-VL, Qwen2.5-VL, Qwen2.5-Omni, InternVL2, InternVL3) and three closed-source models (GPT-4o, Gemini-1.5-Pro, Gemini-2.5-Pro).

Videos are processed at 1 frame per second. Because most open-source models did not support audio input, audio was omitted and WhisperX transcriptions with timestamps were provided instead. The authors used 7B versions of open-source models where available; three models only had 8B parameters available, and models in this 7–8B range are called "standard-scale."

For SI, each model ran 3 experiments without transcriptions and 3 with, and comparisons used paired tests with Holm correction. For the more expensive HSA and DSA dimensions, human annotations were collected on the top 10 models by SI performance, over a randomly sampled 50-video subset matched to the full validation set on SI difficulty, question type, number of people per scene, and dialogue density.

Each HSA or DSA generation was rated by two English-speaking annotators in the United States with at least a high school education, recruited through Prolific under an IRB-approved process and paid $15/hr (prorated). A paired power analysis indicated that N=50 provides 80% power to detect effects of approximately d_z = 0.40. Because each video was annotated by a different pair of annotators, agreement was measured with a binary F1 score, and mixed-effects models were fit with model identity as a fixed effect and video as a random intercept.

For the MLLM judge study, the three judges received the same evaluation instructions as human annotators. Gemini-2.5-Pro and InternVL3-8B were also candidate models, which allowed measurement of self-preference; InternVL3-78B served only as a judge and was excluded from the candidate pool.

Why This Matters

Impact on research. Social Caption reframes how multimodal social understanding should be measured, arguing that multiple-choice accuracy captures only one of three capabilities. It also provides initial evidence that MLLM judges can align with human ratings on a subjective, theory-grounded social task, opening a route to evaluation at a scale human annotation cannot reach. The finding that strong SI does not predict strong HSA or DSA gives benchmark designers a reason to report more than accuracy.

Real-world applications (as motivated by the paper's stated context of assistive robotics and deployment in social interaction domains):

• Assistive robotics, where models must interpret intent, behavior, and interpersonal dynamics to operate effectively alongside people.
• Human-facing AI systems intended to support health, well-being, dignity, and autonomy.
• Automated evaluation pipelines for model developers, using MLLM judges as relative scoring signals for open-ended social outputs.
• Diagnostic analysis of model design choices, such as how streaming or low-latency architectures affect the ability to produce coherent social descriptions.

Industry relevance. For teams building or post-training multimodal models, the paper offers a reusable, dataset-agnostic rubric structure and evidence that careful architecture and post-training decisions let 7–8B open-source models approach or match much larger closed-source models on HSA and DSA. The warning that judge scores are most useful as relative rather than absolute signals is directly relevant to anyone using MLLM judges for reward signals.

Future Directions

• Long-context social understanding. All videos in this work are 1 minute long, a design choice rather than a technical limitation; the authors explicitly call for curating longer-form social intelligence video datasets.
• Broader cultural and linguistic coverage. The current study uses English-language interactions and annotators in the United States, and the authors motivate future work applying Social Caption to more diverse, multilingual, and multicultural video data.
• Architecture and streaming effects. The gap between Qwen2.5-Omni and Qwen2.5-VL motivates studying how architecture and streaming algorithms affect HSA and DSA, beyond what the shared LLM backbone explains.
• Scaling and validating automated evaluation. Judge choice changes which sub-dimensions are evaluated well, and judges show mild self-preference, leaving open how to calibrate MLLM judges as reliable absolute scoring instruments.

Target Audience

This paper is most useful to multimodal NLP and computer vision researchers building or benchmarking video-language models, evaluation researchers designing rubrics and automated judge pipelines, and practitioners in social robotics or human-facing AI who need fine-grained measures of social understanding rather than QA accuracy alone. It also speaks to researchers interested in applying social interaction theory (the APRACE taxonomy, Goffman, Hymes) to machine learning evaluation, and to anyone who needs to decide whether MLLM judges can replace human annotation on subjective social tasks.

Authors’ abstract

Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce SOCIAL CAPTION, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to generate relevant information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges demonstrate a path towards scaling automated evaluation of multimodal social understanding.