SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically multimodal sports understanding, rule-based visual reasoning, and benchmark construction.

Technical level: Intermediate. The benchmark design, question hierarchy, and evaluation metrics are approachable for readers with general machine-learning familiarity, while the reinforcement-learning training details (GRPO) and reward design require more background.

Scope: This paper introduces SportR, a multi-sport, multimodal benchmark of 4,789 images and 2,052 videos with 6,841 human-authored Chain-of-Thought rationales, built to train and evaluate fine-grained rule-based reasoning and visual grounding in MLLMs.

What This Paper Is About

Understanding sports deeply requires more than recognizing that a game is being played; a model must notice fine visual details, apply abstract rule knowledge, and tie that knowledge back to specific visual evidence. Existing sports benchmarks either cover a single sport or lack the detailed reasoning chains and precise visual grounding needed to test these capabilities across multiple sports. SportR addresses this by providing a large-scale, multi-sport benchmark with a progressive question hierarchy, human-written reasoning traces, and a bounding-box grounding task, then using it both to measure and to improve MLLMs.

Key Contributions

1. A progressive QA hierarchy for granular evaluation. The benchmark organizes questions into 13 levels spanning Q1–Q7 for images (infraction identification, foul classification, penalty prediction, free-form explanation, visual grounding, offensive tactic identification, defensive tactic identification) and Q8–Q13 mirroring these in the temporal domain for video. The authors also provide fully human-annotated Chain-of-Thought traces as a gold standard for the most challenging tasks.

2. The first multi-sport benchmark with an explicit visual grounding task. Models must output the precise bounding box of a rule infraction in an image, directly testing whether abstract rule knowledge is connected to precise visual evidence.

3. A demonstration that SportR works as both a training resource and a challenging evaluation set. Supervised Fine-Tuning and GRPO-based Reinforcement Learning produce clear gains, and a model trained only on image data improves on unseen video tasks.

4. A human-authored, expert-verified dataset. All 6,841 CoT annotations and all labels were written by the paper's 16 authors, including two former NCAA Division I student-athletes with over 12 years of competitive experience and 14 members with at least three years of dedicated training in their sports. No model-assisted generation was used for annotations.

Main Findings

• Baselines perform poorly on the hardest tasks. Across all zero-shot baselines, IoU scores on the Explicit Visual Grounding task (Q5) are consistently below 7%. Even GPT-5, the strongest proprietary model evaluated, reaches only 5.70% IoU on Q5 and 44.21% on Foul Classification (Q2).

• Training on SportsImage yields large gains. For Qwen2.5-VL-7B, Foul Classification (Q2) accuracy moves from 14.43% at base to 50.71% after SFT. After SFT+RL, scores reach 84.19% on Q1, 51.54% on Q2, 52.34% on Q3, 60.89% on Q6, and 87.07% on Q7, the highest on 5 of the 7 categories.

• Visual grounding remains extremely difficult even after tuning. The average IoU on the grounding task improves only from 4.61% to 9.94%, which the authors highlight as evidence of the benchmark's difficulty.

• Reasoning skills generalize across modalities. Qwen2.5-VL-7B (SFT+RL), trained exclusively on SportsImage, raises Video Infraction Identification (Q8) from 25.49% to 59.52%, surpassing all other models including GPT-5, though performance drops on Q12 (offensive tactic identification in video).

• Video tasks are harder than image tasks. Overall model performance on SportsVideo is lower than on SportsImage. GPT-5 achieves the highest scores on the majority of video tasks (Q9–Q12) while still scoring relatively low: 34.39% on Q9, 41.83% on Q10, 24.02% on Q11, and 60.82% on Q12.

• Error analysis points to perception failures in video and knowledge gaps in images. In a manual review of 1,500 failure cases sampled across six representative models, video errors were dominated by Visual Perception Error and Visual Hallucination, which combined frequently exceeded 60–70% of total errors. In static images, Domain Knowledge Gap rose sharply; GPT-5's Domain Knowledge Gap increased from 20.00% in video to 36.00% in images.

• LLM-as-Judge scores align with human judgment. A human verification study on a stratified subset of 660 samples found a Pearson correlation above 0.65 between the LLM average score and expert human judgments across modalities.

Methodology in Plain English

The authors first defined a three-level "pyramid" of sports understanding: basic perception (already near-solved by current models), fundamental fouls and tactics (their focus), and elite professional-level edge cases (out of scope). They then built a 13-question ladder over that middle layer, from "is there a foul?" through "what penalty applies?" to "where exactly is the illegal contact?"

Data was collected for five sports — basketball, soccer, table tennis, badminton, and American football — yielding 4,789 images covering all five sports and 2,052 videos covering four (badminton excluded). Annotators followed a "Macro-to-Micro" reasoning flow: identify the court area and involved parties, describe the action leading to the event, then pinpoint the precise point of contact. Every annotator self-reviewed their work; uncertain cases went to a second expert, and unresolved cases were discarded. Bounding boxes were drawn as tightly as possible around the critical visual evidence in images only, since defining consistent coordinates across dynamic frames is a separate research problem.

For evaluation, the authors ran zero-shot tests on proprietary models (GPT-5, Claude 4.0, Gemini 2.5 Pro) and open-source models (LLaVA-OneVision 7B, LLaVA-Next, QwenVL-2.5 7B and 72B, Deepseek-VL, GLM-4.5V, and others) at temperature 0.7 with a consistent prompt template. Text answers were scored by three proprietary judges (GPT-5, Gemini 2.5 Pro, Claude 4.0 Sonnet) with the average reported, and grounding was scored with Intersection over Union.

To test the benchmark as a training resource, they fine-tuned Qwen2.5-VL-7B in two stages: a cold-start SFT phase on 10% of the SportsImage data for 6 epochs, followed by Reinforcement Learning with Group Relative Policy Optimization (GRPO). Training used only the image component, which let them test whether image-trained reasoning transfers to video.

Why This Matters

Impact on research. SportR reframes sports AI from description toward adjudication. It supplies a training resource — over 6,841 human-authored CoT rationales and more than 20,000 derived QA pairs — rather than only an evaluation set, and it argues that multiple-choice formats cannot teach or measure generative step-by-step reasoning. The finding that image-only training transfers to video tasks offers a concrete direction for building general sports reasoning models with less video data.

Real-world applications:

• Automated officiating, where a system must identify a foul, name its type, and predict the resulting penalty.
• Tactical analysis for coaches, using the offensive and defensive tactic identification tasks (Q6, Q7, Q12, Q13).
• Broadcast and commentary assistance that explains in plain language why a call was made.
• Sports analytics and player evaluation pipelines that need grounded, evidence-linked judgments rather than surface descriptions.

Industry relevance. Leagues, broadcasters, and sports-tech vendors are already deploying computer vision for tracking and highlight generation. SportR targets the harder layer those systems avoid: connecting what is visible to what the rulebook says. The low grounding IoU scores (4.61% to 9.94%) indicate that current off-the-shelf models are not yet reliable for calls that hinge on a specific point of contact, which is precisely where officiating support would need to be trustworthy.

Future Directions

• Extending grounding to video. The authors deliberately excluded coordinate-based grounding from SportsVideo, noting that annotating precise spatial coordinates consistently across multiple dynamic frames is a substantial research problem in its own right and an important avenue for future work.
• Pushing toward the top of the pyramid. Elite, professional-level scenarios involving complex tactical combinations or obscure, controversial edge-case rulings are described as an ultimate goal that currently exceeds available annotation expertise and model capability.
• Closing the rule-grounding gap. The error analysis shows that even when strong models perceive visual evidence correctly, they struggle to map it to the correct abstract rule, suggesting targeted work on aligning perception with domain knowledge.
• Improving temporal perception. Since Visual Perception Error and Visual Hallucination dominate video failures, better parsing of fine-grained temporal dynamics is a prerequisite before rule adjudication can be meaningfully evaluated in video.

Target Audience

This paper is most useful to researchers building and evaluating multimodal large language models, particularly those working on video and image question answering, visual grounding, and domain-specific reasoning benchmarks. It is also relevant to sports analytics and sports-tech practitioners who need to know what current models can and cannot do on rule-based judgment, and to machine learning engineers considering supervised fine-tuning or GRPO-based reinforcement learning for specialized reasoning tasks. Readers interested in benchmark design and human annotation methodology more broadly will also find the quality-control protocol and error taxonomy informative.

Authors’ abstract

Deeply understanding sports requires an intricate blend of fine-grained visual perception and rule-based reasoning - a challenge that pushes the limits of current multimodal models. To succeed, models must master three critical capabilities: perceiving nuanced visual details, applying abstract sport rule knowledge, and grounding that knowledge in specific visual evidence. Current sports benchmarks either cover single sports or lack the detailed reasoning chains and precise visual grounding needed to robustly evaluate these core capabilities in a multi-sport context. To address this gap, we introduce SportR, the first multi-sports large-scale benchmark designed to train and evaluate MLLMs on the fundamental reasoning required for sports intelligence. Our benchmark provides a dataset of 4,789 images and 2,052 videos. To enable granular evaluation, we structure our benchmark around a progressive hierarchy of question-answer pairs designed to probe reasoning at increasing depths - from simple infraction identification to complex penalty prediction. For the most advanced tasks requiring multi-step reasoning, such as determining penalties or explaining tactics, we provide 6,841 high-quality, human-authored Chain of Thought annotations. In addition, our benchmark incorporates both image and video modalities and provides manual bounding box annotations to test visual grounding in the image part directly. Extensive experiments demonstrate the profound difficulty of our benchmark. State-of-the-art baseline models perform poorly on our most challenging tasks. While training on our data via Supervised Fine-Tuning and Reinforcement Learning improves these scores, they remain relatively low, highlighting a significant gap in current model capabilities. SportR presents a new challenge for the community, providing a critical resource to drive future research in multimodal sports reasoning.