SoccerMaster: A Vision Foundation Model for Soccer Understanding

AI summary

Overview

Research area: Computer vision, specifically video-based sports understanding and vision foundation models (Haolin Yang, Jiayuan Rao, Haoning Wu, Weidi Xie — School of Artificial Intelligence, Shanghai Jiao Tong University, and Shanghai Innovation Institute; arXiv:2512.11016v2 [cs.CV]).

Technical level: Advanced. The paper assumes familiarity with vision transformers, DETR-style decoders, contrastive vision-language pretraining, multi-task learning, and multi-object tracking.

One-sentence scope: The paper introduces a single soccer-specific vision foundation model, pretrained with supervised multi-task learning on a purpose-built automatic annotation pipeline plus existing soccer datasets, and evaluates it across both fine-grained spatial perception and high-level semantic reasoning tasks.

What This Paper Is About

Soccer video understanding is currently fragmented: separate expert models are built for detecting players, recognizing jersey numbers, spotting events, or generating commentary. The paper argues that this separation also splits the field into two halves — spatial perception (where things are, who is involved) and semantic reasoning (what is happening) — and that models optimized for one half struggle with the other.

The goal is a single backbone that learns both at once, so that one set of representations can be fine-tuned cheaply for many soccer tasks. A major obstacle the authors identify is data: dense spatial labels (boxes, roles, jersey numbers, pitch keypoints and lines, camera parameters) are scarce and expensive, so they build an automated pipeline to manufacture them from broadcast footage.

Key Contributions

1. SoccerMaster, described as the first soccer-specific vision foundation model that unifies diverse soccer visual understanding tasks in a single framework via supervised multi-task pretraining, serving as a general backbone for soccer analysis.
2. A supervised multi-task pretraining strategy that jointly optimizes fine-grained spatial perception and high-level semantic reasoning on soccer videos, producing multi-granularity representations that capture both local spatial detail and global semantic context.
3. SoccerFactory, an automated data curation pipeline that generates large-scale spatial annotations from broadcast footage, integrated with existing soccer video datasets into a comprehensive soccer-specific pretraining resource (roughly 7.45M frames across 248.3K video segments).
4. Extensive evaluation showing SoccerMaster is state-of-the-art or competitive on every downstream task tested after simple fine-tuning, and that it outperforms both general-purpose vision foundation models and a soccer-specific competitor on the four pretraining tasks with direct evaluation.

Main Findings

• Pretraining-task dominance across the board. With frozen baselines and trainable task heads, SoccerMaster reached 91.5 AP@50 and 49.5 mAP on athlete detection and identification, jersey-number accuracy 79.7, role accuracy 99.1, keypoint accuracy 95.3, line accuracy 95.7, event classification accuracy 77.2, and 39.0% top-1 on video-commentary retrieval (computed within batches of 48). The paper reports this as a +17.5% mAP gain on athlete detection and identification and +11.9% accuracy gain on event classification over the second-best methods.

• Largest gap appears in vision-language alignment. SoccerMaster's 39.0% top-1 retrieval far exceeds the baselines: SigLIP 2 scored 3.4% and MatchVision scored 4.0%. The authors attribute SigLIP 2's low score to a severe domain gap, and note DINOv3 lacks a corresponding text encoder for alignment evaluation.

• MatchVision is competitive but narrow. Although tailored to soccer, MatchVision was competitive on keypoint detection (92.0 accuracy, 85.2 precision), line detection (86.6 precision), and event classification (65.3 accuracy), but struggled in athlete detection/identification (51.9 AP@50, 17.0 mAP) and vision-language alignment. The authors suggest this stems from over-reliance on jointly optimizing vision and language encoders at the expense of dense spatial objectives.

• SoccerFactory annotations are competitive with challenge winners. Applied to the official test set of the SoccerNet Game State Reconstruction (GSR) Challenge, the pipeline achieved a GS-HOTA of 64.1, GS-DetA of 51.5, and GS-AssA of 79.9, versus KIST-GSR at 61.5 / 48.5 / 78.0, Metrica-Sports at 58.2 / 44.4 / 76.2, and Playbox & MIXI at 58.1 / 41.3 / 81.6.

• Adding pipeline-generated data lifts other backbones too. Training SigLIP 2 with SoccerFactory data raised its AP@50 from 69.1 to 72.3 and mAP from 29.2 to 32.0; DINOv3 rose from 61.8 to 70.2 AP@50 and 22.9 to 28.0 mAP.

• Camera calibration is state-of-the-art when fine-tuned. On SN22-test-center, SoccerMaster fine-tuned reached JaC₅ 76.9, JaC₁₀ 92.3, JaC₂₀ 94.4, CR 98.6, FS 75.8 at 512×512, compared with PnLCalib at 960×540 scoring JaC₅ 80.3, JaC₁₀ 91.9, JaC₂₀ 94.2, CR 97.9, FS 78.6, and PnLCalib at 512×512 scoring 70.2 / 86.8 / 91.7 / 96.4 / 67.6. Zero-shot SoccerMaster on the same set scored 70.5 / 92.0 / 94.8 / 99.4 / 70.1. On SN23-test, fine-tuned SoccerMaster reached 71.1 / 85.4 / 87.8 / 79.1 / 56.2, versus PnLCalib at 960×540 at 76.7 / 87.2 / 90.1 / 79.5 / 60.9 and at 512×512 at 66.9 / 82.4 / 86.7 / 78.4 / 51.8; zero-shot SoccerMaster scored 59.5 / 78.1 / 81.6 / 75.2 / 44.8.

• Tracking is competitive and end-to-end. SoccerMaster + MOTIP reached HOTA 59.1, DetA 65.2, AssA 53.9, MOTA 81.6, IDF1 74.6, against YOLOv8+PRTreID at 59.8 / 61.1 / 58.6 / 73.1 / 74.5, YOLOv8+StrongSort++ at 56.2 / 62.9 / 50.3 / 75.0 / 66.5, and YOLOv8+OC-SORT at 54.6 / 63.5 / 47.1 / 76.2 / 62.5. The paper notes SoccerMaster is the only one of these employing an end-to-end setting.

• Commentary generation shows mixed but leading results. SoccerMaster achieved BLEU@1 31.3, BLEU@4 8.9, METEOR 26.2, ROUGE-L 26.6, CIDEr 38.6, versus MatchVision at 30.9 / 8.7 / 26.9 / 27.6 / 35.7 and SigLIP 2 at 28.9 / 6.9 / 24.7 / 25.4 / 28.3 — higher BLEU@1, BLEU@4, and CIDEr, but lower METEOR and ROUGE-L than MatchVision.

• Ablation results are referenced but not included in the provided content. The paper states that ablation studies are conducted in Section 6.3; the truncated text does not report those results, nor the appendix details referenced for dataset composition (Sec. A) and implementation (Sec. B).

Methodology in Plain English

The authors build a single visual encoder and attach several small, task-specific "heads" to it, then train everything together.

• Encoder design. The backbone follows a ViT structure with a TimeSformer-like twist: most layers apply spatial attention within each frame, and only the final layers apply spatiotemporal attention across frames. Specifically, the encoder is initialized from siglip2-large-patch16-512 and uses 16 spatial transformer blocks followed by 8 spatiotemporal transformer blocks, with hidden dimension 1024. Videos are sampled into clips of 30 frames at 512×512 resolution with 16×16 patches. Spatial attention produces fine-grained spatial features; after an attention-pooling (MAP) head across spatial dimensions, the model produces global semantic features.

• What it is trained on. Four pretraining objectives share the encoder: athlete detection and identification (bounding box, role — goalkeeper/player/referee — and jersey number, where the number is null for goalkeepers, referees, or when not visible), pitch registration (field keypoints and line segments, following PnLCalib), event classification over the 24 event categories defined in SoccerReplay-1988, and vision-language alignment against commentary text embedded by a SigLIP 2 text encoder using SigLIP contrastive loss. The total loss is a weighted sum of the task losses.

• Where the data comes from. Because dense spatial labels are scarce, SoccerFactory generates them in three stages: (1) field registration, detecting keypoints and lines and using the PnL module to estimate camera parameters and refine annotations by projection from canonical pitch coordinates; (2) tracking and identification, using a soccer-fine-tuned YOLOv8 detector, StrongSORT tracking with PRTReID embeddings, Qwen2.5-VL for role and jersey-number recognition filtered by a legibility classifier, and clustering of tracklet-averaged ReID embeddings plus pitch coordinates for team affiliation; (3) post-processing refinement, using SAM2-based segmentation to recover missed detections and fix identity switches, majority voting across tracklets for jersey numbers and roles, and merging of short fragments using ReID embeddings and jersey-number consistency. Pipeline output is applied to main-camera clips from 500 matches in SoccerNet-v2, excluding replays, close-ups, and alternative angles.

• Pretraining dataset composition. The combined resource is about 7.45M frames over 248.3K video segments: 2.75M frames for spatial perception and 4.71M frames for semantic reasoning sampled at 1FPS. Token contributions listed are SoccerNet-GSR (42.8K frames), SoccerFactory-generated (2.7M), SoccerNet-v2 (1.6M), MatchTime (7.1K), and SoccerReplay-1988 (3.1M).

• Evaluation protocol. For baselines (SigLIP2, DINOv3, MatchVision), encoders were frozen and only lightweight task heads were trained on the same pretraining data; SoccerMaster was evaluated directly on the pretraining tasks without further tuning since it was natively pretrained on them.

• Downstream adaptation. Commentary generation uses a Q-Former to aggregate semantic features projected into Llama-3-8B's embedding space as prefix embeddings; camera calibration reuses the pitch-registration keypoints and lines through the PnL refinement module; tracking follows MOTIP by treating association as classification, with object-level features from the DETR decoder and an ID decoder attending to historical trajectories.

Why This Matters

Impact on research. The paper argues against the prevailing fragmented paradigm of one expert model per soccer task, and provides evidence that a single frozen-then-lightly-tuned backbone can match or beat specialists. It also shows that automated annotation can substitute for expensive manual spatial labeling — the same pipeline that produced the training data scored 64.1 GS-HOTA on the SoccerNet GSR test set. And it highlights a specific weakness in generic vision-language models for sports: SigLIP 2 managed only 3.4% top-1 retrieval on soccer commentary alignment, suggesting general-purpose alignment does not transfer to this domain.

Real-world applications (derived from tasks the paper covers):

• Broadcast production, via automated commentary generation and event classification for highlights.
• Tactical and performance analysis, via athlete detection, identification (roles and jersey numbers), and multiple object tracking in pitch coordinates.
• Camera calibration pipelines that map broadcast frames to standardized pitch coordinates, enabling downstream spatial analytics.
• Automated, large-scale video annotation for rights holders and analysts, replacing part of the manual labeling effort.

Industry relevance. Soccer has a vast global audience with substantial commercial and cultural impact (as the paper frames it), and the tasks addressed — tracking, event detection, commentary, camera geometry — map directly onto analytics and broadcast products. The authors state that the data, code, and model will be publicly available, which lowers the barrier for both academic and commercial follow-up.

Future Directions

• Extending beyond main-camera and standard broadcast views. The pipeline deliberately excludes replays, close-ups, and alternative angles to keep field coverage stable, leaving those viewpoints and their annotations unaddressed.
• Covering subjective or rule-based reasoning. Foul recognition appears in the related work survey but is not among SoccerMaster's pretraining or downstream tasks, so officiating-style judgments remain outside the current framework.
• Filling in the missing experimental detail. The ablation studies in Section 6.3 and the dataset/implementation appendix sections (Sec. A and Sec. B) are referenced but not present in the provided content, so the sensitivity of results to data composition, task weighting, and encoder design choices is not reported here.
• Testing transfer beyond soccer. The paper positions SoccerMaster as soccer-specific; whether the same supervised multi-task pretraining recipe and automated curation pipeline transfer to other sports with broadcast footage is left as an open question.

Target Audience

Researchers and engineers working on sports video understanding, video foundation models, and multi-task visual pretraining; practitioners building broadcast analytics, tracking, or automated commentary systems; and computer vision groups interested in automated annotation pipelines as an alternative to manual labeling. Readers need working familiarity with transformer architectures, object detection heads, contrastive vision-language training, and tracking metrics to follow the experimental sections in detail.

Authors’ abstract

Soccer understanding has recently garnered growing research interest due to its domain-specific complexity and unique challenges. Unlike prior works that typically rely on isolated, task-specific expert models, this work aims to propose a unified model to handle diverse soccer visual understanding tasks, ranging from fine-grained perception (e.g., athlete detection and identification) to high-level semantic reasoning (e.g., event classification). Concretely, our contributions are threefold: (i) we present SoccerMaster, the first soccer-specific vision foundation model that unifies diverse tasks within a single framework via supervised multi-task pretraining; (ii) we develop an automated data curation pipeline, SoccerFactory, to generate scalable spatial annotations, and integrate multiple existing soccer video datasets as a comprehensive pretraining data resource for multi-task pretraining; and (iii) we conduct extensive evaluations demonstrating that SoccerMaster consistently outperforms task-specific expert models across diverse downstream tasks, highlighting its breadth and superiority. The data, code, and model will be publicly available.