Event-STVG: Spatio-Temporal Video Grounding for Long-Form Event Reasoning

AI summary

Overview

Research area: Computer vision and multimodal video-language understanding, specifically spatio-temporal video grounding (STVG) and long-form event reasoning with multimodal large language models (MLLMs).

Technical level: Advanced. The paper assumes familiarity with video grounding metrics (tIoU, vIoU), semantic role schemas, and MLLM evaluation protocols, though the core idea is explained clearly enough for adjacent-field readers.

Scope: The paper introduces a new task formulation (Event-STVG) that turns video grounding from description matching into event-level reasoning, plus an accompanying 4,016-occurrence, 8-domain benchmark and a systematic evaluation of ten models.

What This Paper Is About

Existing spatio-temporal video grounding benchmarks give a model a detailed description of a target ("the woman in the red jacket passing the ball") and ask it to find that target in space and time. This mostly tests description matching rather than the long-form reasoning that MLLMs are increasingly expected to perform. The authors instead ask a model to resolve which occurrence of a repeated event is meant (for example, the third "shot on target"), infer who fills each semantic role (shooter, last passer, goalkeeper), and ground both the event interval and every role's trajectory — without ever describing the participants.

Key Contributions

1. A new task formulation. Event-STVG reframes spatio-temporal grounding as a test of long-form event reasoning: given an event type, an ordinal index, and a set of required roles, a model must locate the m-th occurrence and ground the spatial trajectory of each queried role inside that interval.
2. A large, manually annotated benchmark. 4,016 annotated event occurrences spanning 8 domains, 26 event types, 30 semantic roles, 7,323 event-role pairs, and 59,381 annotated bounding boxes, with roughly 10% cross-annotated (mean temporal IoU 0.9039, mean spatial role IoU 0.9161).
3. Comprehensive model evaluation. Ten vision-language models are tested — nine general-purpose VLMs (Gemini-3.7 Flash, GPT-5.6 Sol, Qwen-3.8 Max, Qwen-3.7 Plus, Qwen-3.5 9B, GLM-5.3 Flash, GLM-5V Turbo, GLM-4.6V, MiniMax-M3) and one STVG specialist, PTD with a Qwen3-VL-4B checkpoint — across oracle, 1-, 3-, 5-, and 10-minute settings and a multi-domain split.
4. A three-level error analysis. The authors isolate distinct failure modes in event-instance resolution, spatial role grounding, and multi-role composition, and show they recur across domains rather than being artifacts of soccer.

Main Findings

• Even oracle temporal windows are hard. When models are handed the ground-truth event interval and scored only on role grounding, the best result is Gemini-3.7 Flash at 24.18% m-vIoU (vIoU@0.3 38.40, vIoU@0.5 11.35). GPT-5.6 Sol reaches 13.38% m-vIoU under the same oracle setting.
• End-to-end 1-minute grounding nearly collapses. In the 1-minute setting, six of the nine general-purpose models score m-vIoU below 1% and never achieve vIoU@0.5 on any event. Gemini-3.7 Flash leads on joint grounding at 10.50% m-vIoU; GPT-5.6 Sol leads on temporal localization at 48.23% m-tIoU but only 2.27% m-vIoU. The STVG specialist PTD reaches 1.72% m-vIoU at 1 minute.
• Performance degrades with context length. On the fixed 300-query subset, Gemini drops from 11.36% m-vIoU at 3 minutes to 7.37% at 5 minutes and 5.40% at 10 minutes — the 5.4% figure the abstract highlights as the strongest 10-minute result. GPT-5.6 Sol falls from 43.16% to 36.82% m-tIoU over the same span, and Qwen-3.8 Max from 39.00% to 24.57% m-tIoU.
• No model dominates both components. GPT retains the strongest temporal localization at every duration while Gemini is substantially stronger in joint spatio-temporal grounding, showing that better event localization does not translate into better end-to-end grounding.
• Difficulty generalizes beyond soccer. Across 500 multi-domain events, the best domain (daily work) averages only 21.30% m-vIoU across the three tested models, and the worst (ice hockey) reaches 0.69%; entertainment reaches 15.31%, animal activity 7.00%, volleyball 5.76%, esports 3.56%, and basketball 1.87%.
• Bottleneck I — instance confusion, not boundary error. As input grows longer, models increasingly pick a different valid occurrence of the same event type. Hit rates fall and confusion rates rise significantly when same-type distractors are present (all Holm-adjusted p < 0.0001). Among queries the model hits at every duration, tIoU does not decline significantly with length, implying boundaries are fine once the right occurrence is found. In basketball, later targets show a 41.81% average confusion rate.
• Bottleneck II — temporal overlap does not imply spatial grounding. The authors define a Relative Spatial Grounding Gap; across non-soccer domains it averages 71.12%, corresponding to only 28.88% mean spatial IoU within temporally overlapping frames.
• Bottleneck III — complete role composition is rare. Across domains, the average rate of grounding all required roles correctly is only 0.73%. Partial coverage is the common outcome, even when the event interval is correct.
• Repetition is built into the benchmark. The share of soccer contexts containing repeated same-type events rises from 13.68% at one minute to 68.69% at ten minutes, and 99.40% of queries in the other domains contain repeated same-type occurrences.
• Multi-role grounding is the norm, not an edge case. 58.91% of all event instances involve multiple semantic roles.

Methodology in Plain English

The authors start by redefining the question a grounding model must answer. Instead of giving it a description of a person, they give it three things: an event category, an ordinal ("the second occurrence"), and a list of roles to find. The model must work out which moment in the video that ordinal refers to, then draw boxes around whoever fills each role for the duration of that event. Event boundaries are pinned to canonical role actions — for instance, a shot on target runs from the last passer's first possession to the moment the shot outcome becomes clear — so that annotations are consistent without prescribing how the play unfolds.

For data, soccer is the main testbed because players look alike, scenes are uniform, and many agents interact at once. Candidate events are pulled automatically from SoccerNet-v2 annotations, then human annotators set exact start and end times. To make trajectory annotation practical, an automated tracker (SoccerMaster plus tracking and re-identification tools) proposes boxes and identities at 2 FPS, which annotators bind to roles such as shooter or goalkeeper, with full freedom to reassign, delete, or draw boxes by hand. All ground-truth trajectories are human-verified. Seven further domains — basketball, volleyball, ice hockey, animal hunting, fast-food packing, entertainment programs, and esports — are annotated with a source-agnostic manual procedure, yielding 500 events organized into 100 minute-scale videos.

Evaluation is stratified. All 3,516 soccer events are scored in an oracle setting (ground-truth interval given, role grounding only) and a 1-minute setting, with a fixed 300-event subset re-run at 3, 5, and 10 minutes; ordinal indices are recomputed per duration because longer contexts contain more occurrences. Metrics are mean temporal IoU for intervals and mean video IoU for role tubes, with vIoU@0.3 and vIoU@0.5 as thresholded event-level scores; failed or unparseable responses score zero and stay in the denominator. Models are queried with a fixed prompt that requests JSON output, boxes normalized to [0, 1000], and role boxes every 0.5 seconds, with null allowed when a role is not visible.

Finally, the authors decompose errors into a hierarchy — did the model pick the right occurrence, did it ground the roles once the event was found, and did it ground all roles together — and measure each level separately rather than reporting a single aggregate score.

Why This Matters

Impact on research. The paper argues that current STVG benchmarks measure description-conditioned detection rather than reasoning. Table 1 in the paper shows that no prior benchmark jointly requires temporal grounding of an event and spatial grounding of its semantic-role fillers — MultiVENT-G grounds role evidence but separates temporal and spatial retrieval, while VidEvent and NEST do not spatially ground roles at all. Event-STVG therefore provides a harder, more diagnostic target for MLLM evaluation, and the authors note that its verifiable grounding signals could supply fine-grained feedback for reinforcement-learning post-training.

Real-world applications:

• Sports analytics: automatically indexing which players were involved in a specific shot, foul, or transition, and producing per-player spatial records from broadcast footage.
• Incident review in traffic or workplace safety: identifying the third near-miss in a shift and grounding the vehicles or people involved.
• Industrial and logistics monitoring: locating a specific production or packing cycle and tracing which items and operators participated.
• Video archive and broadcast search: retrieving an exact occurrence of a recurring event and the entities involved, rather than a whole clip.

Industry relevance. Any organization that needs structured, verifiable extraction from long video — sports rights holders, safety and compliance teams, media archives, and robotics or surveillance vendors — has a stake in whether models can resolve which event happened and who was in it. The finding that leading proprietary models score 5.40% m-vIoU at 10 minutes, and that the best complete-role grounding rate across domains is 0.73%, is a clear signal that current systems are not deployable for this class of task without further work.

Future Directions

• Explicit occurrence tracking. Since the dominant long-context failure is confusing one same-type event with another, models may need an explicit mechanism for enumerating and indexing repeated events before attempting localization.
• Decoupling and recombining the two skills. GPT leads on temporal localization while Gemini leads on joint grounding; closing the gap likely requires architectures or training regimes that preserve both capabilities rather than trading one for the other.
• Compositional role grounding. With a 0.73% average complete-success rate, grounding roles one at a time is clearly insufficient. Methods that enforce consistency across roles within one event are an open problem.
• Using grounding as training signal. The authors propose that verifiable spatio-temporal grounding could feed reinforcement-learning post-training; whether that improves general visual reasoning, not just this benchmark, is untested here.

Target Audience

Researchers working on video-language grounding, multimodal large language models, and video question answering who need a harder benchmark than description-guided localization. It is also relevant to practitioners building sports analytics, video retrieval, or compliance-review systems, and to benchmark designers interested in how to structure semantic roles and ordinal queries. Readers without a grounding background will find the task formulation accessible, but the metric definitions and error analysis assume comfort with IoU-based evaluation.

Authors’ abstract

Spatio-Temporal Video Grounding (STVG) aims to jointly localize a target in space and time given a natural-language query. Despite continued advances, existing STVG benchmarks remain largely centered on localizing targets through instance-specific descriptions, leaving them largely decoupled from long-form reasoning increasingly emphasized in multimodal large language models (MLLMs). In this paper, we introduce Event-STVG, a task that shifts STVG from locating what is described toward event-level reasoning by unifying long-form context, repeated-event reasoning, event localization, role inference, and multi-target grounding. To facilitate research on Event-STVG, we construct a large-scale benchmark comprising 4,016 annotated event occurrences across soccer and seven additional domains. Experiments with state-of-the-art MLLMs reveal substantial difficulty, with the strongest model achieving only 5.4% m-vIoU at the 10-minute horizon. Across domains, we consistently observe confusion among repeated same-type events, spatial grounding bottlenecks, and rare success in jointly grounding all required roles. Event-STVG establishes a new challenge for visual understanding in MLLMs, calling for future models that can jointly reason about complex events across space and time. Code and annotations are available at https://github.com/THU-KEG/EventSTVG