Coherent Multi-Agent Trajectory Forecasting in Team Sports with CausalTraj

AI summary

Overview

• Research area: Multi-agent trajectory forecasting for team sports analytics, spanning generative sequence modeling, spatiotemporal interaction modeling, and evaluation methodology for machine learning.
• Technical level: Advanced. The paper assumes familiarity with autoregressive/causal likelihood models, mixture density outputs, transformers, state-space models (Mamba2), and standard trajectory metrics (minADE/minFDE).
• Scope: The paper introduces CausalTraj, a temporally causal, likelihood-based model that forecasts joint multi-agent futures in team sports, and argues for joint metrics (minJADE, minJFDE) over per-agent metrics alone, evaluated on NBA SportVU, Basketball-U, and Football-U.

What This Paper Is About

Most multi-agent trajectory forecasting models are trained and measured with per-agent metrics (minADE, minFDE), which score each agent independently using its own best-of-k prediction—so different agents' "best" trajectories can come from different predicted futures. The paper's core claim is that this overlooks whether a model learns which predicted trajectories fit together into a plausible joint scenario, and it proposes CausalTraj to model the joint distribution of all agents directly, evaluated with joint metrics (minJADE, minJFDE).

Key Contributions

1. A causal, likelihood-based model for joint forecasting. CausalTraj decomposes the joint future distribution as a product of per-timestep conditional distributions, uses a mixture of Gaussians over per-step displacements, and generates whole multi-agent scenarios by autoregressive sampling. It combines a transformer-based inter-agent relation encoder, lightweight spatial adaptations, and a Mamba2-based temporal encoder.
2. Emphasis on joint metrics for sports. The paper highlights minJADE and minJFDE—which select the best scenario among k samples rather than the best trajectory per agent—and demonstrates them in the team-sports domain, extending a perspective previously shown on pedestrian datasets.
3. Benchmark results across three sports datasets. On NBA SportVU, Basketball-U, and Football-U, CausalTraj achieves competitive per-agent accuracy, the best recorded joint-metric results, and qualitatively more coherent gameplay evolutions.
4. Ablations isolating what matters. Removing the Spatial Relation Transformer Encoder, collapsing the 8-component mixture to a single Gaussian, or sampling only component means each degrade joint metrics on Basketball-U.

Main Findings

• Joint metrics are the differentiator, not per-agent metrics. On NBA SportVU, the default MoFlow leads on per-agent metrics for longer horizons (above 2.0s), while CausalTraj is slightly better on short horizons (up to 2.0s) and competitive at longer horizons—but substantially lower on minJADE and minJFDE. Reported SportVU joint results for CausalTraj (Mamba2): 0.27/0.49 at 1.0s, 0.62/1.21 at 2.0s, 1.00/1.93 at 3.0s, 1.38/2.57 at 4.0s (minJADE20/minJFDE20, metres).
• Per-agent SportVU numbers. CausalTraj (Mamba2) reports minADE20/minFDE20 of 0.14/0.20 at 1.0s, 0.33/0.49 at 2.0s, 0.54/0.78 at 3.0s, and 0.77/1.02 at 4.0s in metres.
• The Mamba2 variant stands out on Basketball-U and Football-U. On Basketball-U (metres), CausalTraj (Mamba2) reports 0.24/0.34 at 10 frames and 0.56/0.71 at 20 frames on per-agent metrics, with joint metrics of 0.45/0.85 and 0.97/1.77. On Football-U (yards), it reports 0.16/0.31 and 0.50/0.99 on per-agent metrics, with joint metrics of 0.37/0.85 and 1.12/2.68.
• A joint-accuracy loss alone does not solve the problem. A MoFlow variant trained with a joint-accuracy objective performs almost the same as its default version on the min-based joint metrics across the board; its averageJADE does improve substantially, but the authors did not adopt average metrics because they could favour models predicting a single, safe mode. The authors interpret this as evidence that modifying the loss may not suffice without explicitly modeling inter-agent causal dependencies.
• Qualitative coherence improves with joint accuracy. Visualizations show CausalTraj producing coordinated directional changes and realistic ball passes, while LED and default MoFlow scenarios look relatively homogeneous across agents (agents drifting toward the same point or direction). Compared with MoFlow (joint obj.), CausalTraj yields more diverse, physically plausible behaviors, such as abrupt direction changes to track others and ball passes traveling faster along straight paths rather than unrealistic arcs.
• Failure modes remain. Occasional implausible behaviors persist, such as a player carrying the ball with an unrealistically large gap between them and the ball colliding with the court boundary. The authors conjecture this is partly due to limited player-ball covariance learning capacity.
• Ablations confirm the design choices. On Basketball-U over a 20-frame horizon, CausalTraj (Mamba2) achieves minJADE20/minJFDE20 of 0.97/1.77; removing the Spatial Relation Transformer Encoder gives 0.99/1.81; a single Gaussian gives 1.03/1.86; and sampling only from component means gives 1.05/2.13.
• Comparison was limited by available checkpoints. The authors note that many prior models do not provide pretrained weights on the dataset, preventing consistent re-evaluation under joint metrics, and express hope that the study encourages broader adoption of joint-metric protocols.

Methodology in Plain English

The model treats a multi-agent future as a sequence of small steps. Instead of predicting the entire future horizon at once through a single global latent code, it predicts the next-frame displacement of every agent conditioned on everything observed so far, then repeats. At each step it outputs a mixture of eight Gaussians over displacement: mixture weights, per-agent means, and per-agent covariance blocks (parameterized through Cholesky factors). Training uses teacher forcing—all timesteps are trained in parallel against ground-truth next positions—by minimizing the negative log-likelihood, plus an entropy regularizer on the mixture weights (weight 0.05) to discourage all probability mass collapsing onto one component early in training.

Architecturally, each agent's history is first encoded independently, either with a PointNet-style encoder adapted to be causal via "lookback max-pooling" (aggregating only timesteps up to the current one) or with a Mamba2 module that compresses each agent's sequence into per-timestep hidden states. Learned agent embeddings distinguish only three categories: the two teams and the ball. The encodings then pass through inter-agent transformer blocks at each timestep (no cross-temporal attention here), including a Spatial Relation Transformer Encoder that builds a pairwise "mesh" tensor of relative positions plus both agents' features, so exact pairwise geometry feeds into attention values. Finally, all agents' features at a timestep are concatenated into a scene-level representation, which is passed through a 3-layer MLP to produce the next-step mixture parameters.

At inference, the model samples displacements from its own predicted distributions and updates positions recursively, producing 20 independent scenario samples for evaluation. Two variants were trained per dataset (Causal PointNet and Mamba2), with sizes of 3.0M and 3.2M parameters for the NBA dataset, using AdamW and a OneCycle schedule with maximum learning rate 0.02 and weight decay 0.01.

Why This Matters

For research, the paper reframes the objective in multi-agent sports forecasting: rather than chasing per-agent minADE/minFDE through independent multi-hypothesis heads, it argues for learning the true joint distribution and evaluating it with joint metrics, and it demonstrates that a joint-accuracy loss by itself does not automatically yield better min-based joint scores. It also provides an open-source baseline and flags the practical obstacle of missing pretrained checkpoints for fair joint-metric re-evaluation.

Real-world applications:

• Gameplay simulation and synthetic data generation, where a simulator must roll out mutually consistent player and ball movement rather than independently drawn agent paths.
• Tactical analysis and coaching tools, letting analysts inspect multiple plausible coordinated evolutions of a possession instead of one averaged path per player.
• Broadcast and fan-facing visualization, producing plausible alternative scenarios or animated reconstructions that look physically coherent.
• Transfer to other interacting-agent domains such as autonomous driving and crowd navigation, which the paper cites as sharing the same interaction and coordination challenges.

Industry relevance: professional sports teams, leagues, and sports-tech vendors invest in tracking-data pipelines (NBA SportVU-style logs, NFL Big Data Bowl data) for analytics and fan products; a model that generates coherent multi-agent futures supports simulation, decision-support, and content generation rather than only point-prediction error minimization.

Future Directions

• General, distribution-driven modeling of player-ball dynamics to address remaining implausible behaviors, which the authors attribute partly to limited player-ball covariance learning capacity in the current block-diagonal covariance design.
• Better joint modeling through distributionally grounded methods, which the paper names as future work.
• Controllable, conditional generation of tactical patterns, extending CausalTraj beyond unconditional forecasting.
• Improved metrics for multi-agent realism, to build a more robust link between quantitative scores and perceptual coherence of generated plays.

Target Audience

Researchers and practitioners working on multi-agent trajectory prediction, sports analytics, and generative sequence modeling, particularly those who design or evaluate multi-agent forecasting benchmarks; engineers building gameplay simulation or tactical-analysis systems from tracking data; and methodologists interested in why per-agent best-of-k metrics can hide failures in joint prediction.

Authors’ abstract

Jointly forecasting trajectories of multiple interacting agents is a core challenge in sports analytics and other domains involving complex group dynamics. Accurate prediction enables realistic simulation and strategic understanding of gameplay evolution. Most existing models are evaluated solely on per-agent accuracy metrics (minADE, minFDE), which assess each agent independently on its best-of-k prediction. However these metrics overlook whether the model learns which predicted trajectories can jointly form a plausible multi-agent future. Many state-of-the-art models are designed and optimized primarily based on these metrics. As a result, they may underperform on joint predictions and also fail to generate coherent, interpretable multi-agent scenarios in team sports. We propose CausalTraj, a temporally causal, likelihood-based model that is built to generate jointly probable multi-agent trajectory forecasts. To better assess collective modeling capability, we emphasize joint metrics (minJADE, minJFDE) that measure joint accuracy across agents within the best generated scenario sample. Evaluated on the NBA SportVU, Basketball-U, and Football-U datasets, CausalTraj achieves competitive per-agent accuracy and the best recorded results on joint metrics, while yielding qualitatively coherent and realistic gameplay evolutions.