X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning

AI summary

Overview

Research area: Computer vision and multi-agent representation learning, applied to esports gameplay understanding (Counter-Strike 2).

Technical level: Advanced — the paper assumes familiarity with contrastive self-supervised learning, vision transformers and video encoders, multi-label classification, and multi-agent reinforcement learning concepts.

Scope: The paper introduces a cross-egocentric esports video benchmark (X-Ego-CS) and a contrastive learning method (CECL) that aligns teammates' first-person video streams so a single agent can infer team-level tactical state.

What This Paper Is About

Most existing work on team behavior in sports and games relies on third-person broadcast footage with a global, omniscient view, which does not reflect how real players actually experience a match — from their own partial, first-person perspective. The authors argue that modeling team tactics requires reasoning about what each individual agent can perceive and infer about teammates and opponents.

Their goal is to give a single agent the ability to reconstruct team-level situational awareness from only its own egocentric view. To do this, they build a dataset of synchronized first-person video from every player in professional Counter-Strike 2 matches, and train a model to align teammates' visual representations at the same moment in time.

Key Contributions

1. X-Ego-CS dataset. A benchmark of 124 hours of gameplay footage from 45 professional-level Counter-Strike 2 matches, containing synchronized first-person (cross-egocentric) video streams from all players plus state-action trajectories sampled at 64 ticks per second. The authors state it is the first dataset with synchronized egocentric video streams and structured state-action trajectories from all players in professional esports matches.

2. Cross-Ego Contrastive Learning (CECL). A visual contrastive learning method that aligns entire teams' first-person view representations at the same timestep, creating a shared latent representation of team state without explicit communication or centralized supervision.

3. A teammate-opponent location prediction benchmark task. Two downstream multi-label classification tasks — Teammate Location Nowcast (TLN) and Enemy/Opponent Location Nowcast (ELN/OLN) — that predict which of 23 named locations on the de_mirage map are occupied by teammates or opponents at a given time.

4. Empirical validation across four video encoders. CECL is evaluated on top of DINOv2, ViViT, VideoMAE, and SigLIP, showing improvements concentrated in low-observability settings where only 1–2 player viewpoints are available.

Main Findings

• CECL improves single-POV inference overall. On the Teammate Location Nowcast and Enemy Location Nowcast tasks with a single agent's POV as input, CECL produced a mean absolute gain of 1.45 percentage points over all baselines across model-and-metric combinations.

• Best single-encoder results. CECL on DINOv2 reached 18.13 Subset Accuracy (+1.24), 9.67 Hamming Distance (−0.23), 37.61 Macro F1 (+1.95), and 58.37 Micro F1 (+1.18) on Teammate Location Nowcast; and 14.60 (+1.13), 11.00 (−0.13), 25.02 (+2.10), 50.54 (+0.77) on Enemy Location Nowcast. CECL on SigLIP produced the largest Enemy Location Nowcast gains: +2.45 Subset Accuracy, +4.59 Macro F1, +2.72 Micro F1.

• Not every combination improved. CECL on ViViT reduced Enemy Location Nowcast performance (12.92 Subset Accuracy, −1.28; 20.43 Macro F1, −2.41), and CECL on SigLIP reduced Teammate Location Nowcast Macro F1 (−3.90) and Micro F1 (−2.29).

• Gains shrink as more viewpoints are added. Performance was compared as the number of agent POVs increased from 1 to the full team size of 5. CECL helped substantially at 1–2 POVs but showed diminishing returns at higher POV counts, with slight underperformance at full-team settings. The authors attribute this to the contrastive objective producing a more concentrated embedding space and some information loss from the original visual embedding space.

• Bias and temperature initialization matter. The sigmoid loss is sensitive to the imbalance between positives and negatives. With bias b = 0 the model struggled (13.90 Teammate Subset Accuracy, 9.07 Enemy Subset Accuracy), while b = −3 with temperature log 10 generally performed best (17.01 Teammate Subset Accuracy, 12.38 Enemy Subset Accuracy). With b = −10 and temperature log 1, performance collapsed to 1.40 Subset Accuracy for both tasks.

• Embedding structure improved after training. t-SNE visualizations showed an 8.87% increase in location-based separation ratio (from 1.272 to 1.385) and a 15.86% improvement in team-based separation (from 1.040 to 1.205), while temporal gradients remained smooth.

• V-JEPA2 did not follow the pattern. The authors report that CECL did not consistently improve the V-JEPA2 0.3B model, unlike the other encoders. They hypothesize its high capacity (over 3 times more parameters than the rest of the models) and predictive pretraining already encode strong spatiotemporal coherence, or that it needs longer training horizons or refined temperature scaling.

• Dataset characteristics. X-Ego-CS spans 45 matches, 1011 rounds, 372 unique players, and an average round duration of 44 seconds of player alive time. All videos are 720p at 30 FPS on the de_mirage map. Data came from FACEIT's top 100 players on the public Elo leaderboard (Elo above 2000 corresponds to the highest FACEIT level 10 rank, roughly the top 10% of players; above 3000 represents roughly the top 1%).

Methodology in Plain English

Data collection. The team downloaded professional Counter-Strike 2 demo replay files from the FACEIT platform, extracting metadata, player state-action trajectories, and game events with off-the-shelf parsers (AWPY and demoparser2). They built an automated in-game recording system: the demo metadata gives exact tick ranges for each player's alive period in a round, and console commands jump to the right tick and lock the camera onto the correct player. NVIDIA Shadowplay then records only the segments where that player was alive.

The learning idea. For each timestep, the model takes every teammate's first-person video segment and pushes their learned representations to be similar to each other, while pulling them apart from representations of other teams, other timesteps, and other matches. Teammates who see the same round at the same moment form positive pairs; everything else forms negatives. The intuition is that shared visual experiences — for example, multiple teammates being blinded simultaneously by the same flashbang — carry tactical information about where opponents are and how the team is moving, so forcing similar embeddings makes the encoder learn a latent representation of the team's shared situation.

Architecture. A frozen vision encoder converts each agent's video to an embedding, plus a trainable two-layer MLP projector mapping to 768 dimensions. For multi-agent inference, selected embeddings are aggregated by concatenation and a two-layer MLP, then combined with a team-side indicator vector and passed to task-specific predictor heads.

Training and preprocessing. Videos are resized to 224×224 (distorting aspect ratio), normalized with ImageNet statistics, and sampled at 4 FPS over 5-second windows with ±0.3 second jittering. The data is split 70:15:15 train:validation:test by rounds. Optimization uses AdamW with learning rate 3×10⁻⁴, weight decay 10⁻², bFloat16 mixed precision, batch size 32, and 8 epochs, on a single A100, A40, or RTX4090 GPU. Downstream tasks use binary cross-entropy, and the total loss combines the contrastive loss and BCE loss with a weighting coefficient.

Fairness control. The top-left corner of every video is masked so that the in-game mini-map cannot leak teammate or opponent locations to the model.

Why This Matters

The paper argues that cross-egocentric representation learning is a path toward models that internalize a shared team state without requiring explicit communication or centralized supervision — an implicit coordination mechanism that resembles how human teammates build joint situational awareness through mutual observation and inference. It positions gameplay understanding as a controllable testbed for multi-agent modeling, spatiotemporal reasoning, and human-AI teaming.

Potential real-world applications (as discussed by the authors):

• Collaborative robotics — robots inferring the unobserved state of human or robotic teammates from their own partial sensor readings.
• Defense operations — team-level situational awareness under partial observability.
• Autonomous vehicle fleets — vehicles inferring the state of other vehicles in the fleet from individual perception.
• Human-AI teaming in mixed systems — interpretable, coordination-aware world models that integrate perception, prediction, and planning.

Industry relevance. The work speaks directly to esports analytics and AI teammates in competitive games, and the dataset and code are released at https://github.com/HATS-ICT/x-ego under a CC BY 4.0 license. The work was sponsored by the U.S. Army Combat Capabilities Development Command – Soldier Centers under contract number W912CG-24-D-0001.

Future Directions

• Hybrid contrastive-reconstruction objectives. The authors note that the contrastive objective compresses representations and slightly reduces expressivity when full team visibility is available, and suggest balancing shared and individual encoding fidelity.

• Masked video modeling of viewpoints. Inspired by masked language modeling, future work could randomly mask one agent's viewpoint and train a video generator to reconstruct the missing perspective, enhancing cross-agent imagination.

• Full cross-team alignment. CECL currently aligns only teammates at the same timestep, not teammates and opponents jointly. Contrasting all agents' viewpoints in a unified embedding space could capture inter-team dependencies and adversarial intent.

• Transfer to naturalistic real-world teamwork. Cross-egocentric data already exists abundantly outside games, from body-worn or vehicle-mounted cameras, offering opportunities to model multi-human coordination in real settings.

• Embodied evaluation. Integrating CECL-trained representations into controllable actor agents remains open, which would allow direct measurement of CECL's influence on decision-making rather than only on prediction tasks.

Target Audience

Researchers working on multi-agent systems, self-supervised video representation learning, and esports or sports analytics who want a benchmark and method for modeling team behavior from first-person perspectives. It is also relevant to those studying theory-of-mind-style reasoning, human-AI teaming under partial observability, and spatiotemporal representation learning. Given the technical density (contrastive objectives, video transformer encoders, multi-label evaluation), readers will benefit most from a background in deep learning for vision and multi-agent modeling.

Authors’ abstract

Human team tactics emerge from each player's individual perspective and their ability to anticipate, interpret, and adapt to teammates' intentions. While advances in video understanding have improved the modeling of team interactions in sports, most existing work relies on third-person broadcast views and overlooks the synchronous, egocentric nature of multi-agent learning. We introduce X-Ego-CS, a benchmark dataset consisting of 124 hours of gameplay footage from 45 professional-level matches of the popular e-sports game Counter-Strike 2, designed to facilitate research on multi-agent decision-making in complex 3D environments. X-Ego-CS provides cross-egocentric video streams that synchronously capture all players' first-person perspectives along with state-action trajectories. Building on this resource, we propose Cross-Ego Contrastive Learning (CECL), which aligns teammates' egocentric visual streams to foster team-level tactical situational awareness from an individual's perspective. We evaluate CECL on a teammate-opponent location prediction task, demonstrating its effectiveness in enhancing an agent's ability to infer both teammate and opponent positions from a single first-person view using state-of-the-art video encoders. Together, X-Ego-CS and CECL establish a foundation for cross-egocentric multi-agent benchmarking in esports. More broadly, our work positions gameplay understanding as a testbed for multi-agent modeling and tactical learning, with implications for spatiotemporal reasoning and human-AI teaming in both virtual and real-world domains. Code and dataset are available at https://github.com/HATS-ICT/x-ego.