eSkiTB: A Synthetic Event-based Dataset for Tracking Skiers

AI summary

Overview

Research area: Computer Vision, with focus on event-based vision, neuromorphic sensing, visual tracking, spiking neural networks, and winter-sports analytics. Technical level: Advanced. This paper introduces eSkiTB, a synthetic event-based ski tracking dataset built from SkiTB, and benchmarks event-based SDTrack against RGB-based STARK under an iso-informational conversion constraint.

What This Paper Is About

Tracking skiers in RGB broadcast footage is difficult because fast motion, static digital overlays, fences, banners, and other clutter obscure the athlete. Event cameras respond to brightness changes rather than static appearance, so fixed overlays produce no events, but no controlled winter-sport event benchmark existed. The paper creates eSkiTB from SkiTB using v2e without neural interpolation, then compares event-based SDTrack with RGB-based STARK to test whether temporal contrast helps in cluttered, high-speed ski scenes.

Key Contributions

1. eSkiTB dataset: a synthetic event-based ski tracking dataset generated from SkiTB, with 300 sequences, approximately 235 minutes of footage, and 1280 × 720 event streams.
2. Iso-informational conversion: a video-to-event pipeline using v2e that rejects neural frame interpolation, so event streams contain no hallucinated temporal information beyond the source RGB.
3. Event vs. RGB benchmarking: a systematic comparison of SDTrack (spiking transformer) and STARK (RGB transformer) on eSkiTB, including a high-clutter stress test.
4. Dense temporal ground truth: bounding boxes interpolated at 1 ms intervals (1000 Hz) using cubic spline interpolation to support event-rate evaluation.

Main Findings

• Event robustness under clutter: On a “High-Clutter” split of nine multi-camera sequences with at least 50% background clutter or partial occlusion annotations, SDTrack achieves 0.685 IoU, 0.694 Precision@20px, and 0.860 Success@0.5. This is +20.0 IoU, +15.2 precision, and +33.8 success over the generic STARK baseline.
• Overall event performance: Across the eSkiTB test split, fine-tuned SDTrack attains a mean IoU of 0.711, Precision@20px of 0.720, and Success@0.5 of 0.873. It outperforms the standard STARK baseline by +19.9 IoU points.
• Domain adaptation is necessary: Off-the-shelf SDTrack without fine-tuning reaches only 0.312 mean IoU, 0.354 Precision@20px, and 0.418 Success@0.5. Fine-tuning on eSkiTB improves this by +0.399 IoU.
• RGB remains stronger overall: STARK (Fine-tuned) reaches 0.795 mean IoU, 0.847 Precision@20px, and 0.904 Success@0.5. STARK (Ski-Specific) reaches 0.829, 0.887, and 0.935. In the high-clutter split, STARK_ski still reaches 0.833 IoU, leaving a gap of approximately 10 IoU points.
• Discipline-specific behavior: SDTrack performs strongly on ski jumping (JP) with 0.974 IoU, 0.982 Precision@20px, and 0.998 Success, a +5.9 IoU improvement over STARK_ski and a +35.6 point gain over generic STARK. It is competitive on alpine skiing (AL) at 0.762 IoU versus 0.815 for STARK_ft, but degrades on freestyle skiing (FS) at 0.569 IoU versus 0.728 for STARK_ft.
• Low-resolution advantage: In extreme low-resolution sequences, SDTrack maintains an IoU above 0.72, while the RGB STARK baseline degrades to 0.47–0.59 IoU.
• Fixed-grid limitation: The 128 × 128 × T voxel grid acts as a low-pass spatial filter. In freestyle skiing, where angular velocity is highest and scale is smallest during high jumps, this quantization causes aliasing and loss of lock.
• Source temporal quantization: Because neural interpolation is disabled, events represent a linearized approximation of motion and smooth out micro-dynamics below 40 ms. The authors describe the benchmark as a conservative lower bound.

Methodology in Plain English

The researchers started with SkiTB, an RGB ski broadcast dataset containing 300 high-resolution video sequences across alpine skiing (AL), freestyle skiing (FS), and ski jumping (JP). They converted raw RGB frames directly into synthetic event streams using the v2e simulator. They deliberately did not use neural frame interpolation before conversion, which keeps the event data iso-informational with the RGB data: every event corresponds to a real observed photometric change rather than a predicted one.

The conversion used photoreceptor exposure mode, contrast thresholds C_pos = C_neg = 0.2, a leak rate of tau_leak = 0.01 Hz, a photoreceptor cutoff frequency of f_3dB = 300 Hz, and photon shot noise at 0.001 Hz. Events are stored in HDF5 at 1280 × 720 with tuples (t, x, y, p), where t is in microseconds and p in {0, 1} represents polarity. Ground-truth boxes for the single class “Skier” are provided in JSON as [x, y, w, h] relative to 1280 × 720, with (x, y) as the top-left corner. The dataset provides event-aligned annotations and dense interpolated annotations at 1 ms intervals.

The dataset is split into 240 training sequences, 30 validation sequences, and 30 test sequences. It covers 98 unique geographical locations, 10 weather categories, and both day and night events. Sequences range from 275 to 3,582 frames, with an average duration of 1,176 frames. Average event rate exceeds 10 × 10^6 events per second. Target bounding box area ranges from 0.001% of the image area at the take-off gate to 100% during extreme close-ups, with a mean coverage of 4.99%.

For benchmarking, the authors enforce architectural parity by comparing two Transformer-based trackers: STARK for RGB and SDTrack for events. Because current state-of-the-art SNNs are computationally limited, the continuous event stream is discretized into 128 × 128 × T voxel grids. SDTrack is fine-tuned from a pretrained model using GIoU, L1, and location losses with AdamW, a learning rate of 1 × 10^-4, weight decay of 1 × 10^-4, batch size 2, and a single NVIDIA RTX 4090 GPU. Validation performance peaks at epoch 20, so the epoch 20 checkpoint is used. Evaluation follows the one-pass evaluation (OPE) protocol from SkiTB on the official eSkiTB test split of 30 multi-camera sequences.

Why This Matters

This work provides the first controlled setting for event-based tracking in winter sports. It shows that event sensing can be more robust than RGB when static broadcast overlays and clutter dominate a scene, because fixed overlays generate zero events and physical clutter follows global background flow while the skier produces independent motion. The paper also quantifies the domain gap between generic event data and winter-sport event data, showing that fine-tuning on eSkiTB raises SDTrack from 0.312 to 0.711 mean IoU.

Real-world applications include:

• Autonomous ski-filming drones that follow athletes in visually congested alpine terrain.
• Broadcast production and sports analytics that need reliable athlete localization despite overlays and banners.
• Low-bandwidth or edge tracking systems where event-only sensing avoids the bandwidth cost of sensor fusion.
• Winter-sport athlete analysis, including pose estimation and action analysis, where RGB clutter currently degrades annotation and tracking quality.

Industry relevance spans neuromorphic camera makers, sports broadcasting, drone cinematography, and event-based vision companies. The paper argues that in high-clutter regimes, event-only tracking can outperform RGB-only tracking without the bandwidth cost of sensor fusion, which matters for resource-constrained platforms such as drones.

Future Directions

• Validate synthetic eSkiTB results with a smaller real-world event validation set. The paper notes that simultaneous real-world event data collection is a logistical challenge because non-broadcast drone flights are prohibited during competitive events.
• Develop adaptive or multi-scale event representations to address the freestyle skiing failure caused by the fixed 128 × 128 × T voxel grid and aliasing during large aerial excursions.
• Exploit the full 1280 × 720 event resolution instead of the 128 × 128 × T bottleneck imposed by current SNN computational limits.
• Test whether hybrid RGB-event systems add value under clutter as an open question, although this paper deliberately excludes hybrid baselines to isolate the marginal utility of the asynchronous modality.

Target Audience

Researchers and practitioners in event-based vision, neuromorphic computing, visual tracking, spiking neural networks, sports analytics, and autonomous drone cinematography. It is most useful for readers already familiar with tracking benchmarks, event cameras, and Transformer or SNN architectures, but the paper’s dataset and benchmarking protocol are also relevant to dataset creators working on winter sports or broadcast footage.

Authors’ abstract

Tracking skiers in RGB broadcast footage is challenging due to motion blur, static overlays, and clutter that obscure the fast-moving athlete. Event cameras, with their asynchronous contrast sensing, offer natural robustness to such artifacts, yet a controlled benchmark for winter-sport tracking has been missing. We introduce event SkiTB (eSkiTB), a synthetic event-based ski tracking dataset generated from SkiTB using direct video-to-event conversion without neural interpolation, enabling an iso-informational comparison between RGB and event modalities. Benchmarking SDTrack (spiking transformer) against STARK (RGB transformer), we find that event-based tracking is substantially resilient to broadcast clutter in scenes dominated by static overlays, achieving 0.685 IoU, outperforming RGB by +20.0 points. Across the dataset, SDTrack attains a mean IoU of 0.711, demonstrating that temporal contrast is a reliable cue for tracking ballistic motion in visually congested environments. eSkiTB establishes the first controlled setting for event-based tracking in winter sports and highlights the promise of event cameras for ski tracking. The dataset and code will be released at https://github.com/eventbasedvision/eSkiTB.