RacketVision: A Multiple Racket Sports Benchmark for Unified Ball and Racket Analysis

AI summary

Overview

Research area: Computer vision for sports analytics — specifically multi-sport ball tracking, articulated object (racket) pose estimation, and trajectory forecasting.

Technical level: Intermediate. The paper assumes familiarity with standard detection/pose/tracking architectures (RTMDet, YOLO11, RTMPose, TrackNetV3, LSTM, Transformer), but its core argument — that fusion architecture matters more than raw multi-modal data — is accessible to a general reader.

Scope: The paper introduces RacketVision, a large-scale video benchmark covering table tennis, tennis, and badminton with joint ball and racket annotations, defines three interconnected tasks on it, and reports baseline results showing that Cross-Attention fusion is required for racket pose data to improve trajectory prediction.

What This Paper Is About

Existing racket-sport datasets focus narrowly on ball tracking within one sport at a time, and none annotate the racket even though it is the object that determines where the ball goes next. The authors build a unified dataset spanning three racket sports and add the first large-scale racket pose annotations, then define three linked tasks — fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting — to study how ball and racket information should be combined. The goal is to give the field a shared benchmark for multi-modal, dynamic human-object interaction in sports video.

Key Contributions

1. RacketVision dataset and benchmark. A large-scale, multi-sport resource with 1,672 video clips and 435,179 frames (12,755 seconds) spanning badminton, tennis, and table tennis, providing detailed annotations for both ball position and racket pose to support cross-sport analysis. The authors describe it as the first to provide large-scale racket pose annotations (denoted "R" in their dataset comparison table) alongside traditional ball positions ("B").

2. Three interconnected task definitions. Ball tracking, racket pose estimation, and ball trajectory prediction given history, formulated as a progressive pipeline from low-level perception to high-level prediction.

3. Strong baselines plus a fusion finding. Extensive evaluations across model architectures, multi-sport vs. single-sport training, background modeling, and multi-frame inputs, revealing that naive concatenation of racket pose features degrades trajectory prediction while a Cross-Attention mechanism reverses that and surpasses strong unimodal ball-only baselines.

Main Findings

• Multi-sport training improves generalization on perception tasks. Comparing the best multi-sport model (MS-TrackNetV3, #F=4) with the best single-sport model (TrackNetV3, #F=4), multi-sport training raises mAP by 19.2% in tennis (81.9 vs. 68.7) and 14.6% in badminton (83.1 vs. 72.5). The single-sport model retains a slight edge in tennis precision (0.962 vs. 0.945) and tennis MDE (1.66 vs. 1.96), suggesting a trade-off between broad robustness and single-domain localization precision.

• Background modeling sharply reduces localization error. Adding a median frame reduces MDE by 54.0% for table tennis, 61.4% for tennis, and 54.8% for badminton (TrackNetV3, #F=1, with vs. without background modeling).

• Multi-frame input helps detection but not always precision. With MS-TrackNetV3 in tennis, 4 frames raise Recall (0.880 vs. 0.820) while the single-frame version achieves a slightly better MDE (1.70 vs. 1.96). The authors attribute this to occasional jitter or motion blur introduced by temporal context.

• Best overall ball tracking results come from MS-TrackNetV3 with background modeling and #F=4: table tennis Precision 0.924, Recall 0.762, MDE 3.41, mAP 71.1; tennis 0.945, 0.880, 1.96, 81.9; badminton 0.915, 0.865, 1.54, 83.1.

• Multi-sport training also helps racket pose estimation. The multi-sport RTMPose improves PCK@0.2 by 6.17% (table tennis), 6.36% (badminton), and 5.97% (tennis). Tennis reaches the highest PCK@0.2 at 89.69% under multi-sport training; badminton has the lowest MPJPE (5.00 px) and highest mOKS (0.668).

• Side keypoints are the hard part of racket pose. Structural keypoints (top, bottom, handle) exceed 92% PCK@0.2, while side keypoints (left, right) range from 64.85% to 80.11%, which the authors attribute to occlusion by the hand grip, motion blur, and sensitivity to viewing angle.

• Naive fusion hurts trajectory prediction. Concatenating racket pose features consistently performs worse than the ball-only baseline for both LSTM and Transformer backbones. In short-trajectory table tennis, LSTM ADE rises from 41.9 (ball-only) to 58.1 (concat); the authors suggest much of the data captures the ball in mid-flight where racket information is absent or irrelevant.

• Cross-Attention fusion is what unlocks racket data. The LSTM with Cross-Attention is the best-performing model overall, beating the ball-only baseline in every sport and setting, e.g. long-trajectory table tennis ADE 101.3 vs. 113.9, tennis 55.5 vs. 62.5, badminton 114.6 vs. 118.7.

• The overall gains are real but modest, and that is explained. In short-trajectory badminton, ADE improves only from 37.5 to 37.0. The authors attribute this to many samples consisting entirely of the ball in flight, where the Cross-Attention model correctly learns to ignore the racket modality and behaves like the ball-only model.

Methodology in Plain English

The authors collected 942 top-level professional game broadcasts from YouTube across the three sports. Crowd-sourced annotators first cut raw video into valid clips of 5–10 seconds where the ball is actively in play. Rather than labeling every frame, they sampled 20% of the frames in each clip evenly and had a separate group of annotators label those: the ball as a single point with a visibility flag, and each racket with a bounding box plus five keypoints (top, bottom, handle, left, right) tailored to each racket type. A median frame is also provided per clip as a stable background reference.

The three tasks form a pipeline. Ball tracking is evaluated with one frame or with the target frame plus five preceding frames. Racket pose estimation predicts a bounding box and five keypoints from a single frame. Trajectory prediction takes a history of ball positions — optionally with racket keypoints — and predicts future ball positions, under a short setting (history 20, future 5) and a long setting (history 80, future 20).

The pipeline works in two stages: the ball tracker and racket pose estimator are trained on the sparse human annotations, then run over full clips to produce dense per-frame predictions ("soft labels"). Those continuous sequences are split into history and future segments to train the trajectory predictor. For trajectory prediction the authors test two backbones — a 2-layer LSTM and a 4-layer encoder-only Transformer — under three input/fusion schemes: ball-only, naive concatenation of ball and racket features, and Cross-Attention where the ball trajectory is the Query and the racket pose sequence is the Key and Value, letting the model weight racket information per time step.

Why This Matters

Impact on research. The paper supplies a shared, multi-sport benchmark with a novel annotation type (racket pose), which previous work either lacked or obtained only via specialized hardware such as high-speed or stereo cameras. It also delivers a transferable methodological lesson: adding a second modality is not automatically useful, and the fusion architecture determines whether the extra data helps or hurts. The authors frame this as lowering the barrier for future research on general-purpose models applied to complex, interconnected sports tasks.

Real-world applications (as identified or implied in the paper):

• Tactic analysis and automated coaching insights from predicted ball trajectories.
• Robotics, where trajectory forecasting supports systems that interact with fast-moving objects.
• Neural avatar modeling, which the authors cite as a reason racket-specific annotations matter.
• Broadcast and officiating tooling — ball tracking, hit anticipation, and stroke recognition, all of which the paper situates in the pipeline it formalizes.

Industry relevance. Sports analytics, broadcast augmentation, and robotics are the obvious beneficiaries. The dataset's cross-sport design is also relevant to teams that want one model serving multiple sports rather than a bespoke model per sport, given the measured mAP gains from multi-sport training.

Future Directions

• Better fusion for sparse, event-driven modalities. The paper shows Cross-Attention wins, but also that gains are limited because many samples contain no racket interaction at all. Methods that explicitly detect "event" frames or gate the racket modality could widen the advantage.

• Improving side keypoint accuracy. Left and right keypoints lag far behind top, bottom, and handle (64.85%–80.11% vs. above 92%), driven by grip occlusion, motion blur, and viewing angle. This is an open pose-estimation problem the benchmark now makes measurable.

• Extending beyond the current annotation scope. The dataset provides ball points, racket boxes, and five racket keypoints. Human pose, player identity, and 3D or spin reconstruction — cited as recent research directions — are not part of the current annotation set.

• Resolving remaining scale questions. The paper reports only ball and racket annotations plus the supplied median frames; it does not report per-sport statistics on how many trajectory samples contain an actual racket interaction, which is the variable the authors use to explain the modest headline gains. Quantifying that composition is a natural next step.

Note on internal consistency: Table 1 lists RacketVision with 461 games, while Table 2 and the text report 942 games in total, and Table 1's frame count (435k) matches Table 2's 435,179. The paper does not reconcile this discrepancy.

Target Audience

This paper is most useful to computer vision researchers working on multi-object tracking, pose estimation, or multi-modal fusion; to sports-analytics practitioners who need a public benchmark spanning more than one sport; and to robotics or graphics researchers interested in human-object interaction where the object's pose carries predictive signal. Readers looking for a fully solved trajectory-prediction method will find a strong baseline and an architectural insight rather than a large performance leap, and readers seeking implementation or compute details will not find them reported here.

Authors’ abstract

We introduce RacketVision, a novel dataset and benchmark for advancing computer vision in sports analytics, covering table tennis, tennis, and badminton. The dataset is the first to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. It is designed to tackle three interconnected tasks: fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting. Our evaluation of established baselines reveals a critical insight for multi-modal fusion: while naively concatenating racket pose features degrades performance, a CrossAttention mechanism is essential to unlock their value, leading to trajectory prediction results that surpass strong unimodal baselines. RacketVision provides a versatile resource and a strong starting point for future research in dynamic object tracking, conditional motion forecasting, and multimodal analysis in sports. Project page at https://github.com/OrcustD/RacketVision