Promptable Animal Pose Tracking Across Species

AI summary

Overview

• Research area: Computer vision for ecology and wildlife monitoring, specifically animal pose estimation and tracking (APE/APT) using vision foundation models.
• Technical level: Intermediate. Readers need some familiarity with keypoint/pose estimation, feature correspondence matching, and vision foundation models (DINOv3, diffusion features), but the paper's architecture is described at a conceptual level.
• Scope: The paper proposes a promptable animal pose tracking framework with two routes — one supervised and one unsupervised — that tracks user-specified keypoints across video frames from a single annotated reference frame, evaluated on APTv2 and TigDog.

What This Paper Is About

Animal pose tracking matters for wildlife monitoring and conservation, but it is hard because species differ enormously in shape and behaviour and annotated animal data is scarce. Existing methods either optimise generic keypoints on annotated datasets like APTv2 with poor generalisation, or track custom keypoints with visual tracking at a cost in accuracy. The paper's goal is to show that vision foundation models trained on large datasets can track animal pose with limited labelled data, by letting a user annotate keypoints in just one reference frame and propagating them through the rest of the video.

Key Contributions

1. A Promptable Animal Pose Tracking framework that leverages diverse pretrained foundation-level visual representations instead of conventional annotation-intensive pose estimation pipelines.
2. A supervised Animal Pose Tracking model that combines a keypoint encoder with pretrained foundation-model representations, so the model explicitly focuses on the structural priors given by the reference-frame keypoints.
3. An unsupervised Animal Pose Tracking model that pairs pretrained visual features with a lightweight matcher carrying drift-correction constraints, enabling robust tracking of annotated keypoints with no task-specific training.
4. Comparative and ablation analysis on APTv2 and TigDog showing a trade-off between accuracy (supervised route) and cross-species generalisation (unsupervised route).

Main Findings

• Diffusion features dominate unsupervised matching: On APTv2, Diffusion Hyperfeatures (DHf) with a species-type prompt reach 86.6 PCK@0.1_img and 70.5 PCK@0.05_img, versus 84.0 / 61.9 for CleanDIFT (CD) with prompt, 69.4 / 43.9 for DINOv3 all-layer concatenation, and 42.9 / 21.9 for BioCLIP.
• Species-name prompts help: Diffusion-based features consistently benefit from species-name prompts over empty prompts — for example DHf improves from 85.1 to 86.6 (PCK@0.1_img) and 68.6 to 70.5 (PCK@0.05_img) on APTv2, and from 81.3 to 83.0 and 67.4 to 69.7 on TigDog.
• Competitive with dedicated point trackers: Although the comparison is deliberately asymmetric (TAP methods are trained for point tracking on large-scale tracking and optical-flow datasets and use temporal information; this method does pairwise frame matching with no tracking-specific training and no temporal context), the unsupervised route reaches 86.6 / 70.5 on APTv2, compared with AllTracker 88.6 / 74.5, CoTracker3 87.1 / 74.5, BootsTAPIR v2 85.6 / 72.3, and TAPIR 84.1 / 70.1.
• Distal limbs are hardest: Binning APTv2's 17 keypoints into head (3), central body (6), and limb extremities (8), limb extremities show the lowest PCK@0.05_img for every backbone — from 46.0 (DINOv3-Last) to 67.2 (DHf with prompt). For CleanDIFT- and Diffusion Hyperfeatures-based backbones, distal keypoints fall more than 10%, and in some cases over 20%, below each backbone's best region.
• Species-level differences are large: The best-performing species on APTv2 with DHf with prompt are Giraffe (83.1), Zebra (82.4), and Cow (79.8) in PCK@0.05_img; the worst are Fox (62.2), Chimpanzee (61.1), and Gorilla (60.3). The standard deviation across all 30 species ranges from 6.30 to 8.34 depending on the backbone.
• Drift correction consistently helps the unsupervised route: Ablation shows gains in PCK@0.1_img of +14.4 for BioCLIP (28.5 to 42.9), +3.3 for DINOv3 last-layer (64.6 to 67.9), +2.8 for DINOv3 all-layer (66.6 to 69.4), +2.3 for CleanDIFT without prompt (80.3 to 82.6), and +1.3 for DHf without prompt (83.8 to 85.1).
• Supervised route is accurate and lightweight: On APTv2 the supervised route with DHf (no species prompt) reaches 99.1 PCK@0.1_img and 94.7 PCK@0.05_img, versus HRNet W48 at 98.5 / 95.2, HRNet W32 at 98.4 / 94.8, HRFormer Base at 98.3 / 94.5, SimpleBaseline ResNet50 at 98.2 / 94.1, and the DINOv3 variant (2-layer concatenation) at 91.8 / 74.0. The baselines were pretrained on Microsoft COCO for 200 epochs and fine-tuned on APTv2, whereas the proposed backbone stays frozen and only the keypoint prompt encoder, feature projection module, and matcher are trained for 50 epochs.
• Supervised learning specialises, unsupervised transfers: In the leave-one-out study over six families (PCK@0.05_img), seen-family averages range from 80.4 to 96.8 while unseen-family scores drop to between 41.9 and 55.8, with seen-unseen gaps (Δ) of 29.0, 40.4, 52.4, 49.6, 41.9, and 40.7. The leave-one-out results are often lower than the unsupervised route on the same families, suggesting supervised optimisation specialises to the training distribution.
• Performance decays with temporal distance: Because each target frame is matched directly to the reference frame rather than propagated sequentially, accuracy gradually decreases as the temporal distance from the reference frame (r = 0) grows; selecting a reference frame from the middle of the sequence results in less pronounced degradation in both temporal directions. Note that the paper does not report inference speed, model size, or runtime.

Methodology in Plain English

The input is one RGB video plus keypoint annotations drawn by a user on a single frame, called the reference frame. The system propagates those keypoints to every other frame.

First, each frame is passed through a pretrained vision foundation model to get dense per-pixel feature maps. The authors tried DINOv3, BioCLIP, CleanDIFT, and Diffusion Hyperfeatures, and found Diffusion Hyperfeatures to be the most effective representation for dense correspondence in their setting.

From there, the pipeline splits in two:

• Unsupervised route (no training at all). For each annotated keypoint, the method finds the location in the target frame whose feature is most similar to the reference keypoint's feature (nearest-neighbour matching). A drift-correction step uses dataset bounding boxes to confine the search region when an animal's limbs look similar enough to cause keypoints to jump to a different animal. Because instance identities are ambiguous across frames, boxes are matched softly by the spatial proximity of their centres rather than by explicit box tracking. This step is optional, and the TigDog results are reported without it.
• Supervised route (lightweight fine-tuning). A convolutional projection adapts the frozen features. The reference keypoints are turned into Gaussian heatmaps, which pass through a lightweight keypoint prompt encoder that produces a single structural attention map. That map modulates the reference-frame features so the model focuses on keypoint-relevant structure. A matcher, adapted from RoMa v2's coarse matching module, then estimates cross-frame correspondences and confidences. Only the keypoint encoder, the projection, and the matcher are trained — the foundation backbone stays frozen.

Evaluation uses the standard Percentage of Correct Keypoints (PCK) metric with image-normalized distances, reported at PCK@0.1_img and PCK@0.05_img. Training and testing used a Quadro RTX 6000 GPU.

Why This Matters

• Impact on research: The paper shows that frozen visual foundation-model features already capture most of the correspondence ability that point tracking requires, and that a simple matching pipeline can approach large end-to-end trackers without being trained for the task. It also reframes the field's goal from fixed-skeleton supervised pose estimation toward flexible, promptable tools, and explicitly characterises the accuracy-versus-generalisation trade-off between supervised and unsupervised routes.
• Real-world applications:
  • Wildlife monitoring and conservation research, including automated analysis of camera-trap or field video.
  • Livestock monitoring.
  • Laboratory animal behaviour analysis, where downstream tasks such as animal action recognition depend on pose tracks.
  • Online or streaming animal monitoring, since the pairwise design operates without lookahead.
• Industry relevance: The method requires only standard RGB video and a single annotated frame, avoiding multi-view setups, non-RGB modalities, and large-scale dense annotation. Substituting expensive labelling with a frozen backbone and a short fine-tuning stage (50 epochs, with only three components trained) lowers the cost of building applied animal-tracking systems. The unsupervised route additionally supports test-time deployment with no task-specific training.

Future Directions

• Improving the unsupervised route through more advanced correspondence estimation and matching strategies, which the authors explicitly leave for future work.
• Closing the gap on limb extremities, where fine-grained articulated motion in distal parts remains unsolved by current foundation-model features due to fast motion, self-occlusion, and left/right appearance ambiguity.
• Reducing the seen-versus-unseen gap in the supervised route, since leave-one-out results suggest supervised optimisation specialises to the training distribution and transfers less well to novel families than the unsupervised route.
• Deciding when to use which route — the paper frames supervised and unsupervised as complementary, which raises the question of how to combine or select between them for a given species or setting. The paper does not report work on unifying them.

Target Audience

• Computer vision researchers working on pose estimation, correspondence matching, and foundation-model transfer.
• Ecologists, conservation scientists, and wildlife researchers who need practical tracking tools rather than fixed-skeleton models.
• Practitioners building animal monitoring systems for livestock, laboratories, or camera-trap pipelines.
• Graduate students looking for a compact case study in adapting frozen foundation-model features to a new domain under limited annotation.

Authors’ abstract

Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that vision foundation models trained on large datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in videos. The supervised approach delivers superior tracking accuracy by employing a keypoint prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.