PriVi: Towards A General-Purpose Video Model For Primate Behavior In The Wild

AI summary

Overview

• Research area: Computer vision / self-supervised video representation learning, applied to animal behavior recognition (primate ethology).
• Technical level: Advanced (assumes familiarity with self-supervised learning, masked latent prediction, and video action recognition benchmarks).
• Scope: The paper introduces PriVi, a 424-hour primate-centric video pretraining dataset and curation pipeline, and shows that "domain-level" continued pretraining of V-JEPA on it outperforms prior work on four primate behavior recognition benchmarks.

What This Paper Is About

Computer vision could help researchers study non-human primate behavior, but existing methods mostly start from human-centric pretrained models and are specialized to a single dataset, so they do not generalize. The authors argue the bottleneck is data, not architecture, and build a large, diverse primate video dataset so that one pretrained model can serve many primate behavior tasks. Their goal is "domain-level pretraining" — pretraining on diverse primate video without including the target dataset — and they test whether this works for video models.

Key Contributions

1. A data curation pipeline that requires neither a seed dataset nor text captions. It uses CLIP-embedding-based relevance filtering and zero-shot primate detection (GroundingDINO) to build the corpus automatically.
2. PriVi, a 424-hour primate video dataset, combining 174 hours of curated research and observational footage from 11 source datasets with 250 hours of filtered web (YouTube) video; 720,000 three-second snippets in total with bounding boxes and CLIP keyframe embeddings.
3. A frozen evaluation setup using a lightweight attentive classifier (220k parameters) with a downprojection to 64 dimensions and one learned CLS token per class, which scales well to small labeled datasets.
4. The first demonstration that domain-level pretraining improves video models, with results surpassing prior work on ChimpACT, PanAf500, BaboonLand, and ChimpBehave, including fully finetuned baselines.

Main Findings

• PriVi beats human-centric pretraining across all four benchmarks. On test sets, V-JEPA initialized from VideoMix2M reaches 36.33 mAP on ChimpACT and 82.96 accuracy / 56.69 B-Acc on PanAf500, while PriVi pretraining reaches 39.25 mAP and 86.74 / 62.75; adding dataset-level pretraining (PriVi + DaLP) reaches 40.00 mAP and 85.01 / 62.96.
• Gains are consistent on BaboonLand and ChimpBehave too. PriVi + DaLP reaches 96.02 accuracy / 75.14 B-Acc on ChimpBehave and 76.42 accuracy / 38.57 B-Acc on BaboonLand, versus 94.99 / 68.41 and 74.91 / 26.99 for the V-JEPA-only classifier.
• Every part of the dataset helps. On validation sets, V-JEPA alone gets 32.00 mAP / 47.88 mAP-w on ChimpACT and 84.70 Acc / 71.95 B-Acc on PanAf500; YT-Filtered alone gets 37.88 / 53.70 and 89.17 / 76.33; R&O alone gets 33.01 / 49.58 and 89.35 / 73.85; combined as PriVi they reach 38.75 / 54.32 and 89.65 / 79.95.
• Curation matters. Randomly selected YouTube video (YT-Random, 280.0 hours) reaches only 33.87 mAP on ChimpACT and 86.13 / 71.61 on PanAf500, below the 37.88 and 89.17 / 76.33 of the relevance-filtered YT-Filtered (250.0 hours). Even uncurated YouTube data mostly outperformed the original V-JEPA baseline.
• Domain-level pretraining beats dataset-level pretraining. Continuing pretraining only on ChimpACT (1.4 hours) gives 35.86 mAP / 51.12 mAP-w there but 28.94 mAP / 45.22 mAP-w on ChimpACT when pretraining on PanAf500 (1.5 hours) instead. PriVi pretraining (424.0 hours) performs well on both, and the authors observed no transfer between ChimpACT and PanAf500.
• A tiny frozen classifier outperforms much larger specialist models. The 220k-parameter attentive classifier beats ChimpVLM (167M parameters), VideoMAEv2, X3D (27.05 mAP on ChimpACT, 80.00 / 50.35 on PanAf500), UniformerV2-B, InternVideo-L, and VideoPrism-g on the reported metrics.
• It works with predicted detections. Using the zero-shot detector SAM 3 instead of ground-truth boxes, the method reaches 30.76 mAP / 41.28 mAP-w on ChimpACT without GT detections, versus 25.35 / 40.23 for the fully finetuned AlphaChimp.
• Label efficiency. On PanAf500 the method loses only 4.4% accuracy going from 100% to 10% of training data, and with 10% of labels it outperforms X3D trained on 100%. On ChimpACT the method at 25% still outperforms X3D at 100%. PanAf500 contains 400 training video sequences while ChimpACT contains only 127.
• Ablations confirm design choices. Removing primate-centric cropping drops validation results from 38.75 / 54.32 and 89.65 / 79.95 to 32.62 / 47.74 and 85.32 / 72.93. Removing the linear downprojection raises classifier parameters from 0.22M to 37.84M and drops ChimpACT mAP to 30.15; a single layer with 0.12M parameters gives 35.68 mAP.
• Data curation pipeline statistics. The relevance classifier is a 2-layer MLP trained on 2,500 manually labeled images, achieving 82.8% recall, 90.3% precision, and 95.9% ROC-AUC on held-out validation. The zero-shot primate detector reaches 82.7 mAP-All on the PanAf500 primate detection task.

Methodology in Plain English

The authors first gather raw video from two sources: past and ongoing behavioral research projects (721 hours from 11 source datasets, covering wild, semi-free-ranging, and captive settings) and YouTube playlists (458 hours from 19 manually selected playlists). Rather than hand-curating, they automate filtering: a cut detector removes abrupt scene changes, video is chunked into 3-second snippets at a 2-second stride, and a small classifier built on CLIP embeddings of center frames decides which snippets actually show primates doing something relevant. A zero-shot object detector then draws primate bounding boxes and empty chunks are discarded; research footage is subsampled by target proportions because source datasets vary widely in size.

They then continue pretraining of V-JEPA, a self-supervised video model that predicts the latent representation of masked tokens rather than pixels, on this dataset. The model takes 16 frames at 224x224 resolution, tokenized into 1568 tokens of dimension 1024, and inputs are cropped around detected primate bounding boxes. The context encoder (304M parameters) and predictor (22M) are trained while the target encoder is updated as an exponential moving average. Pretraining runs 75k steps with batch size 80 (about eight epochs over PriVi) on four A100 GPUs for 11 hours.

For each target dataset, the frozen encoder is evaluated with a small trained classifier head: patch tokens are downprojected from 1024 to 64 dimensions, C learned class tokens are concatenated, three self-attention blocks are applied, and softmax or sigmoid produces the prediction. Optionally, an extra unsupervised "dataset-level pretraining" step of 10k steps runs on the target dataset's own (unlabeled) video before the classifier is trained.

Why This Matters

This work shifts the field from building one specialist model per dataset toward a shared, general-purpose representation for primate behavior. It shows that collecting and curating in-domain unlabeled data — which is abundant and cheap in behavioral research — can substitute for large amounts of expert-labeled annotation.

Real-world applications:

• Behavioral and cognitive research: researchers can run behavior recognition on far larger video corpora than manual annotation allows, reducing expert time and human bias.
• Conservation monitoring: camera traps and drone footage (as in PanAf500 and BaboonLand) can be screened automatically for species-typical behaviors across tropical Africa and other habitats.
• Zoo and sanctuary management: recordings from indoor and semi-free-ranging enclosures (ChimpBehave, Leipzig zoo, Basel zoo) can be analyzed without ground-truth bounding boxes.
• Field experiments: datasets with experimental interventions such as feeding boxes (Kirindy, Simenti) can be analyzed with the same pretrained backbone.

Industry relevance: the paper supports a general "domain-level pretraining" recipe that could apply beyond primates to any animal or specialized domain where web-scale captions do not exist, and its frozen 220k-parameter classifier shows that small, cheap heads can compete with large finetuned models.

Future Directions

• Broadening coverage. The authors note the dataset captures only 11 different research setups and that evaluation was limited to chimpanzees and baboons due to the scarcity of labeled datasets.
• Extending domain-level pretraining to other animal domains, testing whether the recipe transfers to non-primate species and to animal behavior in general.
• Scaling dataset size and diversity beyond 424 hours and 11 source datasets, and testing whether the trend of improving representation quality continues.
• Improving end-to-end pipelines, since the evaluation with SAM 3 predicted boxes suggests detection quality and false positives remain a practical issue worth addressing.

Target Audience

Researchers and practitioners in computer vision working on video foundation models, self-supervised learning, and data curation; behavioral ecologists, primatologists, and conservation scientists who want automated analysis of large video archives; and machine learning engineers interested in domain-level pretraining strategies for specialized, caption-poor domains.

Authors’ abstract

Non-human primates are our closest living relatives, and analyzing their behavior is central to research in cognition, evolution, and conservation. Computer vision could greatly aid this research, but existing methods often rely on human-centric pretrained models and focus on single datasets, which limits generalization. We address this limitation by shifting from a model-centric to a data-centric approach and introduce PriVi, a large-scale primate-centric video pretraining dataset. PriVi contains 424 hours of curated video, combining 174 hours from behavioral research across 11 settings with 250 hours of diverse web-sourced footage, assembled through a scalable data curation pipeline. We continue pretraining V-JEPA, a large-scale video model, on PriVi to learn primate-specific representations and evaluate it using a lightweight frozen classifier. Across four benchmark datasets, ChimpACT, PanAf500, BaboonLand, and ChimpBehave, our approach consistently outperforms prior work, including fully finetuned baselines, and scales favorably with fewer labels. These results demonstrate for the first time that domain-level pretraining, where pretraining is conducted on similar data but not the target dataset itself, works for video models. Our primate-centric pretraining substantially improves data efficiency and generalization, making it a promising approach for low-label applications. Dataset, code, and models are available: https://privi.eckerlab.org