Web-Scale Collection of Video Data for 4D Animal Reconstruction

AI summary

Overview

Research area: Computer vision for animals, specifically large-scale web video dataset construction and 4D (3D shape plus motion over time) quadruped reconstruction from monocular, in-the-wild video.

Technical level: Intermediate. Readers will benefit from familiarity with 3D animal reconstruction, pose estimation, and common 2D evaluation metrics such as IoU and PCK, though the paper explains each.

Scope: The paper introduces an automated YouTube-scraping pipeline that produces object-centric animal video clips with auxiliary annotations, uses it to build the Animal-in-Motion (AiM) 4D reconstruction benchmark of 230 sequences (11,061 frames), and presents 4D-Fauna, a model-free 4D reconstruction baseline obtained by optimizing 3D-Fauna at the sequence level.

What This Paper Is About

Progress in animal shape and motion analysis depends on large-scale data, but existing animal video datasets are small (as few as 2.4K 15-frame clips) and are not processed into object-centric form with the masks, keypoints, and features that 3D/4D reconstruction methods need. The only dataset described as truly suitable for 4D animal reconstruction contains just 11 videos. The authors build a fully automated pipeline that turns noisy YouTube videos into ready-to-use object-centric clips, and use it to create a 4D quadruped reconstruction benchmark plus a baseline method.

Key Contributions

1. A unified, automated data pipeline that scrapes noisy YouTube videos and processes them into object-centric clips prepared for downstream tasks such as 4D animal reconstruction, without human annotation (verification requires only minimal human effort).
2. Animal-in-Motion (AiM), described as the first benchmark evaluation dataset for 4D quadruped reconstruction: 230 manually filtered sequences totaling 11,061 frames with clean silhouettes and keypoints.
3. 4D-Fauna, a new model-free 4D animal reconstruction baseline that adapts 3D-Fauna with additional keypoint supervision and losses (including smoothness terms) to enable per-sequence optimization.
4. An analysis of benchmarking results for current model-based and model-free methods, revealing a gap between 2D metrics and perceptual 3D quality and suggesting directions for better evaluation design.

Main Findings

• Dataset scale: Using the pipeline, the authors collected and processed 29,979 animal videos totaling 2,046,414 frames (described in the abstract as 30K videos and 2M frames), starting from 23 common animal categories. This is an order of magnitude more than prior works.
• Prior dataset limitations: The largest prior animal video dataset cited, APT-36K, contains only 2.4K videos of 15 frames each; BADJA, the only existing dataset described as suitable for 4D animal reconstruction, has only 11 videos.
• Benchmark composition: Animal-in-Motion contains 230 human-validated videos with 11,061 frames, curated at 10 videos per category.
• Model-based methods win on 2D metrics: SMALify achieves the best score on every reported metric (IoU 0.867, PCK@0.1 0.954, PCK@0.05 0.787, KT-PCK@0.1 0.623, KT-PCK@0.05 0.372, MPJVE 0.023), but the authors report it produces unnatural poses and shapes, including incorrect poses that still fit the 2D silhouette, arbitrary elongation of body parts in depth, legs bending sideways, and drastic deformation to fit a frontal-view image.
• Model-free methods look better but score lower: 3D-Fauna yields more natural reconstructions yet scores 0.670 IoU, 0.470 PCK@0.1, 0.177 PCK@0.05, 0.329 KT-PCK@0.1, 0.130 KT-PCK@0.05, and 0.058 MPJVE. AniMer, a model-based feed-forward method, scores 0.677 IoU, 0.537 PCK@0.1, 0.199 PCK@0.05, 0.566 KT-PCK@0.1, 0.256 KT-PCK@0.05, and 0.038 MPJVE.
• 4D-Fauna improves on 3D-Fauna across all metrics: 0.814 IoU, 0.664 PCK@0.1, 0.317 PCK@0.05, 0.418 KT-PCK@0.1, 0.193 KT-PCK@0.05, and 0.044 MPJVE (lower MPJVE is better).
• Qualitative gains from sequence optimization: 3D-Fauna exhibits sudden leg switching between frames 2 and 4 in the comparison figure, which 4D-Fauna resolves; smoother motion and more accurate camera and animal pose fitting result from mask supervision, keypoint supervision, and smoothness losses.
• Evaluation gap identified: Because 2D metrics can be satisfied by incorrect 3D solutions, the authors conclude that 2D-based metrics widely used by the community are limited and that 3D-aware evaluation and qualitative assessment are needed.
• MPJVE is newly adopted here: Since the paper is the first to focus specifically on 4D animal reconstruction, it states there are no established temporal metrics and adopts Mean Per-Joint Velocity Error from human motion estimation.
• Limitations stated by the authors: Automatically processed data is not perfectly clean and still requires manual validation for reliable benchmarking; the benchmark relies on 2D projection-based metrics subject to view ambiguities; the baseline shows only limited understanding of temporal coherence.

Methodology in Plain English

The authors built a four-stage data engine backed by a database of intermediate metadata so multiple stages can run in parallel.

Stage 1, raw video collection: Starting from an animal category such as horse, GPT generates sub-category breeds (for example, Clydesdale and Mustang) and context phrases (for example, racing competition, in a farm). These are randomly combined into search queries, and videos are queried and downloaded from YouTube using Selenium Webdriver and pytube.

Stage 2, preprocessing: Videos are split into clips at shot changes with PySceneDetect, because tracking algorithms can falsely associate objects across shots. CLIP and an average CLIPScore against a caption such as "a photo of a horse" are used to discard clips that do not clearly depict the target animal. All clips are downsampled to 10 frames per second.

Stage 3, tracking: Grounded-SAM-2, which uses GroundingDINO for bounding-box detection and SAM-2 for tracking, is run in an iterative grounding-tracking loop to follow instances over long spans and pick up newly appearing animals. Track proposals are then filtered: frames with substantially overlapping animals (thresholded by IoU between instances) are removed; frames where the animal's bounding-box area is less than 1/4 of the final crop size (described as 256x256 for a 512x512 crop) are removed; frames where the bounding box is too close to the border, indicating truncation, are removed; tracks where the IoU between adjacent frames drops are cut to prevent identity switches or tracking through fades. A temporal postprocessing step splits and merges tracks according to minimum length, maximum length, and allowed gap parameters, resegmenting missing masks with SAM-2. Final square object-centric crops are centered on the animal bounding box, sized by a predefined ratio to mask area, smoothed with a moving average, and standardized in resolution. A randomly selected crop per track is passed to GPT to remove false detections or heavily occluded animals.

Stage 4, feature extraction: Off-the-shelf models add annotations: ViTPose++ for keypoints, DINOv2 for image features (with PCA DINO features used by model-free methods), SEA-RAFT for optical flow, and Depth Anything V2 for depth. Occlusion boundaries are computed by comparing depth at dilated and eroded mask boundaries to decide whether pixels outside the silhouette are foreground (occluding) or background. Optical flow and occlusion boundaries can optionally filter for samples with greater motion and minimal occlusion.

Benchmark construction: Human annotators accept or reject samples by reviewing three visualizations (RGB video, RGB video with per-frame masks, RGB video with per-frame keypoints), using criteria about occlusion, recognizable and smooth motion, smooth camera movement, correct segmentation without missing body parts, and accurate, smooth keypoints. This yields 10 videos per category, 230 videos, 11,061 frames.

Evaluation: The authors adopt silhouette IoU, PCK at thresholds 0.1 and 0.05 (normalized by the square root of the ground-truth mask silhouette area), keypoint transfer PCK (projecting 2D ground-truth keypoints onto the reconstructed 3D surface and reprojecting to a target view), and MPJVE for motion.

The baseline: 3D-Fauna learns a prior shape bank from pan-category images and reconstructs feed-forward, whereas model-based methods optimize a mesh to fit 2D labels. 4D-Fauna takes the pretrained 3D-Fauna and optimizes per sequence. It adds explicit 2D keypoint supervision to fix pose ambiguity such as wrong leg ordering in gait cycles, applies temporal smoothness losses on camera pose change and animal pose velocity, and directly optimizes per-frame camera pose and articulation parameters initialized from the pretrained predictors so the sequence can be overfit efficiently.

Why This Matters

Impact on research: The work shifts animal 4D reconstruction from small, controlled, or manually annotated datasets toward scalable web-sourced data. It supplies a benchmark where none existed for 4D quadruped reconstruction, and its central negative finding, that the 2D metrics the community relies on can reward implausible 3D geometry, is a prompt for the field to design 3D-aware metrics. By combining the overfitting strength of model-based optimization with the natural shape priors of model-free models, 4D-Fauna also sketches a hybrid recipe that later work can build on.

Real-world applications:

• Wildlife research and conservation, where markerless analysis of animal shape and motion from ordinary video could support behavioral study without capture or tags.
• Biomechanics and veterinary gait analysis, examining how animals move over time from single-view footage.
• Robotics and animation, where plausible quadruped shape and motion capture from video can inform character animation or legged robot motion references.
• Ecological monitoring at scale, since the pipeline can mine large public video archives for many animal categories automatically.

Industry relevance: The pipeline is a template for automated, web-scale dataset construction in specialized domains beyond generic video classification, and it demonstrates value in combining large vision foundation models (CLIP, GroundingDINO, SAM-2, ViTPose++, DINOv2, SEA-RAFT, Depth Anything V2) as labeling components. It is also relevant to media and VFX pipelines that need animal motion and shape from ordinary footage, and to any team that wants benchmark-ready data without large annotation budgets.

Future Directions

• 3D-aware evaluation metrics. The authors explicitly call for more robust, 3D-aware metrics, since current 2D projection-based metrics are limited by view ambiguities and do not fully capture 3D reconstruction quality.
• Stronger temporal modeling. The authors note that 4D-Fauna shows only limited understanding of temporal coherence and suggest more expressive paradigms such as autoregressive models to capture inter-frame dynamics.
• Cleaner automatic data. Because the automatically processed data still needs manual validation, improving filtering and quality control in the pipeline is an open direction.
• Extending beyond quadruped animals and beyond shape and motion. The pipeline targets categories generally and the benchmark targets quadruped 4D reconstruction; scaling the benchmark and method to other animal groups and downstream tasks such as pose estimation and tracking remains open.

Target Audience

Researchers and practitioners in computer vision working on animal pose estimation, tracking, 3D/4D reconstruction, and dataset or benchmark construction; wildlife and biomechanics researchers seeking markerless analysis tools; and engineers building automated data pipelines who want an example of assembling off-the-shelf foundation models into a large-scale, object-centric annotation system.

Authors’ abstract

Computer vision for animals holds great promise for wildlife research but often depends on large-scale data, while existing collection methods rely on controlled capture setups. Recent data-driven approaches show the potential of single-view, non-invasive analysis, yet current animal video datasets are limited--offering as few as 2.4K 15-frame clips and lacking key processing for animal-centric 3D/4D tasks. We introduce an automated pipeline that mines YouTube videos and processes them into object-centric clips, along with auxiliary annotations valuable for downstream tasks like pose estimation, tracking, and 3D/4D reconstruction. Using this pipeline, we amass 30K videos (2M frames)--an order of magnitude more than prior works. To demonstrate its utility, we focus on the 4D quadruped animal reconstruction task. To support this task, we present Animal-in-Motion (AiM), a benchmark of 230 manually filtered sequences with 11K frames showcasing clean, diverse animal motions. We evaluate state-of-the-art model-based and model-free methods on Animal-in-Motion, finding that 2D metrics favor the former despite unrealistic 3D shapes, while the latter yields more natural reconstructions but scores lower--revealing a gap in current evaluation. To address this, we enhance a recent model-free approach with sequence-level optimization, establishing the first 4D animal reconstruction baseline. Together, our pipeline, benchmark, and baseline aim to advance large-scale, markerless 4D animal reconstruction and related tasks from in-the-wild videos. Code and datasets are available at https://github.com/briannlongzhao/Animal-in-Motion.