VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes

AI summary

Overview

Research area: Computer vision / video foundation model pretraining data pipelines (data curation infrastructure).

Technical level: Intermediate to Advanced. The workflow itself is describable in plain terms, but the paper includes controlled from-scratch pretraining experiments, loss-curve analysis, and benchmark evaluation that assume familiarity with generative and self-supervised video models.

Scope: VidaForge is an open, executable, and traceable five-stage infrastructure for video pretraining data recipes, demonstrated by a coverage-versus-quality study on Wan 2.1-1.3B and V-JEPA 2.1-1B and accompanied by a released 3.14-million-clip dataset.

What This Paper Is About

Leading video foundation models depend on large-scale pretraining data, but the pipelines that produce that data are largely closed: technical reports rarely describe concrete filtering, deduplication, and captioning choices, and almost never provide controlled ablations showing which choices matter under which pretraining objective. As a result, a researcher who wants to test even one focused hypothesis about how a data recipe affects pretraining must first build an entire pipeline from raw videos to a training-ready dataset, while also preserving the per-sample processing record needed to attribute differences between training runs to specific decisions. VidaForge addresses this by making video data recipes executable as a five-stage workflow that saves every stage output, recipe parameter, and sample-level decision, so a decision can be changed to produce an alternative dataset while retaining the trace of how each sample was produced.

Key Contributions

1. VidaForge infrastructure. An open-source, end-to-end system that represents a video data recipe as an executable, traceable chain of five stages — ingestion, segmentation, selection, annotation, and packaging — from raw videos to target-specific training datasets. It ships with runnable recipes, documentation, and a complete video walkthrough covering installation and usage.
2. A coverage-versus-quality study. Using VidaForge, the authors compare data recipes that balance dataset coverage and quality across both generative and self-supervised pretraining, showing that loss-based and task-level evaluations can favor different dataset recipes.
3. VidaForge-3M release. An open dataset of 3.14 million scene-level clips totaling 6,475 hours, with fine-grained annotations and curation signals, built from 800k source videos and intended for video data-recipe research.
4. A capability comparison. A structured comparison of VidaForge against Data-Juicer (with Sandbox), NeMo Curator, and Cosmos Curator under explicit capability definitions, including per-stage processing records, one video pool feeding different model families, recipe-variant-to-training-to-evaluation workflows, and reported experiments spanning both video generation and self-supervised learning.

Main Findings

• Broader coverage wins downstream. Under the same early-pretraining budget, the broader-coverage Mixed recipe achieves the highest downstream benchmark score under all 3 training seeds for both learning objectives — Wan 2.1-1.3B video generation and V-JEPA 2.1-1B self-supervised representation learning.
• VBench results for Wan 2.1-1.3B. Across 3 seeds (mean ± std): Quality — Mixed 71.65 ± 0.74, Selected 70.61 ± 0.51, Rejected 70.25 ± 1.19; Semantic — Mixed 16.60 ± 1.25, Selected 16.28 ± 0.81, Rejected 16.08 ± 0.89; Total — Mixed 60.64 ± 0.83, Selected 59.74 ± 0.55, Rejected 59.42 ± 1.06; Total excluding Dynamic Degree — Mixed 64.95 ± 1.52, Selected 64.04 ± 0.74, Rejected 63.73 ± 1.47. Mixed improves VBench Total over Selected in all 3 matched training seeds, with gains of 0.53–1.15 points.
• Where Wan's gain comes from. Mixed has higher mean scores in Imaging Quality, Aesthetic Quality, Color, Subject Consistency, and Appearance Style. Selected improves Imaging Quality over Rejected but remains behind Mixed.
• Losses disagree with benchmarks for Wan. Rejected ends with the lowest mean training loss, while Selected reaches the lowest final mean loss on all three validation sets, even though Mixed tops VBench. Training loss favors Rejected, validation loss favors Selected, and VBench favors Mixed.
• V-JEPA 2.1-1B probe accuracy. Final SSv2 top-1 accuracy (mean ± std) is 16.26 ± 0.15% for Mixed, 15.71 ± 0.38% for Selected, and 15.10 ± 0.34% for Rejected. Mixed ranks first under all 3 matched training seeds.
• Loss and accuracy diverge again for V-JEPA. Rejected reaches the lowest final mean loss on all three validation sets and in training, while Mixed has the lowest validation-loss variability across seeds. SSv2 accuracy improves despite rising masked-prediction loss.
• Lower loss does not identify the better dataset. The main finding is that pretraining losses favor different recipes than downstream benchmarks do, so lower loss is not a reliable proxy for the better training dataset.
• Selection output scale. From 200k standardized videos and the fixed ingestion/segmentation path, the pipeline produces 716k scene-level clips; applying the filtering and deduplication policy yields 352k Selected and 364k Rejected clips.

Methodology in Plain English

VidaForge treats a data recipe as a chain of five stages applied to a fixed snapshot of raw videos. Ingestion probes media files (including videos stored inside tar shards), applies coarse screen rules on resolution, frame rate, and duration, and transcodes accepted videos into a standard format. Segmentation detects scene boundaries (using PySceneDetect variants, TransNetV2, or uniform and seeded-random baselines) and cuts bounded-duration clips. Selection prepares reusable frames and audio, measures optical quality, motion, aesthetics, and visible-text area, computes two kinds of duplicate relations, and then applies one YAML policy that combines score gates with per-duplicate-group quotas. Annotation adds camera-motion labels, four levels of captions, and structured tags. Packaging converts annotated clips into the input format expected by a specific training codebase — the paper instantiates both a Wan 2.1 path (with cached latents and text embeddings via NeMo-AutoModel) and a V-JEPA 2.1 path (a video manifest read by a TorchCodec-based dataset, with no captions or precomputed representations).

Because every step saves its outputs, recipe parameters, run identifiers, and the input run it consumed, changing one decision creates a variant that starts from the saved output before that step and reruns only the dependent later steps. Expensive work — quality scoring, duplicate matching — is reusable, so threshold changes are cheap. The authors call this property attribution-ready: each training sample can be traced back through the decisions that produced it.

For the study, the same 200k-video pool is used to build three training sets. Clips scoring at least 0.9, 0.1, 0.1, and 0.5 on optical, motion, aesthetic, and visible-text signals pass the filters; PDQ retains one clip per perceptual duplicate group, and Cosmos-Embed retains 20% per semantic group bounded between 1 and 20 clips. Three 10k-clip validation sets (Selected, Rejected, and Mixed, the last drawing 5k from each pool) are built first, and every clip sharing a parent video_id with any validation clip is excluded from training. After validation isolation, Selected and Rejected each use approximately 290k clips for 2 epochs, while Mixed uses approximately 580k clips for 1 epoch, so every run processes approximately 580k clips and Mixed covers twice as many clips within the same budget. Wan 2.1-1.3B is trained from scratch with 3 seeds on 32 H200 GPUs at global batch size 96; V-JEPA 2.1-1B is trained from scratch with 3 seeds at global batch size 256. Within each family, architecture, optimization, training steps, and evaluation protocols are held fixed.

Why This Matters

Impact on research. The paper reframes infrastructure, not model architecture, as the entry cost of video data research. By providing an executable and traceable path from raw videos to controlled training comparisons, it lowers the barrier for academic groups to study data recipes, and it produces an explicit demonstration that lower pretraining loss does not reliably indicate a better training dataset — a result relevant to anyone using loss curves to select data.

Real-world applications.

• Curating large video corpora for training video generation models, where filtering thresholds and duplicate quotas must be tuned against generation quality rather than loss.
• Building self-supervised video representation models for downstream recognition tasks, where the paper shows frozen-probe accuracy on SSv2 responds to coverage in a way masked-prediction loss does not.
• Auditing or reproducing the data side of published video foundation models when technical reports omit pipeline details.
• Constructing reusable, traceable video datasets for regulated or compliance-sensitive settings, where knowing how each training sample was produced matters.

Industry relevance. The paper's comparison table explicitly positions VidaForge against Data-Juicer with Sandbox, NeMo Curator, and Cosmos Curator — production-adjacent systems built by or widely used in industry — and identifies which capabilities (per-stage processing records carried through to training data, one video pool feeding multiple model families, recipe-variant-to-training-to-evaluation workflows spanning both generation and self-supervision) it claims the others do not fully cover. The released 3.14-million-clip dataset with quality scores, duplicate relations, and fine-grained annotations is directly usable as a starting corpus.

Future Directions

1. Extend the study beyond selection. The conclusion states that future work can extend the selection study to segmentation and annotation, tracking their effects throughout training — both stages are logged by VidaForge but were not varied in this study.
2. Move past early pretraining. The paper explicitly restricts its experiments to early from-scratch pretraining and studies initial learning dynamics; how these coverage-versus-quality conclusions behave under longer training budgets is not reported.
3. Broaden the model and objective coverage. Only Wan 2.1-1.3B and V-JEPA 2.1-1B were used, one per learning objective; whether the downstream-versus-loss divergence generalizes to other architectures, sizes, or objectives is left open.
4. Reuse the released artifacts for new recipe questions. The authors frame VidaForge-3M and the runnable recipes as a base on which subsequent recipe comparisons can reuse existing paths with new selection rules and training settings, rather than rebuilding pipelines.

Target Audience

Researchers and engineers who work on video data curation, video foundation model pretraining, or dataset construction — especially academic groups without frontier-lab data infrastructure. It is also relevant to practitioners who need to justify filtration and deduplication policies with measured downstream effects rather than loss curves, and to those evaluating whether an existing curation system (Data-Juicer, NeMo Curator, Cosmos Curator) already covers their tracing and cross-model training needs. The paper assumes some familiarity with video generative modeling, self-supervised representation learning, and benchmark evaluation such as VBench and SSv2.

Authors’ abstract

Video foundation models increasingly rely on large-scale pretraining data, yet the end-to-end data pipelines behind them remain largely closed and difficult to inspect or reuse. Researchers seeking to understand how video data recipes affect model pretraining often need to build substantial infrastructure before testing even a focused hypothesis. We present VIDAFORGE, an open research infrastructure that represents a video data recipe as an executable five-stage workflow from raw videos to training datasets. A decision in this workflow can be varied to construct alternative datasets while preserving how every sample was produced. To demon strate this research workflow, we compare data recipes with different coverage and quality in early from-scratch pretraining of Wan 2.1 and V-JEPA 2.1. Across both learning objectives, the broader-coverage recipe achieves the highest downstream benchmark scores, while loss-based evaluation favors different recipes. This study demonstrates how VidaForge connects data-recipe choices to downstream model performance. We further release VIDAFORGE-3M, containing 3.14 million scene level clips totaling 6,475 hours, with fine-grained annotations and curation signals for video data-recipe research.