How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Models

AI summary

Overview

Research area: Computer Vision — generative video modeling, diffusion/flow-matching models, and empirical scaling laws, applied to autonomous driving data.

Technical level: Intermediate. The core conclusions are stated in plain terms, but the methodology assumes familiarity with diffusion or flow-matching training, transformer architectures, and the standard power-law formulation used in language-model scaling research.

Scope: A controlled empirical study deriving validation-loss scaling laws for driving-video diffusion models trained from scratch on a fixed 5,500-hour corpus, then using those laws to justify and validate training a 9B-parameter flagship model.

What This Paper Is About

Unlike web-scale image or language models, driving video cannot be scraped freely: collection is expensive, footage is privacy-sensitive, and most labs work with a fixed dataset and a limited GPU budget. The authors ask a concrete budget-allocation question — given 5,500 hours of driving video, is compute better spent on a larger model, on longer training over the same data, or on acquiring more data? They answer it by fitting scaling laws across more than 200 training runs and then using those laws to predict, before training, what a 9B-parameter model should achieve.

Key Contributions

1. A systematic scaling-law study for video diffusion models in a data-constrained regime. The authors fit asymptotic power laws of the form L(x) = L₀ + A·x^(−α) along three axes — model size (N), training exposure (D), and total compute (C) — across roughly 200 runs spanning 1.6M to 1.1B parameters, using training exposure (samples seen, including repeats) rather than unique dataset size as the scalable data resource.

2. A controlled data-restriction ablation. Holding training exposure fixed while shrinking the pool of unique footage by up to 1,000×, they show that heavy repetition of the same clips is largely harmless up to roughly 200 epochs, isolating the point at which a fixed corpus actually becomes a bottleneck.

3. A flagship 9B model and confirmation of extrapolation. They train what they describe as the largest open-source video diffusion model trained from scratch on driving data, reaching a validation loss of 0.0781 against a pre-training prediction of 0.0753 — a 3.6% relative error, despite extrapolating 8× beyond the largest fitted model with no intermediate scale points.

4. A new open-source state of the art on nuScenes driving video generation, achieved by full fine-tuning with pseudo-trajectory conditioning rather than the parameter-efficient adaptation (e.g., LoRA) used by most prior driving world models.

Main Findings

• Training exposure is the steepest lever. The fitted exposure exponent is α_D ≈ 0.74 (spread 0.66–0.84 across model sizes), while the model-size exponent is only α_N ≈ 0.2125. For a fixed model and a fixed compute budget, longer training improves loss much faster than enlarging the model does.

• But capacity still lowers the floor. No saturation appears up to 1.1B parameters, and the asymptotic loss L₀ falls monotonically with model size. Compute-optimal allocation therefore still shifts toward larger models as the budget grows — the iso-FLOP analysis shows the optimal (N, D) pair moving rightward with more compute.

• The scaling exponent for training is scale-invariant. Across 11 model sizes from 1.6M to 1.1B parameters, convergence dynamics follow nearly identical curves with a stable exponent, meaning training outcomes can be forecast before full convergence.

• Data repetition behaves like fresh data — up to a point. A 100× reduction in unique footage (5,500h → 55h) at fixed exposure changed validation loss only from 0.0939 to 0.0980. Only the extreme 5.5-hour restriction, corresponding to roughly 2,000 epochs of repetition, caused sharp degradation. The authors hypothesize this is because flow matching resamples noise and timestep on every pass, unlike autoregressive training with fixed targets.

• The laws extrapolate accurately. A compute-scaling fit predicted 0.0753 for the 9B configuration; the trained model reached 0.0781 after ~1.2×10⁷ samples (about 2 epochs). A post-hoc fit gives an asymptotic loss of 0.0748, within 1% of the a-priori compute-law prediction.

• Scaling produces the largest gains on video-level metrics. Going from 1B to 9B parameters improved FVD_I3D from 33.94 to 37.16 on NATIX in the unconditioned setting but dropped FVD_VideoMAE from 89.49 to 75.86, and trajectory-conditioned models showed the strongest gains on temporal-consistency metrics — indicating better long-range motion coherence rather than just per-frame fidelity.

• Prior work is outperformed on nuScenes. The 9B model reaches FID_Inception 2.72 and FVD_I3D 25.50 on the Vista split, against 6.9/89.4 for Vista, 10.5/158.5 for GEM, 7.4/90.9 for Driving World, and 7.5/82.8 for Epona.

Methodology in Plain English

The authors build a family of 12 video diffusion models ranging from 1.6M to 9.1B parameters, all with the same architecture: a spatio-temporal diffusion transformer (DiT-style) with alternating spatial and temporal attention blocks, operating on latents produced by a frozen, off-the-shelf Wan 2.1 video autoencoder. Training uses conditional flow matching, which learns to transport noise into video frames along a straight interpolation path, with trajectory conditioning injected through adaptive LayerNorm.

All models train on the same corpus: 5,500 hours of multi-camera driving footage from NATIX, chopped into 2.5-second, 25-frame front-camera clips at 320×416 resolution. With 6.3M distinct clips available, the exposures tested (10M–28M samples) correspond to roughly 1.6–4.5 epochs, so the data is repeated but never excessively.

The experimental design is a three-way grid sweep. For model scaling, they train 72 models at fixed exposure and sweep learning rate, batch size, and seed. For training scaling, they record validation loss throughout training for every model and fit a per-model power law. For compute scaling, they identify the Pareto-optimal (N, D) pairs at each compute budget and fit the lower envelope. Each of the three fitted laws independently predicts what a 9B model should reach, and the authors train that model to check whether the predictions hold.

For evaluation, they generate pseudo-ego-trajectories with an off-the-shelf model (OccAny), filter and rebalance them across 11 driving-maneuver classes, and fine-tune the backbone with trajectory conditioning. Quality is measured with FID on two image encoders, FVD on two video encoders, and Average Displacement Error between real and re-extracted trajectories.

Why This Matters

Impact on research. This is one of the first scaling-law studies for video diffusion models trained from scratch in a genuinely data-limited domain, rather than on web-scale data or via adaptation of pre-trained backbones. It shows that Chinchilla-style reasoning transfers to flow matching, that the steepest axis is training exposure rather than capacity, and that the classic assumption "more unique data is always better" does not hold while repetition stays below a few hundred epochs. It also provides a template for pre-registering expected outcomes before expensive runs.

Real-world applications:

• Corner-case synthesis for autonomous driving. Generating rare, dangerous scenarios — hard braking, unusual turns, near-misses — that a real test fleet could never safely or repeatedly capture.
• Learned simulators and world models. When trained to predict how a scene responds to ego-vehicle actions, these models provide a simulation space for motion planning and policy evaluation.
• Synthetic training data for perception systems. The generated footage can supplement real data for downstream detectors, segmenters, and planners, particularly for underrepresented maneuvers and environments.
• Budget planning for industrial ML teams. The fitted laws tell practitioners whether their next thousand GPU-hours should go toward a bigger model or a longer schedule, and how much new data collection is actually worth.

Industry relevance. For any organization that owns a proprietary driving dataset but not hyperscaler-scale infrastructure, the practical takeaway is direct: train longer before training bigger, expect repetition to be cheap, and use scaling laws to decide when a flagship run is worth the cost. The 9B model is trainable on a single node of four H100 80GB GPUs with FSDP2, which places it within reach of a well-resourced academic or mid-size industrial lab rather than only a hyperscaler.

Future Directions

• Filling the intermediate-scale gap. The largest fitted model was 1.1B, and the flagship was 9B. The authors identify 3B–4B checkpoints as the most obvious way to tighten extrapolated estimates and validate whether the power law holds smoothly across the gap.

• Extending the analysis to the frozen VAE. The scaling study covers only the diffusion transformer; whether the autoencoder's compression also imposes a scaling bottleneck, and how jointly scaling encoder and generator changes the laws, remains open.

• Generalizing beyond a uniform learning rate. The 9B model required a learning-rate drop mid-training for stability, which the fits did not model. A full hyperparameter grid at scale could reduce the residual prediction error and clarify whether instability is intrinsic to large video diffusion transformers.

• Testing the repetition limit at scale. The data-restriction ablation was run only on the base model at exposures under five epochs. Larger models trained for longer will accumulate more epochs over the same 5,500 hours, and the paper predicts genuinely new data will start to matter again as that limit approaches — an empirical question left for future work.

Target Audience

This paper is most useful to machine learning researchers and engineers working on generative video models, diffusion or flow-matching training at scale, and autonomous-driving world models. It is also valuable to industrial ML practitioners making compute-allocation decisions under fixed data and hardware budgets, and to dataset owners deciding whether to invest in collecting more driving footage or in longer training on what they already have. Readers without a background in diffusion training will still follow the conclusions but may find the methodological sections dense.

Authors’ abstract

Video generation for autonomous driving cannot follow the web-scale route: driving data is expensive to collect, bound by privacy requirements, and cannot be scraped at will, so models must make the most of a fixed corpus. We present a systematic scaling-law study of video diffusion models trained from scratch on driving data: a family of models from 1M to 9B parameters, trained at different exposures on up to 5,500 hours of driving. Validation loss follows consistent power laws in both model size and training exposure, answering the questions that shape a training budget: whether compute is better spent on longer training or on a larger model, and whether more data is needed. Loss improves much faster with training exposure than with model size, making longer training the most effective way to improve a fixed model under limited compute. However, larger models continue to achieve lower asymptotic loss, so compute-optimal scaling still favors increasing model size when sufficient compute and data are available. Guided by these laws, we train a 9B-parameter model, to our knowledge the largest video diffusion model trained from scratch on driving data: it sets a new open-source state of the art for driving video generation, as measured on nuScenes. Our code and pretrained models are available at https://github.com/valeoai/VATIX. NATIX is separately releasing the underlying driving data in stages.