DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving

AI summary

Overview

• Research area: Computer Vision / Autonomous Driving — specifically, evaluation benchmarks for generative video "world models" that predict future driving scenes.
• Technical level: Intermediate. The paper is readable at a conceptual level, but full appreciation of the metric suite (FVD, FTD, ADE, DTW, SLAM-based trajectory recovery) benefits from familiarity with video generation and motion prediction literature.
• Scope: The paper introduces DrivingGen, a benchmark combining a deliberately diverse driving evaluation dataset with a four-dimensional metric suite (distribution, quality, temporal consistency, trajectory alignment) used to evaluate 14 state-of-the-art generative world models.

What This Paper Is About

Generative video models are increasingly being pitched as "world models" for self-driving: given a scene (and optionally a commanded route), they imagine how the future unfolds, enabling simulation, corner-case testing, and synthetic data generation. The problem is that the field has no rigorous way to measure whether these imagined futures are actually good — existing benchmarks borrow generic video metrics that ignore driving-specific safety concerns, rarely check whether the implied vehicle motion is physically plausible, and overlook whether the model actually follows a commanded trajectory. DrivingGen is the authors' answer: a benchmark with a carefully balanced dataset and a suite of metrics designed specifically for the driving domain.

Key Contributions

1. A diverse driving evaluation dataset with two tracks. The benchmark contains 400 samples (200 per track), deliberately keeping the total small because video generation inference is slow. The open-domain track draws on internet-sourced footage spanning rain, snow, fog, floods, sandstorms, dawn/day/night, and seven global regions (North America, East Asia & Pacific, Europe & Central Asia, Middle East & North Africa, Latin America & Caribbean, South Asia, Sub-Saharan Africa), with normal-weather daytime clips capped below 60%. The ego-conditioned track aggregates data from five open-source driving datasets (ZOD, DrivingDojo, COVLA, nuPlan, WOMD) across Europe, China, Japan, and the US to test trajectory controllability. Each sample bundles a front-view image, a language scene description (generated with Qwen), and an optional ego trajectory.

2. A four-dimensional, driving-specific metric suite. Metrics cover (a) distribution — FVD for videos plus a new Fréchet Trajectory Distance (FTD) that encodes trajectories using the Motion Transformer encoder; (b) quality — CLIP-IQA+ for perceptual quality, the IEEE P2020 Modulation Mitigation Probability (MMP) metric for flicker artifacts, and a composite trajectory quality score combining comfort (jerk, lateral acceleration, yaw rate), motion (penalizing near-static output), and curvature (penalizing zig-zags); (c) temporal consistency — motion-aware video consistency using optical flow plus DINOv3 features, agent-level appearance consistency using YOLOv10 detection and SAM2 tracking, abnormal agent disappearance detection via the Cosmos-Reason1 vision-language model, and trajectory consistency over velocity/acceleration; (d) trajectory alignment — Average Displacement Error (ADE) and Dynamic Time Warping (DTW) against the conditioning trajectory.

3. The first large-scale head-to-head evaluation of driving world models. Fourteen models are benchmarked across three categories: closed-source general models (Kling 2.1, Gen-3 Alpha Turbo), open-source general models (CogVideoX, Wan2.2-I2V, HunyuanVideo-I2V, LTX-Video, SkyReels-V2-I2V), physical-world models (Cosmos-Predict1 and Cosmos-Predict2), and driving-specialized models (Vista, DrivingDojo, GEM, VaViM, UniFuture). All are evaluated over a 100-frame prediction horizon.

4. Quantified trade-offs and exposed failure modes. The benchmark demonstrates that strong performance on any single metric does not imply plausible driving, and that no current model achieves both photorealism and physically faithful motion.

Main Findings

• Closed-source models lead overall. Kling 2.1 ranks first on both tracks, with Gen-3 Alpha Turbo close behind. Both maintain stable agent behavior, rarely show abnormal object disappearance, and preserve scene coherence over time.
• Open-source general models are competitive on narrow dimensions. CogVideoX and Wan2.2-I2V achieve low FVD in both tracks, showing open-source models can match leaders on distributional realism even without topping the overall ranking.
• No model combines visual realism with trajectory fidelity. The authors identify distinct "personas": general models produce visually appealing traffic scenes but break physical consistency in vehicle motion, while driving-specific models follow commanded paths with plausible kinematics (low ADE/DTW) but show noticeable visual artifacts. Bridging this gap is the field's key open problem.
• Trajectory alignment remains poor across the board. Under ego conditioning, ADE and DTW errors are large. The authors attribute this to two separate causes: video artifacts (texture repetition, blur, unstable geometry) that corrupt SLAM-based trajectory recovery, and the model's genuine failure to follow the intended path.
• Single-metric evaluation hides failure modes. Videos can score well on FVD while exhibiting stop-go jitter, identity drift, or non-physical agent disappearances; low flicker (good MMP) can coexist with poor subjective quality or unstable agents. Joint reporting across all dimensions is what exposes these problems.
• The benchmark correlates with human judgment. The authors compute a Spearman's correlation coefficient and confirm their rankings align reasonably well with human evaluation.

Methodology in Plain English

The authors start from the observation that the datasets everyone uses (nuScenes, OpenDV) are over 80-90% sunny daytime driving, which makes them poor proxies for real-world deployment. They build a replacement dataset by pulling internet video from many countries and weather conditions for one track, and aggregating five existing labeled driving datasets for a second track where each clip comes with a commanded ego trajectory.

To evaluate generated videos, they need to recover the vehicle motion the model implicitly produced. They run a standard computer-vision pipeline — SIFT features with RANSAC-based perspective-n-point solving, plus UniDepthV2 for depth — to reconstruct a trajectory from each generated video. They verify that every video in the benchmark successfully reconstructs a trajectory.

The metric design philosophy is that driving has requirements generic video benchmarks miss. So they keep FVD for visual distribution, add FTD for motion distribution (borrowing an encoder from the motion-prediction literature rather than inventing one), import automotive imaging standards for flicker, use a vision-language model to judge whether surrounding agents vanish normally (occlusion, leaving frame) or unnaturally, and even the temporal consistency metric is motion-aware — videos are adaptively downsampled based on their optical flow so that a near-static video cannot game a high consistency score.

Finally they run all 14 models through the pipeline and report a transparent full table of every metric, using average rank only as a quick summary rather than a definitive score.

Why This Matters

Impact on research. The paper gives the driving world model community its first shared scoreboard with metrics that actually reflect what matters for self-driving. It reframes progress away from "does the video look good?" toward "does the imagined future obey physics, preserve agents, and follow commands?" The finding that visual fidelity and motion fidelity are currently in tension is a concrete research agenda, not just a leaderboard result.

Real-world applications:

• Simulation-based testing: Driving world models can generate safety-critical scenarios (night snowstorms, pedestrian cut-ins, floods) on demand rather than waiting for them to occur on real roads.
• Synthetic training data: Generated driving footage can augment datasets for perception and planning models, particularly for underrepresented conditions.
• Model selection for deployment teams: Engineering groups can use the benchmark to pick a world model matching their priorities (visual realism for demonstration video, trajectory accuracy for closed-loop planning).
• Safety validation: Quantifying abnormal agent disappearance and trajectory plausibility helps flag models whose failures could propagate into unsafe driving decisions.

Industry relevance. Autonomous driving companies invest heavily in simulation and synthetic data pipelines. A standardized, reproducible benchmark with public dataset and evaluation code lets them compare vendor models, closed- and open-source, on equal footing, and it sets expectations that trajectory controllability — not just visual appeal — is a first-class product requirement.

Future Directions

1. Scaling the dataset. The 400-sample size trades coverage for evaluation speed. Expanding to thousands of clips as generation and evaluation get cheaper would improve long-tail scenario coverage.
2. Closed-loop and interactive evaluation. All models tested here are open-loop. A unified closed-loop benchmark — inserting world models into interactive simulators like CARLA or pairing them with closed-loop dataset simulation like Navsim — is the natural next step and a prerequisite for judging safety and planning utility.
3. Multi-modal and downstream-task metrics. The current dataset is single front-view camera only. Extending to multi-view video and additional sensors (LiDAR, HD maps) would enable structural consistency metrics and let researchers measure how well downstream autonomous driving stacks perform on synthetic footage.
4. Scene controllability and counterfactual reasoning. The benchmark covers ego-trajectory controllability but not control over scene content (placing a pedestrian, rearranging lanes) or counterfactual generation (physically impossible or hypothetical events). Both are flagged as high-value but currently unsolved evaluation problems, alongside the challenge of defining a single composite score from heterogeneous normalized metrics.

Target Audience

Researchers and graduate students working on video generation, world models, or end-to-end autonomous driving who need a rigorous way to evaluate generative models; autonomous driving engineers responsible for simulation and synthetic data pipelines; benchmark and evaluation researchers interested in how domain-specific metrics should be designed; and technically literate readers tracking the trajectory of generative AI applied to safety-critical robotics.

Authors’ abstract

Video generation models, as one form of world models, have emerged as one of the most exciting frontiers in AI, promising agents the ability to imagine the future by modeling the temporal evolution of complex scenes. In autonomous driving, this vision gives rise to driving world models: generative simulators that imagine ego and agent futures, enabling scalable simulation, safe testing of corner cases, and rich synthetic data generation. Yet, despite fast-growing research activity, the field lacks a rigorous benchmark to measure progress and guide priorities. Existing evaluations remain limited: generic video metrics overlook safety-critical imaging factors; trajectory plausibility is rarely quantified; temporal and agent-level consistency is neglected; and controllability with respect to ego conditioning is ignored. Moreover, current datasets fail to cover the diversity of conditions required for real-world deployment. To address these gaps, we present DrivingGen, the first comprehensive benchmark for generative driving world models. DrivingGen combines a diverse evaluation dataset curated from both driving datasets and internet-scale video sources, spanning varied weather, time of day, geographic regions, and complex maneuvers, with a suite of new metrics that jointly assess visual realism, trajectory plausibility, temporal coherence, and controllability. Benchmarking 14 state-of-the-art models reveals clear trade-offs: general models look better but break physics, while driving-specific ones capture motion realistically but lag in visual quality. DrivingGen offers a unified evaluation framework to foster reliable, controllable, and deployable driving world models, enabling scalable simulation, planning, and data-driven decision-making.