Thinking Ahead: Foresight Intelligence in MLLMs and World Model

AI summary

Overview

Research area: Computer Vision / multimodal vision-language models, visual question answering, world models, and autonomous driving.

Technical level: Intermediate. The paper assumes familiarity with vision-language models (VLMs), VQA benchmarks, trajectory forecasting, and world models, though its core argument is stated in accessible terms.

Scope: The paper introduces FSU-QA, a VQA dataset built on nuScenes, and FSU-Bench, its evaluation split, to train and measure "Foresight Intelligence" in VLMs while also using VLMs as proxy judges of whether world-model-generated futures carry semantically useful information.

What This Paper Is About

Most existing benchmarks for VLMs in driving ask about what is happening now, or what the ego vehicle should do next, treating perception and short-horizon planning as the target. This paper argues that a distinct capability is missing: Foresight Intelligence, the ability to anticipate and reason about long-horizon, multi-agent, counterfactual futures from historical observations. To study it, the authors build FSU-QA and FSU-Bench, requiring models to produce language-grounded answers about future outcomes rather than geometric waypoints, and they additionally test whether world-model-predicted future video and trajectories actually help VLMs answer those questions.

Key Contributions

1. FSU-QA dataset. A VQA dataset for training and evaluating Foresight Intelligence in VLMs, built from nuScenes front-view video, with tasks requiring counterfactual simulation and multi-agent causal reasoning that go beyond the reactive decision-making scope of existing VQA benchmarks. The authors report that finetuning on FSU-QA effectively enhances VLMs' foresight reasoning.

2. Proxy world-model evaluation protocol. A method for measuring whether predicted future videos and trajectories provide useful semantic cues for downstream foresight QA, with the explicit acknowledgment that this protocol is probe-model dependent. This replaces standard pixel-level generation metrics such as FVD and FID with downstream QA performance gains.

3. FSU-Bench evaluation across models. A comprehensive evaluation across multiple VLMs and world models showing that current models consistently struggle with high-level counterfactual reasoning, while world-model-generated futures provide measurable but architecture-dependent semantic gains.

4. Validated annotation and evaluation design. Rule-grounded counterfactual annotations validated through human-in-the-loop verification (150 sampled CFP annotations, 89% agreement with automated labels, Cohen's kappa = 78%), plus a shuffled control experiment confirming that gains from world-model inputs come from semantically informative content rather than extra visual tokens.

Main Findings

• High-level counterfactual reasoning is the weakest area across the board. GPT-5, the top-ranked model with an overall score of 48.66, still scores 20.75 on the high-level CFP task. Gemini models show high-level CFP scores of 12–18, and several Qwen models fall between roughly 5 and 11.

• GPT-5 ranks first overall. GPT-5 achieves the highest overall score (48.66) and Rank 1, with particular advantages in mid-level spatio-temporal reasoning and the best scores on Relative Position (32.32) and Ego–VRU Relative Position (46.91).

• Claude-Sonnet-4.5 ranks second. It scores 46.38 overall and 49.33 on Risk Area, the highest among closed-source models, suggesting sensitivity to safety-critical scene cues.

• Qwen2.5-VL-72B is the strongest open-source entry. It reaches 45.86 overall (Rank 3), with strong low-level results on Turn Change (90.17) and Lane Change (97.50), but drops noticeably on mid-level tasks. Medium-sized models Qwen2.5-VL-7B and Qwen3-VL-32B both reach 44.74.

• Finetuning on FSU-QA gives the largest gains. Qwen3-VL-8B-FI, finetuned from Qwen3-VL-8B, reaches 59.59 overall versus 41.48 for the base Qwen3-VL-8B, and 50.20 on the high-level CFP task, surpassing all baselines.

• Scale alone does not explain foresight ability. The authors report that performance patterns suggest foresight reasoning is not merely a function of model scale but is sensitive to the richness of semantic scene understanding in the pretraining corpus.

• Shuffled controls behave as expected. Replacing world-model predictions with predictions from a randomly selected different scene consistently decreases VLM accuracy across all task categories relative to baseline, supporting the claim that gains from genuine predictions reflect informative content.

• DrivingWorld outperforms Epona on relational reasoning. DrivingWorld-generated predictions produce larger and more consistent gains across VLMs than Epona, particularly on tasks requiring temporal consistency and agent–agent relational reasoning such as Relative Position and Pedestrian Intent, with pronounced improvements on Risk Area and Ego–VRU Relative Position. Epona's gains are described as more modest and task-selective.

• Modalities contribute complementarily. Predicted video mainly helps short-term motion anticipation (GPT-5 benefits most on Relative Distance and Ego–VRU Relative Position), while predicted trajectories shift improvements toward relational reasoning (Claude-Sonnet-4.5 gains on Pedestrian Intent and Risk Area). Combining both yields the most comprehensive benefits.

Methodology in Plain English

The researchers start from the nuScenes dataset, which supplies front-facing camera video, 3D object annotations, trajectories, and HD-map lane lines. Each roughly 15-second video is split into a 3-second historical observation window and a 12-second future window. An automatic annotation pipeline applies a question checklist to each scene and generates questions from predefined templates, with answers derived by rule-based criteria over kinematic states, spatio-temporal relations, VRU behaviors, environmental risks, and counterfactual verification.

The tasks are organized into nine levels of cognitive complexity. Low-level spatio-temporal dynamic reasoning covers Speed Change, Turn Change, Lane Change, Relative Distance, and Relative Position. Mid-level VRU-centric risk assessment covers Pedestrian Intent, Ego–VRU Relative Position, and Risk Area. High-level causal reasoning covers Counterfactual Prediction, where the model is asked what would happen under a hypothetical maneuver such as a constant-radius left turn.

Counterfactual ground truth is generated without an external simulator, using closed-world geometry: given accurate 3D trajectories, HD-map lane boundaries, and kinematic states, collision outcomes under a prescribed maneuver can be determined via spatial intersection checks. Human experts adjudicated a random sample of 150 CFP annotations and agreed with the automated labels 89% of the time (Cohen's kappa = 78%). Separately, 45 scenes (approximately 30% of FSU-Bench) were reviewed by three domain experts over three rounds until the human-flagged error rate dropped below 5%, with 654 QA pairs revised or removed.

For evaluation, VLMs receive a task-specific system instruction, encoded historical frames, a trajectory representation, and a multiple-choice question. Frames are sampled at 2 fps from the 3-second history, giving 7 input frames per query, and trajectories are encoded as (x, y, θ) waypoints at 2 Hz. API models use default official settings in a single pass with no retries; locally deployed Qwen models use BF16 inference with temperature 0.1. Answers are scored with Accuracy, requiring exact match, with continuous tasks discretized into labeled bins.

For the world-model side, the baseline is VLM zero-shot performance on historical input only. Future predictions from a world model (predicted video, predicted trajectory, or both) are added to the prompt, and the accuracy gain is treated as a proxy for semantic utility. A shuffled control replaces each scene's predictions with those from a different scene.

Why This Matters

Impact on research. The paper reframes driving VQA away from reactive planning and geometric waypoint generation toward semantic, counterfactual, long-horizon foresight. It also proposes an alternative to pixel-level world-model metrics: rather than asking whether generated video looks realistic, it asks whether generated futures help a downstream model answer questions correctly. The shuffled control adds a validation step that is often missing from such proxy evaluations.

Real-world applications:

• Autonomous driving safety. Anticipating whether a hypothetical maneuver would cause a collision or traffic violation directly targets safety-critical decision support.

• Vulnerable road user protection. Pedestrian intent prediction and Ego–VRU relative position estimation address the most unpredictable agents in urban scenes.

• World-model development for simulation. The proxy protocol gives world-model builders a downstream utility signal, distinguishing architectures (here, DrivingWorld's autoregressive design versus Epona's combination of autoregressive and DiT components) on relational reasoning tasks.

• Training data generation for driving assistants. The finding that finetuning on FSU-QA lifts a smaller model above much larger closed-source systems suggests a route to targeted supervision where pretraining corpora are thin.

Industry relevance. Autonomous driving stacks increasingly incorporate VLMs and world models. The result that foresight capability is not explained by parameter count alone, and that targeted finetuning on 18K QA pairs outperforms larger baselines inside the nuScenes domain, is directly relevant to teams deciding where to invest between scaling and domain-specific data. The dataset and benchmark are released publicly via Hugging Face.

Future Directions

• Cross-domain generalization. The authors state that generalization to other driving datasets such as Waymo or nuPlan remains important future work, and that FSU-QA's finetuning gains are currently demonstrated within the nuScenes domain.

• Domain adaptation across driving datasets. Listed explicitly in the conclusion as an open direction.

• Tighter integration between world model generation and VLM-based semantic reasoning. The paper calls for closer coupling of the two rather than treating generated futures as optional prompt additions.

• Reducing probe-model dependence. The authors acknowledge that the proxy world-model evaluation protocol is probe-model dependent, and the finding that Epona and DrivingWorld gains differ by architecture raises the question of how stable such proxy judgments are across different VLM judges.

Target Audience

Researchers and practitioners working on vision-language models, multimodal reasoning, world models for driving, and autonomous driving perception and planning. It is also relevant to benchmark designers interested in counterfactual or long-horizon evaluation, and to industry teams deciding whether to invest in scaling models or in targeted domain-specific supervision. Readers without background in VLM architectures or driving datasets will need to consult the cited work on models such as DriveLM, OmniDrive, Epona, and DrivingWorld to follow the comparative claims.

Authors’ abstract

In this work, we define Foresight Intelligence as the capability to anticipate and interpret future events-an ability essential for applications such as autonomous driving, yet largely overlooked by existing research. To bridge this gap, we introduce FSU-QA, a new Visual Question-Answering (VQA) dataset specifically designed to elicit and evaluate Foresight Intelligence. Using FSU-QA, we conduct the first comprehensive study of state-of-the-art Vision-Language Models (VLMs) under foresight-oriented tasks, revealing that current models still struggle to reason about future situations. Beyond serving as a benchmark, FSU-QA also enables the assessment of world models by measuring the semantic coherence of their generated predictions, quantified through performance gains when VLMs are augmented with such outputs. Our experiments further demonstrate that FSU-QA can effectively enhance foresight reasoning: even small VLMs fine-tuned on FSU-QA surpass much larger, advanced models by a substantial margin. Together, these findings position FSU-QA as a principled foundation for developing next-generation models capable of truly anticipating and understanding future events. Furthermore, beyond model performance, we examine whether WM-generated predictions remain semantically consistent by using VLM-based proxy judges, and validate this evaluation protocol through shuffled control experiments. Fine-tuning models on FSU-QA leads to substantial improvements in foresight understanding, demonstrating the dataset's effectiveness and offering a principled foundation for future research.