FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

AI summary

Overview

Research area: Multimodal large language model (MLLM) evaluation — specifically benchmarking future event forecasting from joint audio–visual ("omni-modal") context.

Technical level: Intermediate. The paper is a benchmark-and-training-strategy paper; readers need familiarity with MLLM architectures (video encoders, audio encoders, LoRA fine-tuning) but the core ideas are explained in accessible terms.

Scope: The paper introduces FutureOmni, a 919-video / 1,034-question multiple-choice benchmark for audio-visual future forecasting, evaluates 13 omni-modal and 7 video-only models on it, and proposes a 7K-sample instruction-tuning dataset plus an Omni-Modal Future Forecasting (OFF) training strategy that improves both forecasting and out-of-domain generalization.

What This Paper Is About

Existing multimodal benchmarks (WorldSense, DailyOmni, AVQA, MUSIC-AVQA) mostly test retrospective understanding — describing and reasoning about events that have already occurred in a video. Prior future-prediction benchmarks (VLEP, IntentQA, MM-Forecast, FutureBench, ForecastBench, FutureX, MIRAI) are either text-only or vision-centric, effectively discarding the audio track. The goal of this work is to close that gap by building the first benchmark that forces models to predict what happens next using both audio and visual cues jointly, and then to show that targeted instruction tuning can measurably improve this ability.

Key Contributions

1. FutureOmni benchmark. The first benchmark for evaluating future forecasting ability of MLLMs under omni-modal context, containing 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains (21 fine-grained subcategories). All 1,034 QAs are future-forecasting questions, and average video duration is 163.5 seconds.
2. Extensive evaluation of 20 MLLMs. 13 omni-modal and 7 video-only models, spanning open-source and proprietary systems, are evaluated. The best result is 64.8% (Gemini 3 Flash), showing that both omni-modal and video-only models have limited future-forecasting ability.
3. FutureOmni-7K instruction-tuning dataset and the OFF training strategy. A 7K-sample dataset that integrates the rationales from the data-construction pipeline into each training instance, paired with an Omni-Modal Future Forecasting training method that improves forecasting and out-of-domain generalization.
4. Attention-level analysis. Keyframe attention-score visualizations indicate that OFF improves the model's ability to identify critical video and audio keyframes, which the authors link to better generalization and reasoning.

Main Findings

• Current models struggle with audio-visual future prediction. Across the 20 evaluated models, the best accuracy on FutureOmni is 64.8%, achieved by Gemini 3 Flash. The lowest reported average in the main table is 30.37% (AVicuna, 7B).

• Proprietary models lead open-source Omni-LLMs. Gemini 2.5 Pro and Gemini 3 Flash reach an average of roughly 61%, while the strongest open-source omni-modal model (Qwen3-Omni, 30B) attains only 53.05%.

• Video-only LLMs consistently underperform omni-modal LLMs because they cannot use audio cues. Even the competitive proprietary video-only model GPT-4o reaches a maximum of 49.70%, lower than open-source omni-modal models such as Qwen3-Omni.

• Speech is the most challenging audio type. Qwen3-Omni shows roughly a 10% gap between Music (57.54%) and Speech (47.99%). Gemini 3 Flash scores 60.52% on Speech versus 68.31% on Music. The authors attribute this to the high-level linguistic decoding and visual–semantic alignment that speech requires.

• A "contextual cold start" phenomenon appears across all Omni-LLMs. All models score lowest on the shortest duration interval: Qwen3-Omni at 34.90% and Gemini 3 Flash at 40.78% for [0,2) min. Performance peaks in the medium duration range ([2,4) min) before dipping slightly for long videos.

• Domain difficulty varies widely. Models do better on Game and Dailylife (e.g., Qwen3-Omni at 62.65% on Game) and worse on Documentary and Emergency, with average scores dropping to the 20–40% range (e.g., AVicuna at 20.83% on Doc).

• Both modalities matter. In modality ablation, the full audio+video (A+V) setting is consistently best. Qwen2.5-Omni achieves 47.48% with both modalities but drops to 42.50% with video only or audio only — a gap of roughly 5%.

• Raw audio beats text substitutes for audio. Ola scores 48.54% with raw audio but only 46.95% with subtitles, suggesting non-verbal latent information (emotional tone, atmosphere, urgency) is not fully captured by transcription.

• The dataset resists modality shortcuts. Omni-modal models perform strikingly similarly on audio-only and video-only settings, indicating they cannot solve the task by visual pattern matching or audio classification alone.

• Visual perception is the primary error bottleneck. Of 318 failure cases analyzed from Gemini 3 Flash, 51.6% are Video Perception Errors, 30.8% are Audio-Video Joint Reasoning Failures, and only 2.5% are Lack of Knowledge errors — meaning the gap is driven by perception and causal reasoning, not missing world knowledge.

• OFF improves forecasting, especially on speech. video-SALMONN 2 gains the largest overall increase at +3.87% (46.03% → 49.90%). Qwen2.5-Omni gains +1.03% (47.48% → 48.51%) and Ola gains +1.65% (48.54% → 50.19%). Qwen2.5-Omni's speech performance rises from 37.83% to 47.75%, a leap of nearly 10%.

• OFF generalizes beyond the forecasting task. For Qwen2.5-Omni: WorldSense 37.67 → 40.22 (+2.55), DailyOmni 45.69 → 49.03 (+3.34), JointAVBench 59.30 → 60.88 (+1.58), OmniVideoBench 30.70 → 31.70 (+1.00), Video-MME 53.77 → 55.51 (+1.74), MLVU 54.00 → 54.37 (+0.37). video-SALMONN 2 shows small gains on audio-visual benchmarks (+0.48, +0.67, +0.95, +0.50) but slight decreases on video-only benchmarks (-0.15 on Video-MME, -0.14 on MLVU).

• OFF beats generic caption-based SFT on temporal reasoning. OFF versus SFT: WorldSense Temporal Localization 1.78% vs -0.59%, Temporal Prediction 5.45% vs 1.82%, JointAVBench PTG 6.62% vs 2.21%, OmniVideoBench Temporal Understanding 2.92% vs 1.46%. On FutureOmni itself, OFF improves video-SALMONN 2 from 46.89 to 49.90 and Ola from 48.02 to 50.19 over the caption-supervised baseline.

• Attention shifts toward keyframes after training. Using the Keyframe Attention Score Difference metric on Qwen2.5-Omni with LongVALE annotations (1,000 samples, top-25% scores per layer), the trained model attends more to both video and audio keyframes at critical layers — visual focus intensifies dramatically at Layer 9 and Layer 20, while audio attention is consistently elevated across middle layers (L8–L17).

Methodology in Plain English

The authors built the benchmark through a multi-stage, LLM-assisted, human-verified pipeline:

1. Video collection and filtering. Approximately 18,000 YouTube videos ranging from 30 seconds to 20 minutes were collected. Videos with limited scene change were removed by computing frame-level visual similarity between adjacent frames and discarding those whose average inter-frame similarity exceeded 70%. An audio-intervention strategy (inspired by AVoCaDO) was then applied: two captions were generated per video using UGCVideoCaptioner — one conditioned on audio plus video, one on video only — and their semantic similarity was computed with Sentence-BERT (all-MiniLM-L6-v2). A larger similarity drop indicates stronger audio dependency. Only the top 50% of videos (the bottom 50% by similarity) were kept, yielding a subset of roughly 9,000 videos.

2. Audio-visual temporal localization and calibration. Gemini 2.5 Flash was instructed to scan each video for plot-relevant events and produce precise MM:SS timestamps, ignoring trivial or static background occurrences. Boundary precision was checked by computing Mel-frequency cepstral coefficients (MFCCs) at each event's start and end points and verifying that the difference exceeded a pre-defined threshold of 2.0. Gemini 2.5 Flash was then prompted to annotate acoustic cues (dialogue, sound effects, background music) occurring synchronously with the visual content.

3. Causal pair discovery. DeepSeek-V3.2 analyzed adjacent event segments to determine whether a later event is a direct logical consequence of an earlier one, with the temporal gap between premise and future event capped at a strict maximum of 30 seconds. Each accepted pair required a Premise Event, a Target Event, and a Rationale. The model also scored the audio causal factor for each pair on a 0–2 scale (0 = no influence, 1 = decoration, 2 = causality) and classified the audio factor as Speech, Sound, or Music.

4. Adversarial distractor design. Four distractor types were created to force genuine cross-modal reasoning: (i) Visual-only Perception — visually plausible but contradicted by audio; (ii) Audio-only Perception — consistent with speech or sound but describing visual actions that do not occur; (iii) Delayed — valid past events occurring before the premise, testing temporal precision; (iv) Reverse-Causal — the antecedent or cause rather than the effect, testing the direction of time.

5. Dual-stage verification. Candidate QAs went first to GPT-4o for automated logical validation, then to human verification for quality.

6. Training. For the OFF strategy, a 7K-sample instruction-tuning dataset (FutureOmni-7K) was curated with rationales embedded in each instance. Three open-source models were fine-tuned: Qwen2.5-Omni-7B, Ola-7B, and video-SALMONN 2-7B. LoRA was used for efficiency; visual and audio encoders were frozen and only text backbones were updated. Learning rate was 1e-5, training ran for 1 epoch, and other hyperparameters followed the official training scripts of each model. LlamaFactory was used with LoRA rank 64 for Qwen2.5-Omni-7B, and vLLM was used to speed up evaluation on general benchmarks.

Why This Matters

Impact on research. The paper argues that existing omni-modal benchmarks reward retrospective understanding and that future forecasting — requiring cross-modal causal reasoning, temporal reasoning, and internal knowledge — is a distinct and underexplored capability. By releasing a benchmark with adversarial distractors designed to defeat single-modality shortcuts, the work provides a harder target for measuring genuine multimodal reasoning. The finding that OFF transfers to unrelated benchmarks (WorldSense, DailyOmni, JointAVBench, OmniVideoBench, and even video-only Video-MME and MLVU) suggests that training on causal/temporal prediction objectives may be a broadly useful recipe, not just a task-specific patch. The error analysis is also useful for the field: the fact that only 2.5% of failures are knowledge deficits while 51.6% are visual perception errors redirects attention toward perception and joint reasoning rather than knowledge injection.

Real-world applications (grounded in the paper's stated motivation and the covered domains):

• Autonomous driving. The paper explicitly uses this example: a system must integrate auditory cues such as honking from nearby vehicles with visual information such as pedestrian positions to anticipate future world states and make timely safety decisions.
• Emergency and disaster response. These are among the 8 benchmark domains (Rescue, Disaster subcategories) and are reported as among the hardest, requiring rapid processing of chaotic audio-visual cues such as sirens and screams.
• Surveillance and safety monitoring. Police footage is a dedicated subcategory, and future forecasting from continuous monitoring streams is directly relevant to proactive alerting.
• Video platforms and content understanding. The benchmark spans movies, TV clips, trailers, comedy skits, vlogs, commercials, education, and documentaries — a broad slice of the content that platforms must index and reason about.

Industry relevance. The persistent gap between proprietary (approximately 61% average) and open-source (53.05% best omni-modal) systems is directly actionable for teams building on open-weight models. The OFF recipe is cheap to apply — LoRA, one epoch, frozen encoders, 7K samples — and the reported gains are largest precisely where models are weakest (speech-heavy scenarios). The finding that raw audio outperforms subtitles for prediction also has implications for how multimodal pipelines should be designed.

Future Directions

1. Improving visual perception, not just knowledge. Since 51.6% of the best model's errors come from failing to recognize fine-grained visual details, methods targeting perception of dynamic visual content appear to be the highest-leverage next step.
2. Closing the audio-video joint reasoning gap. 30.8% of errors occur when the model perceives both modalities correctly but fails to synthesize them logically. New architectures or training objectives for cross-modal causal fusion are needed.
3. Addressing speech-heavy scenarios. Speech is the most challenging audio type across models and durations, and OFF's largest single gain came in speech for Qwen2.5-Omni (37.83% to 47.75%). Scaling or specializing instruction data for dialogue-heavy video is an open avenue.
4. Understanding and stabilizing transfer. video-SALMONN 2 showed slight regressions on Video-MME and MLVU after OFF, while Qwen2.5-Omni improved everywhere. Why transfer helps some architectures and mildly hurts others on video-only tasks is an unresolved question.
5. Scaling the rationale-enhanced data. FutureOmni-7K is relatively small; whether the OFF improvements continue to scale with more instruction-tuning samples and more reasoning-chain supervision is not reported.

Target Audience

This paper is most useful for multimodal LLM researchers and engineers building or evaluating audio-visual systems — particularly those working on video understanding, omni-modal architectures, and instruction tuning. It is also valuable for benchmark designers interested in adversarial distractor construction and LLM-assisted, human-verified data pipelines, and for applied teams in autonomous driving, safety monitoring, and video platform infrastructure who need to know where current omni-modal models actually fail. Readers with a general machine learning background but no multimodal specialization will find the problem framing and error analysis accessible, while the pipeline and training details assume familiarity with MLLM fine-tuning.

Authors’ abstract

Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).