XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models

AI summary

Overview

Research area: Multimodal AI evaluation — specifically benchmarks for omni-modal large language models (OLLMs) that process audio, vision, and text.

Technical level: Intermediate. Readers should understand multimodal LLMs, multiple-choice QA evaluation, and basic concepts of cross-modal alignment.

Scope: This paper introduces XModBench, a 61,320-question tri-modal benchmark that tests whether omni-modal models produce consistent answers when the same semantic content is presented through different modality combinations (audio, vision, text).

What This Paper Is About

Existing benchmarks for omni-modal models mostly measure whether a model can answer a question correctly when given audio, image, or video inputs. They do not test whether the model reaches the same conclusion when information arrives through a different modality — a property the authors call "cross-modal consistency." XModBench fills this gap by generating six modality variants of every question (all pairwise combinations of audio, vision, and text as either context or answer candidates) and diagnosing where models diverge.

Key Contributions

1. First tri-modal cross-modal consistency benchmark. XModBench systematically covers all six modality mappings among audio, vision, and text (A→T, A→V, T→A, T→V, V→A, V→T) with semantically identical questions, ensuring no single modality is privileged.

2. Large-scale, balanced coverage. The benchmark contains 61,320 multiple-choice question–answer pairs derived from 10,220 unique instances, spanning five task families (perception, spatial reasoning, temporal reasoning, linguistic understanding, external knowledge) and 17 subtasks.

3. Two new diagnostic metrics. The authors introduce modality disparity (accuracy gap when one modality is substituted for another) and directional imbalance (accuracy gap when context and candidate modalities are swapped), enabling fine-grained diagnosis beyond aggregate accuracy.

4. Diagnostic analysis linking benchmark behavior to training strategies. The paper correlates observed disparities with publicly reported training choices — showing that interleaved multimodal pre-training reduces directional imbalance, while post-training on non-interleaved speech-focused data (as in EchoInk-R1) can erode cross-modal alignment.

Main Findings

• Even Gemini 2.5 Pro falls short. The strongest evaluated model achieves 70.6 average accuracy (humans score 91.5), with spatial reasoning at 50.1% and temporal reasoning at 60.8%, both below 60% or barely above.

• Audio is the weakest modality. Modality disparity between text and audio reaches −49 points for Gemini 2.5 Pro; vision-vs-audio disparity is −33, and text-vs-vision is only −15. Auditory representations remain the weakest link across all models.

• Directional imbalance is substantial. In vision–text pairs, Gemini 2.5 Pro drops 8.8 points when swapping T→V to V→T; Qwen2.5-Omni drops 16.6 points. Audio–text gaps are 6–8 points. Audio–vision pairs are nearly symmetric but perform much lower overall. This indicates training data bias toward text as the dominant output modality.

• Open-source models lag significantly. Qwen2.5-Omni (58.6) and EchoInk-R1 (59.2) are the strongest open models, still 11–12 points behind Gemini 2.5 Pro and notably weaker on external knowledge (50.5 vs. 89.3).

• Combining audio and vision yields only modest gains. Adding both modalities as context improves Gemini 2.5 Pro by just 1.16 points over the best single-modality result, and Gemini 2.0 Flash actually degrades by 5.36 points — suggesting current models do not fully exploit complementary signals.

• Spoken-language bias limits non-speech audio understanding. Models with pre-trained speech encoders (e.g., Baichuan Omni 1.5 with Whisper) show sharp declines on environmental sound and musical tasks, revealing a semantic gap between linguistic speech and general acoustic signals.

• Post-training can erode alignment. EchoInk-R1, despite being based on the well-aligned Qwen2.5-Omni, exhibits larger modality disparities and directional imbalances after fine-tuning on spoken-instruction data, demonstrating that cross-modal alignment from pre-training can be lost.

Methodology in Plain English

The researchers built aligned text–image–audio triplets from three sources: re-annotated existing multimodal datasets (e.g., VGG-Sound, STARSS23), synthetic generation (e.g., text-to-speech for translation tasks, rendered text images), and web collection (movie trailers, singer portraits). Each item is a four-choice question with a <context> and four <candidates>. By permuting which modality serves as context and which supplies the candidates, they generate six semantically identical variants per question. Distractors are designed to be challenging but unambiguous. GPT-5 refines question phrasing for fluency without changing semantics, and human annotators verify correctness and resolve ambiguities. Evaluation reports accuracy per modality configuration, per task family, and computes two diagnostic gaps (modality disparity and directional imbalance). A 6k-item "XModBench-Lite" version is also released for faster evaluation.

Why This Matters

Impact on research: XModBench shifts evaluation from "can the model answer?" to "does the model answer consistently across modality substitutions?" This reframes cross-modal consistency as a first-class benchmark property and provides actionable diagnostics for architecture and training-data design.

Real-world applications:

• Accessibility tools that must relay identical content through speech, text, or images for users with different needs.
• Autonomous driving and robotics, where spatial and temporal reasoning across cameras and microphones must be reliable regardless of which sensor supplies information.
• Medical triage and telemedicine, where the same patient information may arrive as audio notes, images, or text records and must be interpreted consistently.
• Content moderation and media analysis, where models must recognize the same event through audio, video, or transcripts.

Industry relevance: The findings give concrete signals about training-data composition — interleaved multimodal corpora reduce directional imbalance, and post-training on narrow, non-interleaved data can degrade alignment. These are directly actionable for teams building omni-modal assistants, and the benchmark exposes blind spots (music, spatial audio, non-speech audio) that current pipelines systematically neglect.

Future Directions

1. Mitigating audio weakness. Why do current audio encoders and training pipelines lag so far behind vision and text, and what architectural or data changes could close the −49 disparity?

2. Reducing directional imbalance. Can training objectives be designed to enforce symmetry between inverse modality configurations (e.g., T→V vs. V→T) rather than relying on incidental interleaved data exposure?

3. Better exploitation of complementary signals. The A+V context yields only marginal gains — what prevents models from integrating heterogeneous inputs, and can cross-modal fusion be improved?

4. Extending to new domains and modalities. The open-source generation tools allow expansion to additional tasks and modalities beyond audio–vision–text; evaluating consistency in video, 3D, or sensor-fusion settings is a natural next step.

Target Audience

Researchers and engineers working on multimodal or omni-modal LLMs, benchmark designers, and practitioners evaluating model deployment in settings where input modality varies (accessibility, robotics, media analysis). Also valuable for training-data and post-training specialists seeking evidence on how data composition affects cross-modal alignment. Readers without a multimodal background can still follow the high-level findings, but understanding the diagnostic metrics benefits from some familiarity with model evaluation methodology.

Authors’ abstract

Omni-modal large language models (OLLMs) aim to unify audio, vision, and text understanding within a single framework. While existing benchmarks primarily evaluate general cross-modal question-answering ability, it remains unclear whether OLLMs achieve modality-invariant reasoning or exhibit modality-specific biases. We introduce XModBench, a large-scale tri-modal benchmark explicitly designed to measure cross-modal consistency. XModBench comprises 60,828 multiple-choice questions spanning five task families and systematically covers all six modality compositions in question-answer pairs, enabling fine-grained diagnosis of an OLLM's modality-invariant reasoning, modality disparity, and directional imbalance. Experiments show that even the strongest model, Gemini 2.5 Pro, (i) struggles with spatial and temporal reasoning, achieving less than 60% accuracy, (ii) reveals persistent modality disparities, with performance dropping substantially when the same semantic content is conveyed through audio rather than text, and (iii) shows systematic directional imbalance, exhibiting lower consistency when vision serves as context compared to text. These findings indicate that current OLLMs remain far from truly modality-invariant reasoning and position XModBench as a fundamental diagnostic tool for evaluating and improving cross-modal competence. All data and evaluation tools will be available at https://xingruiwang.github.io/projects/XModBench/.