UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models

AI summary

Overview

Research area: Multimodal AI evaluation — specifically benchmarks for "omni" models that jointly process image, video, audio, and text.

Technical level: Intermediate. The benchmark design and empirical findings are accessible, but the fitted compositional law and the discussion of fusion mechanisms assume some familiarity with multimodal model architecture.

Scope: This paper introduces UNO-Bench, a single benchmark that measures both uni-modal (vision, audio) and omni-modal (joint image-audio-linguistic) capabilities under one shared ability taxonomy, and uses it to derive an empirical scaling law linking the two.

What This Paper Is About

Existing benchmarks test vision, audio, and cross-modal abilities in separate, disconnected suites, and most are English-only. Worse, several widely used omni-modal benchmarks are flawed: the paper reports that 77% of WorldSense questions can be answered without vision or audio, and 25% of OmniBench answers are incorrect, which inflates measured omni capability. UNO-Bench addresses this by building a single, high-quality, human-curated benchmark — heavily grounded in Chinese real-world scenarios — that evaluates uni-modal and omni-modal understanding under one taxonomy, then uses the resulting scores to characterize how the two relate.

Key Contributions

1. A unified benchmark spanning both capability levels. UNO-Bench evaluates uni-modal and omni-modal understanding under one shared ability taxonomy covering 44 task types and 5 modality combinations. It contains 1,250 human-curated omni-modal samples with 98% cross-modality solvability (i.e., each genuinely requires both vision and audio) and 2,480 enhanced uni-modal samples.

2. A human-centric construction pipeline with private materials. Roughly 90% of omni-modal material is privately sourced, all dialogue audio is live-recorded by over 20 speakers (including Mandarin and Sichuan dialect), and every question passes model checks, modality ablation, and multiple rounds of human review. This targets real-world Chinese usage and resists data contamination.

3. The Compositional Law linking uni-modal and omni-modal performance. The authors derive and fit a power-law relation, P_Omni ≈ 1.0332 × (P_Audio × P_Visual)^2.1918 + 0.2422, with R² = 0.9759. The exponent above 1 (super-linear) explains why omni capability behaves as a bottleneck for weak models but provides accelerating, synergistic returns for strong ones.

4. A multi-step open-ended question format plus a general scoring model. Rather than binary pass/fail multiple choice, complex problems are decomposed into scored, interdependent sub-questions totaling 10 points and posed in a single turn. A Qwen3-14B-based scoring model evaluates six question types with 95% accuracy on out-of-distribution models and benchmarks.

5. A clustering-guided hierarchical sampling (CGHS) compression method. By representing each question as a vector of model scores, clustering them with KMeans++, and sampling proportionally, the method cuts evaluation cost by over 90% while maintaining 98% consistency across 18 public benchmarks.

Main Findings

• Clear model hierarchy with a large proprietary gap. Gemini-2.5-Pro leads on every metric (Omni-MC 70.90, Omni-MO 57.32). Smaller open-source omni models hover near the 25% random-guess baseline on Omni-MC (27.80–29.70), while Qwen-3-Omni-30B-A3B reaches 42.10, approaching Gemini-2.0-Flash (44.90).

• Multi-step open-ended questions are far harder than multiple choice. Every model degrades sharply on Omni-MO. Gemini-2.5-Pro drops 13.58 points (70.90 → 57.32); the best open-source model, Qwen-3-Omni-30B, scores only 37.08. The MO format exposes ability decay along long reasoning chains that multiple choice conceals.

• Reasoning is a bigger bottleneck than perception. The gap between the best proprietary and best open-source model is 23.04 points in perception but widens to 33.00 points in reasoning. Spatial reasoning is hardest across the board — even Gemini-2.5-Pro reaches only 45.00.

• Advanced models become better at cross-modal alignment, not just recognition. Weaker models find recognition easier than alignment; stronger models like Gemini-2.5 series and Qwen-3-Omni-30B reverse this, with Gemini-2.5-Pro scoring 74.35 on alignment versus 70.05 on recognition.

• The bottleneck-versus-synergy transition is quantified. With the fitted exponent α ≈ 2.19, the curve is convex. Increasing the product of uni-modal scores yields marginal omni-modal gains at low performance (a short-board effect) but disproportionately large gains at high performance (emergent capability). The bias term b ≈ 0.2422 closely matches the 0.25 random-guess floor, and C ≈ 1.0332 is near unity, suggesting consistent scaling.

• Vision currently contributes more than audio. Across most fitted models, the visual modality offers larger benefits, likely because current models' visual capabilities lag behind their audio ones.

• Raw audio remains a weak point for open-source models. In audio ablations, several open-source models perform better with textual descriptions of environmental sounds than with the original audio, indicating audio encoders struggle with non-speech content. Speech proved easier than environment/music.

• Captions can rival raw pixels for some models. Gemini-2.5-Pro clearly benefits from direct visual input over captions (70.90 vs. 65.10), but several other models perform equally well or better with captions — suggesting their language pathway extracts semantics more effectively than their visual encoder.

• Gemini is human-comparable in perception, not reasoning. In a competition against unexposed human experts, Gemini-2.5-Pro matched perception performance but trailed in reasoning. Humans scored higher on reasoning than perception (81.3% vs. 74.3%) — the opposite of the model's profile.

Methodology in Plain English

The researchers built the benchmark from scratch rather than scraping existing datasets. They first defined a unified taxonomy splitting ability into Perception (object, attribute, scenario, spatial, cross-modal conversion, semantic understanding, cross-modal alignment) and Reasoning (general, STEM, code, spatial/static and dynamic, temporal, and complex multi-conditional multi-step). Annotators then drew from a tagged library of real-world photos, videos, and live-recorded dialogue to write questions and answers matched to each task type.

Quality control works in three layers: a model check to filter ambiguous or off-type questions, a modality ablation step that removes one modality at a time and confirms the question becomes unanswerable (proving both modalities are genuinely required), and final human revision.

For uni-modal evaluation, the team reuses existing public datasets but supplements them with self-constructed items for coverage, selects public samples on four criteria (comprehensiveness, diversity, quality, discriminability), and compresses the set with CGHS — clustering questions by their model-score signatures and sampling proportionally, verified through Spearman/Pearson correlations, RMSE, and confidence interval coverage across repeated random splits.

Finally, for the open-ended format, human experts decompose complex problems into scored sub-questions, and a fine-tuned Qwen3-14B scoring model compares predicted answers against references using type-specific criteria.

To relate uni-modal and omni-modal performance, the authors start from a general function of audio and visual scores, add a fusion-symmetry assumption, simplify to a power law, and fit it by non-linear regression against the benchmark's own results.

Why This Matters

Impact on research. The paper offers both an evaluation resource and a testable quantitative hypothesis. Previous work treated cross-modal ability as something to measure descriptively; UNO-Bench frames it as a relationship with a specific functional form and a super-linear exponent. If the law holds, it changes how researchers interpret scaling results — small uni-modal gains near the frontier should be expected to produce outsized omni-modal returns, while weak foundations cannot be compensated for by better fusion alone. The finding that 77% of WorldSense and 25% of OmniBench data is flawed also serves as a methodological warning about benchmark validity more broadly.

Real-world applications:

• Chinese-language multimodal assistants. The benchmark directly targets Chinese scenarios with live-recorded Mandarin and dialect audio, giving developers a way to evaluate voice-plus-vision assistants for that market rather than relying on translated English datasets.

• Accessibility tools. Systems that describe scenes to blind users or transcribe and interpret environmental sounds for deaf users depend on exactly the joint audio-visual grounding UNO-Bench tests, and the ablation results show where current models fail (environmental audio in particular).

• Video content understanding and moderation. Platforms processing user video need models that fuse speech, sound, and imagery; the MO question format approximates the multi-step judgment real moderation or summarization tasks require.

• Embodied agents and robotics. Spatial and temporal reasoning scores — the weakest areas measured — are prerequisites for agents that must act on what they see and hear simultaneously.

Industry relevance. The CGHS compression method addresses a practical cost problem: full multimodal evaluation is expensive, and cutting evaluation time by 90% while preserving 98% rank consistency across 18 benchmarks makes routine model iteration feasible. The general scoring model similarly replaces costly human or unreliable automated grading for open-ended outputs.

Future Directions

• Closing the reasoning gap. Spatial reasoning tops out at 45.00 even for the best model, and the perception-reasoning gap is wider between leading and open-source models than the perception gap. Whether this reflects architecture, training data, or reasoning-token budgets is unresolved.

• Improving raw audio encoding. Several open-source models perform better with text descriptions of environmental sounds than with the audio itself, which points to a specific, addressable encoder deficiency rather than a general capability limit.

• Testing whether the compositional law generalizes. The fit was derived on UNO-Bench with a limited set of models. Whether α ≈ 2.19, C ≈ 1, and b ≈ 0.24 hold on other benchmarks, other modality pairs (video-audio), or future model generations is an open empirical question.

• Expanding languages and modalities. The benchmark covers English and Chinese; extending to more languages and to additional modality combinations is a natural next step, as is broadening the MO format and scoring model to more question types.

Target Audience

Researchers and engineers building or evaluating multimodal and omni-modal models will get the most from this paper, particularly those working on Chinese-language applications or on cross-modal fusion architecture. Benchmark designers and evaluation teams will find the data-quality methodology and the CGHS compression technique directly reusable. Product and technical strategy leads assessing current omni-model readiness — especially the practical ceiling imposed by weak reasoning — will also benefit. Readers seeking a quick intuition about where multimodal AI stands will find the Compositional Law and the human-versus-model comparison the most portable takeaways.

Authors’ abstract

Multimodal Large Languages models have been progressing from uni-modal understanding toward unifying visual, audio and language modalities, collectively termed omni models. However, the correlation between uni-modal and omni-modal remains unclear, which requires comprehensive evaluation to drive omni model's intelligence evolution. In this work, we introduce a novel, high-quality, and UNified Omni model benchmark, UNO-Bench. This benchmark is designed to effectively evaluate both UNi-modal and Omni-modal capabilities under a unified ability taxonomy, spanning 44 task types and 5 modality combinations. It includes 1250 human curated samples for omni-modal with 98% cross-modality solvability, and 2480 enhanced uni-modal samples. The human-generated dataset is well-suited to real-world scenarios, particularly within the Chinese context, whereas the automatically compressed dataset offers a 90% increase in speed and maintains 98% consistency across 18 public benchmarks. In addition to traditional multi-choice questions, we propose an innovative multi-step open-ended question format to assess complex reasoning. A general scoring model is incorporated, supporting 6 question types for automated evaluation with 95% accuracy. Experimental result shows the Compositional Law between omni-modal and uni-modal performance and the omni-modal capability manifests as a bottleneck effect on weak models, while exhibiting synergistic promotion on strong models.