TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

AI summary

Overview

Research area: Multimodal evaluation and benchmarking for omni-modal large language models (natural language processing / multimodal reasoning).

Technical level: Intermediate. The benchmark's construction and scoring rules are explained with plain definitions, but readers will benefit from familiarity with multiple-choice evaluation, ablations, and group-level accuracy metrics.

Scope: The paper introduces TRIAD, a 327-query, text–image–audio benchmark whose items are built so that only the full triplet determines a unique answer, and uses it to diagnose how 14 omni-modal systems behave when ambiguity must be resolved across all three channels.

What This Paper Is About

Everyday communication is often ambiguous in a single channel — a sentence has two readings, a photo could show two scenes, a tone of voice could signal sincerity or sarcasm — and the intended meaning only becomes clear when the channels are combined. Existing multimodal benchmarks mostly test perception, recognition, or reasoning over inputs that are already unambiguous, so it has been unclear whether omni-modal models can resolve ambiguity that is distributed across text, vision, and audio. TRIAD is built to isolate exactly that capability: each item is deliberately constructed so that any two modalities leave several answers plausible, and only the full triplet pins down one.

Key Contributions

1. An 18-category ambiguity taxonomy. Six leaf categories per modality: text (anaphoric, syntactic, logical, pragmatic, temporal, lexical), image (semiotic, grounding, spatial, kinematic, observational, contextual), and audio (referential, prosodic, indexical, phonetic, environmental, symbolic). Each of the 327 queries carries one label per modality.

2. The TRIAD benchmark itself. 327 hand-written queries organized into 86 scene groups, spanning English (184 queries) and Chinese (143 queries), with group sizes of 2, 3, 4, or 8 (mean 3.8) and option sets of 2, 3, 4, 5, or 8 choices. A formal "mutual-disambiguation" criterion is enforced: the full triplet yields exactly one consistent answer, while every two-modality subset yields more than one. Well-posedness is verified by five annotators using a modality-subset checklist, and a Python reference checker is released for future extensions.

3. A scene-group diagnostic protocol. Sister queries within a group share a core scenario and answer space but perturb one modality so the correct answer changes. This yields two metrics — query accuracy (Acc_q) and group accuracy (Acc_g), where a group counts as correct only if every sister is answered correctly — with Acc_g as the primary metric.

4. A 14-system evaluation under three prompt formats and eight modality conditions, including human and random baselines, ambiguity-tier and category breakdowns, and counterfactual inputs that replace audio with a transcript and the image with a description.

Main Findings

• Item-level accuracy overstates scene-level disambiguation. Gemini 3.1 Pro reaches 90.4 query accuracy versus a 95.1 human reference on a 100-query stratified subset, but only 71.2 group accuracy versus 91.6 for humans. The gap grows from 4.7 points to 20.4 points when moving from queries to groups.

• Hiding the option list causes a collapse. Under open-ended prompting, Gemini 3.1 Pro falls from 90.4 to 18.4 query accuracy and from 71.2 to 4.6 group accuracy, while humans remain at 88.4 query and 79.2 group accuracy. Every model drops to low single digits in open-ended group accuracy.

• Ablations reveal shortcut dependence rather than robustness. For Gemini 3.1 Pro, removing the image or the audio still leaves 57.2 and 51.8 group accuracy (Standard prompt), and text-only reaches 42.6, even though the benchmark admits an item only when no two modalities suffice. Removing text hurts most (16.4 group accuracy), suggesting text acts as an anchor while image and audio serve as confirmation.

• Accuracy degrades with the amount of distributed ambiguity. For Gemini 3.1 Pro, accuracy falls from 95.4 on 1-ambiguous items to 94.1 on 2-ambiguous and 87.8 on 3-ambiguous items, while humans stay above 93 in all tiers (97.8, 96.2, 93.6). Items are distributed as 230 with one internally ambiguous modality, 59 with two, and 38 with three.

• Consistency breaks down as groups grow. For the strongest model, per-query accuracy moves only from 70.5 to 68.0 as group size goes from 2 to 8, while group accuracy falls from 51.8 to 8.9 — indicating a failure of counterfactual control rather than merely harder items.

• Model ranking is consistent but far below human levels. Proprietary systems lead: Gemini 3 Pro (89.2 query / 68.9 group), Gemini 3 Flash (82.5 / 59.4), Gemini 2.5 Pro (80.7 / 55.3), GPT-4o (76.3 / 49.8), Gemini 2.5 Flash (73.2 / 45.6). The best open-source system, Qwen3.5-Omni-Plus, reaches 78.0 / 52.8; Baichuan-Omni-1.5 is lowest at 52.9 / 20.4. The random baseline is 24.3 query and 2.2 group accuracy.

• Audio is underused when text and image are present. Leave-one-out results and transcript/description counterfactuals show that textualizing audio or describing the image recovers only part of full-input performance, and that removing audio often changes little at the query level.

• Human ceiling is not saturated either. Humans reach 95.1 query and 91.6 group accuracy under Standard multiple choice, and 88.4 / 79.2 open-ended, showing a substantial but not perfect margin over the best model.

Methodology in Plain English

The team hand-built items rather than scraping them. Each item is a text sentence, an image, an audio clip, a question, and a small set of answer options. The defining rule is structural: with all three modalities present, exactly one option is consistent with the evidence; with any two modalities, at least two options remain consistent. Annotators verified this with a checklist that records, for every one- and two-modality subset, at least two options that stay plausible, and exactly one option for the full triplet. About 40 percent of candidate items survived the full process of authoring, exclusion checks, subset checklist, independent answer checking, and group integration. Five annotators (three with NLP backgrounds, two with linguistics training) participated; agreement on the labelled answer was Cohen's κ = 0.84 and on modality ambiguity tags κ = 0.79.

Items are clustered into scene groups whose sisters differ by one modality at a time, so a model must notice which cue changed rather than reuse a fixed scene prior. Audio quality was treated as a first-order concern: text-to-speech-only clips are never admitted, voice acting was the default for prosody and speaker-dependent cues (four English and two Mandarin voice actors), roughly a third of audio came from permissively licensed natural recordings, and every clip must contribute information beyond a transcript.

Evaluation covers 14 systems that accept text, image, and audio natively, run at temperature 0 with one sample per query under Standard multiple choice, chain-of-thought multiple choice, and open-ended answering. Grading for closed formats is exact option match; open-ended responses are scored by two independent judge models with the mean reported, and empty or refusal outputs count as incorrect. A separate block of conditions tests empty input, single-modality input, and leave-one-out views to probe shortcut use.

Why This Matters

The paper reframes omni-modal evaluation from "can the model recognize what is in each channel" to "can the model let three individually insufficient cues jointly determine one interpretation." That distinction matters because a model can score highly on recognition and still be unreliable whenever real-world evidence is partial or conflicting.

Real-world applications:

• Assistive and accessibility systems that must infer intent from a spoken request, a photo, and surrounding sound, where any single channel is often underspecified.
• Human–robot and in-vehicle assistants, where a gesture, a phrase, and a background alarm (or prosody) only jointly specify what the user wants.
• Content moderation and media verification, where sarcasm, tone, and scene context change whether a statement or image is benign or harmful.
• Clinical and call-centre documentation, where speaker traits, prosody, and environmental sound carry information a transcript loses.

Industry relevance: The closed-to-open collapse (90.4 to 18.4 query accuracy for the best model) suggests that leaderboard scores obtained with option lists may overstate deployable capability, and the ablation results indicate that systems often treat audio and vision as confirmation rather than as evidence that can overturn a hypothesis. Both are actionable signals for teams building or procuring omni-modal systems.

Future Directions

• Scale and coverage. The authors note that TRIAD is a diagnostic benchmark rather than a training corpus, that some strata are underrepresented — especially fully three-way ambiguous items, which number only 38 — and that language and audio coverage remain narrower than real-world use.

• Broadening linguistic and acoustic diversity. The limitations section calls for expansion across languages, cultures, and noisier audio settings, along with stronger human-centred validation.

• Reducing reliance on human and judge-model decisions. Both the well-posedness checks and open-ended grading still depend on human annotators or judge models; the released reference checker is a step toward automating part of this.

• Understanding premature commitment. Manual error analysis identifies transcript-only listening, image-anchored commitment, option priors, and cross-modal contradiction as surface forms of the same behaviour: models commit to an early hypothesis from the most language-like signal and then use other modalities to justify it. How to train models that let later evidence revise an interpretation remains an open question.

Target Audience

Researchers and engineers working on multimodal and omni-modal models who need a diagnostic that goes beyond recognition accuracy; benchmark designers interested in the mutual-disambiguation construction and the group-level metric; and evaluation or product teams who want to know how much of a model's reported multiple-choice performance survives when the answer options are removed. Linguists and speech researchers will also find the 18-category taxonomy and the audio collection protocol useful as a template.

Authors’ abstract

Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current multimodal benchmarks mostly test perception, recognition, or reasoning over already determinate inputs, leaving unclear whether omni-modal large language models can resolve ambiguity distributed across text, vision, and audio. We introduce TRIAD, a benchmark for omni-modal ambiguity resolution. Each item is a text--image--audio triplet paired with a question and a set of answer options, and is constructed so that the full triplet determines a unique answer while removing any one modality makes the set of consistent options non-unique. TRIAD contains $327$ hand-written queries in $86$ scene groups and covers $18$ ambiguity categories across the three modalities. Evaluating $14$ omni-modal systems reveals that strong item-level performance does not translate into robust scene-level disambiguation: the best model still trails humans by over $20$ points in group accuracy. Removing answer options widens the gap further, driving model group accuracy to the single digits while humans remain robust. Leave-one-out and transcript/description counterfactuals further show that current systems often underuse audio cues when text and image are present. TRIAD reframes omni-modal evaluation around disambiguation rather than recognition, and provides a taxonomy, benchmark, and diagnostic protocol for measuring this capability.