Omni-Modal LLMs Know More Than They Say

AI summary

Overview

Research area: Natural Language Processing / omni-modal large language models (models that jointly handle text, vision, audio, and sensor signals such as IMU data).

Technical level: Intermediate. The paper introduces a measurement tool (Token-level Modality Alignment) and a prompting-style strategy, so readers should be comfortable with how multimodal LLMs encode inputs and generate responses, but the core ideas are described conceptually rather than through heavy math in the abstract.

Scope (one sentence): The paper diagnoses why omni-modal LLMs fail to verbalize non-textual information they actually perceive, measures that gap with a token-level alignment metric, and proposes a simple self-generated-text strategy that improves performance and balances modality preferences across seven models.

What This Paper Is About

Omni-modal LLMs can take in images, audio, and motion-sensor signals alongside text, but the authors observe a recurring failure: the model correctly perceives information from a non-textual modality and still does not express it in its answer. The paper names this disconnect "knows-but-cannot-say" and sets out to explain it rather than just report it. The goal is twofold: build a measurement that exposes the problem inside the model, and design a fix that lets non-textual knowledge surface in the generated response.

Key Contributions

1. Identifies and names a failure mode. The authors document that omni-modal LLMs often possess correct non-textual information but fail to state it, framing this as "knows-but-cannot-say."

2. Introduces Token-level Modality Alignment (TMA). A measurement of how well individual modality representations line up with the representations used when producing response tokens, giving a per-modality diagnostic rather than a single end-task score.

3. Explains uneven modality preferences. Using TMA, the analysis shows non-textual representations are less aligned with response tokens and attract less attention than textual ones, offering a mechanistic account of why text dominates model behavior.

4. Proposes OmniCatalyst. A perceive → infer strategy that supplements the original modality inputs with model-generated text describing what was perceived, making non-textual knowledge easier for the model to use; it is reported to improve performance across seven Omni-LLMs (+7.29%) and to yield more balanced modality preferences, with further gains from diverse training strategies.

Main Findings

• The disconnect is real and distinct from perception failure: the models "often fail to express information from non-textual modalities even when they perceive it correctly," so the bottleneck sits in expression, not intake.

• Non-textual representations are less aligned with responses: TMA measurements show modality representations from non-textual inputs are less aligned with response-token representations than textual ones.

• Non-textual inputs receive less attention: the same representations draw less attention during generation, which the authors present as a plausible explanation for the models' uneven modality preferences.

• A simple text-bridging strategy helps broadly: OmniCatalyst improves performance consistently across seven Omni-LLMs, with a reported average gain of +7.29%.

• Improvements go beyond raw accuracy: the strategy also produces more balanced modality preferences, and combining it with diverse training strategies yields further gains.

• The mechanism appears to be alignment: analysis shows the knowledge-extraction phases of OmniCatalyst increase TMA, suggesting that better alignment is what makes non-textual information more accessible at response time.

• Details not given in the abstract: per-modality breakdowns, individual model scores, benchmark names, and the specific diverse training strategies are not stated in the abstract and would require the full paper.

Methodology in Plain English

The authors work in three steps. First, they probe the model's internals by comparing, at the level of individual tokens, how closely each modality's internal representation matches the representations used to produce the answer — this is TMA. Low alignment means the information is present but not in a form the response-generation process readily draws on. They pair this with attention analysis to see whether non-textual inputs are being under-weighted.

Second, they act on the diagnosis. If the problem is that non-textual knowledge is poorly aligned with language-based response generation, the natural fix is to convert it into language before answering. OmniCatalyst does this in two stages: a perceive stage where the model generates text capturing what it took from the non-textual input, and an infer stage where that self-generated text is added to the original inputs and the final answer is produced. Because the extra text is generated by the model itself, no external annotation or new modality-specific module is required.

Third, they validate: apply the strategy to seven omni-modal LLMs, measure end-task performance and modality-preference balance, test combinations with different training strategies, and re-measure TMA to confirm that the gains coincide with increased alignment.

Why This Matters

Research impact: The paper reframes multimodal LLM failures as an alignment-and-access problem rather than a perception-capacity problem, and it supplies a reusable diagnostic (TMA) that other researchers can apply to different modality pairs. It also argues that standard end-task benchmarks can understate what a model knows, which affects how omni-modal progress is measured.

Real-world applications:

• Assistive and accessibility tools that must describe audio, visual, or sensor context the user cannot supply in words.
• Wearables and robotics using IMU data, where the model must convert motion signals into explicit, actionable statements for a human operator.
• Multimodal clinical or diagnostic triage, where image or signal evidence must be stated in the written report, not merely sensed.
• Industrial and IoT monitoring, where sensor anomalies need to be verbalized in incident summaries and alerts.

Industry relevance: OmniCatalyst is described as a simple strategy applied on top of existing models rather than a retraining-heavy architectural change, which makes it attractive for teams that want better use of non-textual inputs without rebuilding their stack. The modality-balance finding also matters for product teams whose systems quietly ignore audio or sensor channels in favor of text and images.

Future Directions

• Extending TMA beyond the studied modalities and tasks: whether the same alignment gap and the same fix hold for other sensor types and longer-horizon reasoning is untested in the abstract.
• Reducing the cost of two-stage inference: a perceive → infer pipeline adds a generation step, so latency and compute trade-offs, and whether the intermediate text can be shortened or internalized, remain open.
• Training alignment directly instead of prompting around it: if low TMA is the cause, training objectives or data mixtures that raise alignment natively could outperform an inference-time workaround.
• Explaining why text dominates: the abstract identifies misalignment and low attention but does not settle whether the root cause is pretraining data balance, architecture, or instruction tuning.
• Standard evaluation for "knows-but-cannot-say": a benchmark that separates what a model perceives from what it says would let the community track this gap systematically.

Target Audience

Researchers and engineers working on multimodal and omni-modal LLMs, especially those studying modality fusion, modality bias, or interpretability of internal representations. It is also useful for practitioners deploying multimodal systems who need a lightweight way to get more out of non-textual input channels, and for evaluation-focused readers interested in metrics that go beyond end-task accuracy. Readers new to the area will follow the high-level argument but may need background on token-level representations and attention to fully engage with TMA.

Authors’ abstract

Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks. Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly. We term this disconnect knows-but-cannot-say. To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations. Our analysis shows that non-textual representations are less aligned with response tokens and receive less attention than textual representations, suggesting a potential explanation for uneven modality preferences. Motivated by these findings, we propose OmniCatalyst, a simple perceive $\rightarrow$ infer strategy that complements the original modality inputs with self-generated text, making non-textual knowledge more accessible. Across seven Omni-LLMs, OmniCatalyst consistently improves performance (+7.29%) and yields more balanced modality preferences, with further gains from diverse training strategies. Further analysis demonstrates that the knowledge extraction phases increase TMA, suggesting that improved alignment makes non-textual information more accessible during response generation. In summary, our findings uncover how Omni-LLMs can better express modality knowledge through language. Our code is available at https://github.com/naver-ai/OmniLLM.