Training-Free Speech-Centric Omni Understanding with Frozen VLMs

AI summary

Overview

• Research area: Multimodal (omni-modal) understanding — specifically speech-centric audio-visual understanding, and how it relates to frozen vision-language models (VLMs).
• Technical level: Intermediate. The paper uses standard VLM, ASR, and benchmark terminology and includes mathematical notation, but its core idea is a modular pipeline rather than a new architecture.
• Scope: A systematic, matched empirical comparison of native "Omni" models against a training-free alternative that routes confidence-filtered, timestamped Whisper transcripts into a frozen VLM, evaluated across 56 benchmark datasets and 21 languages.

What This Paper Is About

Native omni models add a dedicated audio encoder to a VLM and align it with visual and language representations through large-scale audio-video-text training, which is costly and ties omni capability to one specific backbone. The authors ask whether that training is actually necessary for speech-centric omni understanding, or whether routing speech as text through a frozen VLM's existing language interface is sufficient. Their goal is to test this with a matched, training-free alternative — Training-Free Omni (TFO) — and to identify both where it matches native models and where dedicated acoustic representations remain necessary.

Key Contributions

1. TFO, a plug-and-play training-free framework. TFO converts any frozen VLM into a speech-centric omni model without architectural modification, parameter updates, or audio-video-text training. It uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, leaving the visual pathway unchanged.
2. A systematic matched comparison design. The study compares native omni models, their original VLM backbones, and the corresponding training-free conversions across five matched comparisons and four model families: Qwen2.5-VL-Instruct vs. Qwen2.5-Omni at 3B and 7B, MiniCPM-V-4.5 vs. MiniCPM4.5-O, NVILA-8B vs. OmniVinci, and Qwen3-VL-30B-A3B-Instruct vs. Qwen3-Omni-30B-A3B-Instruct.
3. Extensive evaluation. 56 benchmark datasets spanning audio-visual understanding, audio-only understanding, image and video understanding, visual grounding, coding and mathematical reasoning, medical question answering, and multilingual speech across 21 CoVoST2 languages. Counting the supplementary-only LiveSports3K ablation, the paper reports results on 57 distinct datasets overall.
4. Characterization of where language-level routing fails. The paper isolates the representational limits of transcript-based routing using AVHBench, and quantifies the practical cost in inference latency and parameter count.

Main Findings

• Audio-visual understanding is competitive. Across 9 audio-visual omni benchmarks, TFO improves the Qwen2.5 average by +3.0 points for 3B and +2.2 for 7B, while VILA gains +0.4; MiniCPM4.5 and Qwen3 decrease by 2.2 and 0.8 points respectively. The strongest gains at both Qwen2.5 scales appear in speech-conditioned and temporal video reasoning: WorldSense by +6.1 / +14.5, Video-Holmes by +8.1 / +4.3, AVUT-Human by +8.6 / +6.5, and Daily-Omni by +6.9 / +4.7.
• Audio-only understanding improves across every setting. Across 9 audio-only benchmarks, the average improves in all five model settings by +1.5, +1.4, +4.0, +13.5, and +1.4 points. The largest gain is on VILA (50.3 to 63.8). CoVoST2 and VoiceBench improve across every model family, including VoiceBench gains of +51.3 on VILA and +11.0 on MiniCPM4.5.
• Multilingual speech shows the largest and most consistent gains. Across 21 CoVoST2 languages, TFO improves the average in all five settings by +7.9, +13.5, +18.4, +11.1, and +6.8 points, with the largest gain on MiniCPM4.5 (45.6 to 64.0). The biggest improvements occur where native omni models are weakest, including Estonian (+43.6), Latvian (+35.4), and Swedish (+35.8) on MiniCPM4.5, and Swedish (+54.9) and Turkish (+46.6) on Qwen2.5-7B.
• Image and video capabilities are preserved. Across eight image and up to six video benchmarks, TFO achieves higher image averages in all five settings with gains of +1.8, +1.0, +0.7, +0.3, and +2.4 points; the clearest image gains (except for NVILA) are on document, chart, and OCR-intensive tasks. Video averages are higher in four settings by +3.7, +3.8, +2.3, and +2.9 points, while MiniCPM4.5 is nearly unchanged. VideoMME improves across every model family by +0.4 to +10.5 points.
• Coding is consistently retained; math is more model-dependent. Across seven coding and math benchmarks, TFO achieves higher overall averages in three of four settings, with gains of +4.3, +2.9, and +2.5 points on Qwen2.5-3B, Qwen2.5-7B, and Qwen3. Every TFO variant beats its native omni counterpart on MBPP, MBPP Sanitized, and HumanEval. Larger visual-math gains include +15.5 and +9.0 on MathVerse for 3B and 7B, and +11.3 on MathVista for Qwen3. NVILA is the main exception, with a -2.8 point average decline.
• Medical question answering improves across all five settings. Across 12 medical QA benchmarks, TFO gains +1.7, +1.2, +1.6, +0.5, and +1.5 points on average. MedMCQA, MedQA-USMLE, and OmniMedVQA increase in every setting; larger gains include +9.8 on MedFrameQA for Qwen3, +5.6 on PMC-VQA for MiniCPM4.5, and +6.0 on MMMU-Med-val for both Qwen2.5 scales.
• Visual grounding is retained or improved. PixMo-Count improves across all five settings, including +15.5 on VILA and +11.2 on Qwen3. PixMo-Point error decreases in four settings and ties on VILA; PointArena improves in four settings and ties on VILA. RefCOCO is more model-dependent, with a particularly large gain for Qwen3.
• The core limitation is non-speech acoustic evidence. On AVHBench, TFO degrades on AV Matching and video-driven audio hallucination (V to A), which require verifying non-speech sound, but improves on audio-driven video hallucination (A to V) across all four model families because the visual pathway is preserved. All variants also decline on MMAR-Bench by 1.5 to 9.5 points.
• Auxiliary audio models do not solve the limitation. Appending textual predictions from MELLOW and SenseVoice produces isolated gains, but neither consistently outperforms Whisper alone.
• Timestamps and the frozen VLM each contribute independently. In the ablation with Qwen2.5-VL-3B on six audio-visual benchmarks, the frozen VLM alone outperforms Qwen2.5-Omni on WorldSense and Video-Holmes; adding Whisper improves all benchmarks and surpasses Qwen2.5-Omni on five of six. Removing segment-level timestamps reduces AVUT-Gemini and Daily-Omni by 2.3 and 1.4 points respectively.
• The cost shifts from training to inference. On AVMeme, total latency rises from approximately 0.7–2.4 s for native omni models to 1.3–3.1 s for TFO, with the overhead being sequential ASR inference that can be amortized when questions share one audio-video input. Parameter counts remain comparable, with smaller Qwen2.5 variants and only a marginal increase for MiniCPM4.5.

Methodology in Plain English

The researchers did not train anything. They took an already-frozen VLM and added an external speech recognizer in front of it. Whisper transcribes the audio into segments, each carrying the transcribed words, the detected language, start and end times, and a confidence score. Only segments with confidence at or above a threshold of 0.65 are kept; if none pass, the audio context is simply omitted and the VLM receives its original image/video-text prompt.

The retained segments are turned into an "audio context" and inserted into the VLM's normal prompt alongside the system instruction, the visual input, and the user question. Because speech enters only through the language channel, the VLM's visual encoder and all of its parameters stay untouched — no new audio tokens, no connectors, no re-alignment. For tasks where spoken content must be tied to visual events over time, the timestamps are kept as anchors. For optional spoken responses, CosyVoice3 converts the VLM's text output into speech; it is used only for output generation and never to build or modify evaluation audio.

To test the approach fairly, the team built matched pairs — each native omni model against a training-free conversion of its own VLM backbone at the same scale — and ran both through the same benchmarks and prompting. Multiple-choice tasks are scored with exact-match accuracy, and long-form generation is judged by an LLM evaluator (GPT-5.6, referred to as GPT-5.6 Sol). All inference is run once per benchmark with greedy decoding and temperature set to zero on AMD Instinct MI210 GPUs with 64 GB of memory per GPU, using a unified evaluation toolkit called OmniEvalKit.

Why This Matters

The paper reframes omni capability as a routing problem rather than a training problem, and it provides the matched comparison that prior omni evaluations lacked. If a frozen VLM plus a strong ASR front-end can reach or exceed native omni performance on speech-centric tasks, then building a new omni model for every new VLM generation becomes optional — and the measurable capability drift that native training introduces on image/video understanding, coding, math, grounding, and medical QA becomes avoidable. Equally important, the paper draws a clear boundary: music, environmental sounds, vocal tone, and emotion cannot be recovered from a transcript, so dedicated acoustic training remains necessary there.

Real-world applications implied by the work:

• Multilingual speech interfaces — the largest gains were on CoVoST2 across 21 languages, including languages where native omni models scored near zero, which matters for speech assistants that need broad language coverage without per-language backbone retraining.
• Video and meeting understanding — timestamped transcripts let a frozen VLM connect what was said to what was happening on screen, applicable to long-video question answering and temporal event reasoning without audio-video-text training.
• Accessibility and transcription-grounded assistance — routing speech as language evidence, with optional CosyVoice3 spoken output, supports spoken-query and spoken-response systems built on existing VLM backbones.
• Domain-specific assistants where knowledge retention matters — medical QA improved across all five settings, suggesting the approach suits deployments that cannot afford to degrade an existing model's domain knowledge to gain audio input.

Industry relevance: the approach eliminates backbone-specific audio training and multimodal re-alignment, meaning teams can upgrade to a stronger VLM and inherit speech-centric omni capability through a modular ASR front-end. The trade-off is a shift of cost to sequential inference latency, which is amortizable when many questions share one audio-video input. The authors release OmniEvalKit as an open-source unified evaluation toolkit for multimodal and omni-modal models.

Future Directions

• Combining routing with targeted acoustic training. The authors' own conclusion proposes pairing external ASR-based speech routing with careful acoustic training while preserving the mature capabilities of the underlying VLM, rather than choosing one or the other.
• Recovering non-speech acoustic evidence. The paper shows that appending textual predictions from MELLOW and SenseVoice does not consistently beat Whisper alone, leaving open how music, environmental sounds, vocal tone, and emotion could be represented without reintroducing costly backbone-specific alignment.
• Reducing ASR latency. Sequential Whisper inference raised AVMeme latency from approximately 0.7–2.4 s to 1.3–3.1 s; the paper notes amortization when questions share one input but does not resolve the general case.
• The confidence threshold and routing design. The threshold is fixed at 0.65 and the router is instantiated with Whisper; whether other thresholds, ASR systems, or routing strategies change the trade-off is not reported.

Target Audience

Researchers and engineers working on multimodal and omni-modal models, speech-language integration, and VLM evaluation. It is most useful for those deciding whether to train a native audio pathway for a new VLM backbone or to bolt on an ASR front-end, and for anyone designing matched evaluations that measure capability retention in addition to newly acquired capability. Readers should be comfortable with VLM and ASR terminology and benchmark-based empirical comparisons; the underlying idea is simple enough that the paper does not require deep expertise in audio encoder design.

Authors’ abstract

Audio-visual understanding remains challenging because models must jointly interpret spoken content, visual events, and their temporal relationships. Existing omni models typically introduce dedicated audio encoders and rely on expensive audio-video-text training, tightly coupling omni capability to specific VLM backbones and potentially weakening their existing visual and reasoning abilities. This raises three questions: whether native omni training is necessary for every new VLM, whether speech-centric omni capability can be added while preserving the original backbone, and where richer acoustic representations remain essential. We introduce Training-Free Omni (TFO), a plug-and-play framework that converts a frozen VLM into a speech-centric omni model without architectural modification, or multimodal re-alignment. TFO uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, while leaving its visual pathway unchanged. Across matched comparisons with native omni models on 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves average audio-only performance across all five model settings, and achieves substantial multilingual speech gains. Freezing the VLM also generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical question answering than the corresponding native omni checkpoints. These results show that strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.