Closing the Modality Reasoning Gap for Speech Large Language Models

AI summary

Overview

Research area: Speech Large Language Models (Speech LLMs), specifically cross-modal reasoning alignment and reinforcement learning post-training.

Technical level: Advanced. The paper assumes familiarity with Speech LLM architectures (speech encoder + modality projector + decoder-only LLM), group-relative reinforcement learning, LoRA fine-tuning, and hidden-state representation analysis.

Scope in one sentence: The paper introduces TARS, an on-policy reinforcement learning framework with an asymmetric dense reward that aligns speech-conditioned reasoning trajectories to text-conditioned ones, evaluated on the MMSU and OBQA benchmarks with 7B-scale Speech LLMs.

Paper metadata: arXiv:2601.05543v2 [cs.CL], published 2026-01-09. Authors are affiliated with The Chinese University of Hong Kong, Shenzhen and Microsoft Corporation. The work was done during an MSRA internship. Code and datasets are released at https://github.com/AmphionTeam/TARS.

What This Paper Is About

Speech LLMs typically reuse a text-pretrained LLM as their reasoning engine, but they reason noticeably worse on speech input than on text input — a problem the authors call the modality reasoning gap. Prior fixes either freeze the LLM and only tune the input-side adapter, or apply off-policy supervision that forces the model to mimic exact text tokens, which the authors argue is an unreachable target that suffers from exposure bias. This paper's goal is to close that gap by aligning the model's reasoning trajectory rather than its inputs or its final tokens, using on-policy reinforcement learning.

Key Contributions

1. An on-policy reinforcement learning framework for trajectory alignment that aligns a speech-conditioned reasoning trajectory with its text-conditioned counterpart, closing the modality reasoning gap without any architectural modifications.

2. An asymmetric dense reward design with two complementary alignment signals: a representation alignment reward that reduces layer-wise hidden-state drift, and a behavior alignment reward that enforces semantic consistency between outputs.

3. Modality-specific advantage normalization within GRPO, so that speech-conditioned completions are normalized against other speech-conditioned completions rather than against the inherently higher-scoring text-conditioned completions.

4. State-of-the-art results among 7B-scale Speech LLMs on the reasoning benchmarks MMSU and OBQA, with released datasets and code.

Main Findings

• The gap is real and large at baseline. The base Qwen2.5-Omni reaches 71.30% average accuracy on audio versus 76.17% on text (MRR 91.76%), and Phi-4-MM reaches 63.23% audio versus 78.39% text (MRR 79.59%).

• TARS improves both models. TARS (Qwen2.5-Omni) achieves 76.84% average audio accuracy and MRR 98.89%; TARS (Phi-4-MM) achieves 79.80% average audio accuracy and MRR 100.45%.

• It beats comparable end-to-end alignment baselines. For the Qwen2.5-Omni setting, 76.84% audio accuracy exceeds SALAD-7B (66.30%) and MiniCPM-o 2.6 (66.40%), and far exceeds the Knowledge Distillation baseline (72.87%).

• It beats matched cascaded pipelines. TARS on Phi-4-MM (79.80%) surpasses the ASR + Phi-4-7B pipeline (73.40%), and TARS on Qwen2.5-Omni (76.84%) surpasses ASR + Qwen2.5-7B (75.55%), suggesting direct speech processing avoids some ASR-induced errors.

• Text reasoning is not sacrificed — it improves. Text accuracy rises from 76.17% to 78.56% for Qwen2.5-Omni and from 78.39% to 83.82% for Phi-4-MM. A "Text-only GRPO" ablation on Phi-4-MM gives 83.76% text accuracy, nearly identical to joint alignment (83.56%), while TARS gives higher speech accuracy (79.57% vs. 76.86%) and higher MRR (100.28% vs. 96.86%).

• Prompting and standard post-training are insufficient. On Phi-4-MM, CoT prompting raises average audio accuracy from 63.16% to 70.06% (MRR 79.60% to 88.29%); SFT reaches 72.52% (MRR 91.37%); DPO reaches 75.37% (MRR 94.98%); Standard GRPO with only the base reward reaches 73.17% (MRR 92.21%), still below DPO.

• The two alignment rewards are complementary. Starting from Standard GRPO (MRR 92.21% on Phi-4-MM), adding representation alignment gives MRR 95.56%, adding behavior alignment gives MRR 99.22%, and combining both gives the best result, MRR 100.28%.

• Optimal reward choice is model-dependent. On the heavily post-trained Qwen2.5-Omni, using only the behavior reward works best (average audio accuracy 77.71%, MRR 100.01%), while adding the representation reward causes over-regularization (MRR 98.39%). On Phi-4-MM, the internally trained model with minimal post-training, the combination of both rewards is best.

• Representation alignment reduces layer-wise drift. Cosine similarity between text and speech hidden states decreases with depth, but adding the representation reward consistently lifts the similarity trajectory across layers, and the joint strategy achieves the highest similarity.

• Middle layers matter most for the representation reward. In the 32-layer Phi-4-MM backbone, restricting the reward to Middle layers (11–20) achieved 75.48% accuracy, better than Shallow (1–10) or Last (31–32) layers, while using All layers gave the highest accuracy at 75.83%.

• ASR ability is unchanged. Word Error Rate on LibriSpeech stays at approximately 4.16%–4.24% across all configurations, indicating gains come from reasoning alignment rather than better speech recognition.

• Generalization to real speech holds. On a separate MMSU benchmark assessing paralinguistic features, TARS reaches 57.40% audio versus 56.70% for the Phi-4-MM baseline, while CoT prompting degrades it to 51.50%. On SD-QA, TARS reaches 41.95% audio versus 39.42% (baseline) and 39.60% (CoT).

• Consistent per-domain gains on MMSU. The largest improvements over the CoT baseline were in Business (61.02% to 75.00%, +13.98%), Biology (69.77% to 83.72%, +13.95%), Law (52.94% to 66.67%, +13.73%), and Physics (61.36% to 73.89%, +12.53%); the smallest were in History (+5.77%) and Engineering (+5.61%).

• Efficiency overhead is marginal. Compared with Standard GRPO (27.5 hours), TARS takes 28.7 hours — a 4.4% overhead — with an identical 90% peak memory footprint, while SFT and DPO take about 1.6 hours but exclude over 10.9 GPU-hours of offline data generation for 8 rollouts per prompt.

Methodology in Plain English

The authors treat a Speech LLM as a system that can answer the same question either through text or through audio and should ideally reach the same answer either way. They quantify success with a metric called the Modality Recovery Rate (MRR): the average reasoning score on speech input divided by the average score of the base text model on the text version of the same questions, expressed as a percentage. The target is MRR of at least 100%.

To get there, they train with reinforcement learning (GRPO, using the DAPO loss estimator and Dr. GRPO-style advantage normalization) and give the model a richer reward than usual. During training, each prompt produces a group of eight completions, split evenly between speech-conditioned and text-conditioned ones. Text completions are scored only on task accuracy and format. Speech completions get an extra "asymmetric" reward with three parts:

• A base reward for whether the answer is correct (checked by xFinder) plus a format-compliance reward.
• A representation alignment reward, computed by mean-pooling the generated reasoning tokens at each layer and taking the average cosine similarity between the speech and text hidden-state vectors across all layers. This tells the model whether its internal reasoning looks like the text model's internal reasoning.
• A behavior alignment reward, computed by embedding both the speech output and the text reference with an external embedding model (Qwen3-Embedding-0.6B) and taking cosine similarity. This checks whether the final answer means the same thing, without demanding identical tokens.

Because text completions naturally score higher than speech ones, the authors normalize advantages separately within each modality. This prevents every speech completion from receiving a negative advantage, which would suppress learning. A key motivation is that in speech-conditioned reasoning, the task-accuracy reward is often zero for the entire group; the dense alignment rewards still provide a usable learning signal in that case.

Training data is built from the UnifiedQA training set, with speech synthesized by CosyVoice2 and openaudio-s1-mini using reference speakers from the Emilia-YODAS EN subset, filtered by a 10% WER threshold via whisper-medium, yielding 9,953 samples (203 hours). Models are fine-tuned with LoRA on all linear layers while the audio encoder and projector stay frozen.

Why This Matters

Research impact. The paper reframes cross-modal alignment as a trajectory-alignment problem rather than an input-projection or token-matching problem, and shows that an RL objective can tolerate valid alternative reasoning paths as long as outputs are semantically consistent. It also provides a concrete, dense-reward alternative to sparse correctness-only RL for speech, a domain where the authors note RL application has been nascent.

Real-world applications:

• Voice assistants that must answer knowledge or reasoning questions as reliably by voice as by typed text.
• Accessible interfaces for users who interact with AI systems primarily through speech, including spoken question answering over multi-domain knowledge.
• Spoken dialogue systems in education or tutoring, where elementary science and commonsense questions (as in OBQA) are answered aloud.
• Deployments involving accented, real-world speech, such as call centers or field settings, where the paper's SD-QA results suggest robustness beyond synthetic TTS data.

Industry relevance. The method requires no architectural change, is applied with LoRA on 7B models, trains in roughly 55 hours for Qwen2.5-Omni and 35 hours for Phi-4-MM on 4× A100 or 8× H200 GPUs, and adds only 4.4% overhead over standard GRPO. That makes it attractive for teams already running GRPO-style pipelines who want cross-modal gains without redesigning their model stack.

Future Directions

• Scaling the study beyond 7B. The authors state explicitly that alignment was only evaluated at the 7B scale and that behavior for smaller or larger models is unknown.
• Moving from single-turn to multi-turn reasoning. The paper focuses on single-turn reasoning; multi-turn, interactive, or dialogue-driven speech reasoning may introduce dynamics the current formulation does not capture.
• Handling paralinguistic cues without textual counterparts. The alignment rewards depend on text-only reference completions, so emotion, prosody, and intent that lack explicit textual equivalents may not be fully accounted for.
• Choosing the reward configuration automatically. The results show that the best reward mix depends on how heavily the base model has already been post-trained (Phi-4-MM preferred both rewards; Qwen2.5-Omni preferred behavior-only, since representation alignment caused over-regularization), which raises the question of how to select or adapt the reward design per model without trial and error.

Target Audience

This paper is most useful to researchers and engineers working on multimodal or Speech LLMs, reinforcement learning post-training for language models, and cross-modal representation alignment. It will also interest practitioners building speech-based reasoning products who want an alignment method that does not require changing model architecture and that can be integrated into an existing GRPO training loop. Readers without a background in RL fine-tuning or transformer internals will find the reward design intuitive but the training details dense.

Authors’ abstract

Although Speech Large Language Models have achieved notable progress, a substantial modality reasoning gap remains: their reasoning performance on speech inputs is markedly weaker than on text. This gap could be associated with representational drift across Transformer layers and behavior deviations in long-chain reasoning. To address this issue, we introduce TARS, a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design. The framework employs two dense and complementary signals: representation alignment, which measures layer-wise hidden-state similarity between speech- and text-conditioned trajectories, and behavior alignment, which evaluates semantic consistency between generated outputs and reference text completions. Experiments on challenging reasoning benchmarks, including MMSU and OBQA, show that our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.