Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models

AI summary

Overview

Research area: Natural Language Processing / speech-language modeling — specifically the internal representation mechanisms of Large Speech Language Models (LSLMs).

Technical level: Intermediate. The paper is empirical and expository rather than mathematically dense, but it assumes familiarity with transformer layers, embeddings, cosine similarity, LoRA fine-tuning, and multimodal alignment training.

Scope: A systematic empirical study of why end-to-end speech LLMs perform worse than text pipeline systems, traced through layer-wise and token-wise similarity between speech and text representations, plus small inference-time interventions on those representations.

What This Paper Is About

End-to-end Large Speech Language Models that take speech input directly consistently score worse than traditional systems that chain an ASR model to a text LLM, even when both use the same underlying language model. This paper argues that the dominant cause is not a general loss of ability from fine-tuning, but a modality gap — a performance disparity between speech and text inputs to the same trained model. The authors set out to measure that gap, describe what it looks like inside the network at both sequence and token granularity, and test whether directly manipulating internal representations can close it.

Key Contributions

1. Isolating the source of the performance gap. The authors show through controlled comparison that after speech-text alignment training, the larger share of the degradation comes from the speech-versus-text modality difference rather than from fine-tuning itself — an average drop of 16.46% is attributed to speech-text alignment versus 8.79% to fine-tuning-induced erosion.
2. A coarse-grained (sequence-level) analysis. They track cosine similarity and Euclidean distance between mean speech and text representations across every layer and training checkpoint, and quantify how strongly these metrics correlate with the benchmark-level modality gap.
3. A fine-grained (token-level) analysis and a new metric. They report a spontaneous monotonic alignment path between text tokens and speech frames, and introduce the Alignment Path Score (APS) to quantify token-level alignment quality, showing it correlates with the modality gap more strongly than sequence-level similarity.
4. Causal probing via intervention. They apply angle projection and length normalization to selected speech token embeddings at inference and measure the effect on the sd-qa subset of VoiceBench, providing the paper's claim to the first empirical evidence that targeted representation interventions can improve speech-input correctness.

Main Findings

• The modality gap is large and general. Averaged across the studied models, LSLMs show a 25% performance decline on speech inputs relative to their base models on text. This splits into an 8.79% average drop from fine-tuning erosion of reasoning/generation and a 16.46% average drop from suboptimal speech-text alignment.

• The gap is not unique to this setup. The authors report observing a similar trend across diverse public LSLMs in their Appendix B.1.

• Public benchmark contrast motivates the study. On VoiceBench, the Whisper-large-v3 + LLaMA-3.1-8B pipeline reaches 79.06 overall while the LSLM LLaMA-Omni scores 37.51. On Uro-bench, Whisper-large-v3 + Qwen2-7B-Instruct reaches 78.13 versus Freeze-Omni's 48.28, despite sharing the same underlying LLM.

• Deeper layers align in direction but diverge in magnitude. Cosine similarity between mean speech and text representations increases consistently with network depth, while Euclidean distance also increases, accelerating in deeper layers. Later training checkpoints produce higher cosine similarity across all layers and slightly lower Euclidean distances overall — directional convergence with preserved modality-specific magnitude.

• Sequence-level similarity predicts the modality gap — especially under LoRA. Under LoRA fine-tuning, cosine similarity correlates with the gap at R² = 0.75. Euclidean distance correlates at R² = 0.64 for Qwen and R² = 0.88 for Llama. Under full-parameter fine-tuning the same trend holds but weaker: R² = 0.39 for cosine, and 0.53 (Llama) and 0.27 (Qwen) for Euclidean distance. The authors suggest low-rank adaptation preserves pretrained text representations, whereas full fine-tuning introduces representational flexibility and possible overfitting.

• A monotonic token-level alignment path emerges spontaneously. The token-wise similarity matrix shows extreme values (max similarity or min distance) along a nearly monotonic path across all models and training paradigms. This path does not strictly follow the main diagonal but reflects the temporal alignment structure between speech and text. At the final training epoch, the average Spearman coefficient is 0.85 for cosine similarity and 0.70 for Euclidean distance; the proportion of tokens with perfectly identical alignment paths under both metrics is 0.59.

• APS correlates with the gap more strongly than sequence-level metrics. For LoRA-trained models, APS yields R² = 0.81 for cosine (versus 0.75 for the sequence-level metric) and Euclidean APS gives 0.72 for Qwen (versus 0.64) and 0.95 for Llama (versus 0.88). Under full-parameter fine-tuning, APS correlates similarly poorly to the sequence-level metrics.

• Angle projection helps; length normalization mostly hurts. On the sd-qa subset, angle projection yields improvements or maintains performance in 6 out of 8 intervention settings. For LoRA-tuned models it consistently helps whether applied to the Bottom3 or All tokens. Applying it to only the three least-aligned tokens gives the more robust gains — Llama3.1-8B improving by 7.52% and Qwen2.5-7B by 4.19% (as reported by the authors). Length normalization improves only one case and degrades the rest.

• Selected sd-qa results (Table 1). Qwen2.5-7B full-parameter: Text 43.58, Speech 33.45, Angle Projection Bottom3 38.52 / All 38.70, Length Normalization Bottom3 37.25 / All 32.73. Qwen2.5-7B LoRA: Text 47.92, Speech 38.88, Angle Bottom3 40.51 / All 38.88, Length Bottom3 38.16 / All 31.10. Llama3.1-8B full: Text 41.05, Speech 36.53, Angle Bottom3 32.91 / All 34.00, Length Bottom3 31.65 / All 30.74. Llama3.1-8B LoRA: Text 53.16, Speech 45.75, Angle Bottom3 49.19 / All 47.74, Length Bottom3 42.86 / All 39.96.

• Two error categories are corrected by intervention. Case analysis distinguishes (1) fixing semantic misunderstandings from misinterpreting spoken content (for example, hearing "Brittany" as "Britain") and (2) rectifying factual errors despite correct semantic parsing.

• Pipeline baselines remain stronger. On the identical 4,947-sample test set, ASR+LLM pipelines using Whisper-large-v3 score: Qwen2.5-1.5B 61.72, Llama3.2-3B 65.82, Qwen2.5-7B 77.31, Llama3.1-8B 72.39 overall.

Methodology in Plain English

Setup. Every model in the study uses the same three-part recipe: Whisper-large-v3 as the speech encoder, a lightweight adapter with two fully connected layers that compresses the audio sequence by a factor of 5, and an LLM backbone. Four backbones are tested — LLaMA3.2-3B-Instruct, LLaMA3.1-8B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen2.5-7B-Instruct — under both full-parameter and LoRA fine-tuning.

Data. Training data follows the Ke-Speech-Chat framework and contains only single-turn dialogues, each with a speech instruction, a matching text instruction, and a text response. Raw text was rewritten with Qwen2.5-72B-Instruct into more natural conversational form, then filtered in three stages for safety, semantic clarity, and linguistic naturalness. Speech was synthesized with CosyVoice, and any sample whose Whisper-large-v3 transcription exceeded a 0.1 WER threshold was discarded. The final set has 637,283 samples totaling 1,604 hours of speech instructions.

Training. Two epochs with AdamW at a peak learning rate of 2e-5. LoRA uses r = 8, α = 4.0, dropout 0.1. Experiments ran on 2 nodes with 8 NVIDIA A100 GPUs each, taking roughly 384 GPU hours per model.

Evaluation. Five VoiceBench subsets — AdvBench, IFEval, OBQA, MMSU, and sd-qa — totaling 4,947 test samples after filtering, covering 93% of full VoiceBench. These subsets were chosen partly because their metrics need no auxiliary LLM judge, reducing variability, and the official VoiceBench protocol is followed throughout.

Analysis. For the same input in both modalities, they push the speech version and the text transcription through the model, average the hidden representations over the sequence dimension at each layer, and compute cosine similarity and Euclidean distance between the two modalities' means. They then correlate an average of these layer-wise scores with the observed benchmark gap.

Token-level analysis. Instead of averaging away the sequence, they build a full token-by-token similarity matrix between speech frames and text tokens at each layer, and find for each text token whichever speech frame is maximally similar (or minimally distant). This traces an alignment path. They measure how monotonic that path is with Spearman correlation, and define the Alignment Path Score as the average value along the path, normalized by layers and text tokens.

Interventions. To test causality rather than mere correlation, they pick the three lowest-alignment tokens (Bottom3) or the whole path (All) on sd-qa examples, and either rotate the speech embeddings to match the direction of the corresponding text embeddings (angle projection) or rescale their norms to match (length normalization), then re-measure accuracy.

Why This Matters

For research. The paper reframes LSLM underperformance as a within-model speech-versus-text problem rather than a simple capability deficit, and supplies a measurable internal signal — APS — that tracks downstream gap more tightly than generic representational similarity. It also gives a first causal probe, showing that the correct research direction may be representation-level alignment rather than more data or larger adapters. The finding that token-level alignment outperforms sequence-level alignment as a predictor is a useful methodological pointer.

Real-world applications:

• Voice assistants and conversational agents that must answer spoken questions as accurately as typed ones, where a measurable internal alignment score could act as a training or deployment-time quality signal.
• Accessibility tools for users who rely on speech rather than text, where the same question spoken aloud currently yields materially worse answers.
• Customer-service and call-center automation, where end-to-end speech models are attractive for latency but currently lose semantic accuracy against ASR-plus-LLM pipelines.
• Multilingual or embedded speech interfaces, where a cheap inference-time correction on a handful of poorly aligned tokens is more practical than retraining a large model.

Industry relevance. The absolute numbers matter commercially: pipelines reach 61.72 to 77.31 overall on the same test set while end-to-end models leave a large speech-versus-text gap. Because the proposed fixes are applied post hoc at inference and can target as few as three tokens per sample, they suggest a low-cost deployment-time mitigation path for teams already running end-to-end speech models.

Future Directions

1. Integrate the interventions into training. The authors explicitly note that their interventions are post hoc analysis probes, and that a key direction is to optimize cross-modal consistency during training rather than patching it at inference.
2. Broaden architectural and scale validation. The limitations section states that main experiments focus on a specific set of architectures and alignment frameworks, and that broader validation on larger models and fundamentally novel alignment strategies is still needed.
3. Move beyond English single-turn synthetic speech. The evaluation centers on English, single-turn dialogue with synthesized speech, leaving multi-turn conversations, other languages, and noisy real-world input unaddressed.
4. Determine why length normalization fails. Angle projection helps in most settings while length normalization hurts in nearly all of them, which the paper reports but does not explain mechanistically — an open question about what magnitude divergence is doing for the model.

Target Audience

Researchers and engineers working on speech-language models, multimodal alignment, and speech-to-text integration will get the most from this paper. It is also useful for practitioners deciding between pipeline and end-to-end speech architectures who want evidence about where the accuracy is actually lost, and for interpretability researchers interested in representation-similarity metrics (cosine similarity, Euclidean distance, CCA/CKA-style comparisons) applied to cross-modal transformers. Readers need comfort with transformer layer representations and fine-tuning terminology, but no specialized mathematics beyond cosine similarity and Euclidean distance.

Authors’ abstract

End-to-end Large Speech Language Models (LSLMs) have demonstrated impressive conversational generation abilities, yet consistently fall short of traditional pipeline systems on semantic understanding benchmarks. In this work, we reveal through systematic experimentation that although LSLMs lose some text input performance after speech-text alignment training, the performance gap between speech and text inputs is more pronounced, which we refer to as the modality gap. To understand this gap, we analyze both coarse- and fine-grained text and speech representations. At the coarse-grained level, representations of speech and text in deeper layers are found to be increasingly aligned in direction (cosine similarity), while concurrently diverging in magnitude (Euclidean distance). We further find that representation similarity is strongly correlated with the modality gap. At the fine-grained level, a spontaneous token-level alignment pattern between text and speech representations is observed. Based on this, we introduce the Alignment Path Score to quantify token-level alignment quality, which exhibits stronger correlation with the modality gap. Building on these insights, we design targeted interventions on critical tokens through angle projection and length normalization. These strategies demonstrate the potential to improve correctness for speech inputs. Our study provides the first systematic empirical analysis of the modality gap and alignment mechanisms in LSLMs, offering both theoretical and methodological guidance for future optimization.