Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Tracking

AI summary

Overview

Research area: Natural Language Processing / spoken language understanding, specifically end-to-end spoken dialogue state tracking (DST) with large language models.

Technical level: Advanced — the paper assumes familiarity with speech encoders, LoRA adapters, connector modules, cross-entropy multitask training and standard DST evaluation metrics.

Scope: The paper proposes and evaluates a joint speech-and-text training scheme that lets an LLM-based end-to-end spoken DST model generalize to new dialogue domains using only written DST data from those domains.

What This Paper Is About

End-to-end spoken DST systems can read dialogue states directly from speech, but they need annotated spoken in-domain data, which is expensive to collect for every target domain. The authors show that by adding a text encoder during training and sharing parameters (connector and LoRA layers) between the speech and text pipelines, a model trained on speech from one domain plus text DST data from another domain can perform well on the text-only domain without ever seeing spoken training data from it.

Key Contributions

1. A joint speech-and-text training recipe for LLM-based end-to-end DST. The baseline architecture (speech encoder + connector + LLM) is augmented with a text encoder whose output passes through the same connector and LLM, so spoken DST data and unpaired textual DST data can be optimized together. The text encoder is used only for training and can be discarded at inference, adding no inference cost.

2. Demonstration of cross-domain generalization without target-domain speech. Experiments on SpokenWOZ and Speech-aware MultiWOZ (MW) show that training on source-domain speech plus target-domain text markedly improves target-domain joint goal accuracy (JGA) relative to a no-text baseline, across two LM families (Gemma-3-1B-it and OLMo-1B).

3. A comparison against the alternative of Text-to-Speech augmentation. The authors compare joint text training with generating synthetic spoken user turns via Qwen3-TTS-12Hz-0.6B-CustomVoice (speaker "Ryan"), and show the text approach reaches comparable results without needing high-quality TTS.

4. Analysis of when joint training helps. The paper varies the relevance of the unpaired text (target-domain text, target text mixed with DialogStudio, DialogStudio only), tests larger LMs (Gemma-3-4B-it, Gemma-3-12B-it), and breaks down improvements into ASR transcription quality, entity-level WER and slot-key F1.

The authors released code and data at https://github.com/kackav/dialogue_state_tracking.

Main Findings

• Target-domain text closes most of the cross-domain gap. Training SpokenWOZ speech with unpaired MultiWOZ text (A2) reaches 36.3% / 19.0% JGA on the SpokenWOZ and MultiWOZ validation sets versus 36.1% / 15.1% with no text (A1), reducing the gap to the paired MultiWOZ model by 28.9%. The converse (MultiWOZ speech + SpokenWOZ text, B2: 30.6% / 24.7%) cuts the gap by 64.7% relative to B1 (20.5% / 28.6%).

• Mixing target text into a larger corpus still helps. Adding DialogStudio text alongside target text (A3: 37.9% / 18.4%; B3: 28.8% / 23.6%) slightly degrades target performance versus target text alone but remains considerably better than the no-text baseline.

• Irrelevant unpaired text gives only uneven gains. DialogStudio-only training (A4: 37.1% / 17.1%; B4: 20.5% / 28.1%) moderately improves the MultiWOZ validation set but not SpokenWOZ, which the authors attribute to DialogStudio containing MultiWOZ training data but not SpokenWOZ.

• Oracle toplines. Using target train+test text as unpaired data yields C1 = 38.7% / 23.1% (a 21.6% relative improvement on MultiWOZ) and C2 = 32.6% / 25.7% (a 6.5% improvement over B2 on SpokenWOZ). Training on paired data from both domains without extra text (C3) gives 36.2% / 16.9%, while adding both validation sets as unpaired text (C4) jumps to 55.3% / 42.4%.

• Part of the gain comes from the text encoder itself, not just exposing the LLM to target dialogue states. Removing the text encoder (D1: 36.1% / 17.4%; D3: 28.6% / 24.9%) performs worse on the target domain than with it (A2, B2) but still beats the no-text baseline.

• Text training is competitive with TTS augmentation. Synthetic speech recovers 49% of the JGA gap for SpokenWOZ (D4: 33.3% / 23.6%) and 27% for MultiWOZ speech (D2: 33.3% / 21.6%), at some cost on the original speech domain. Tuning the text-loss weight to 8 gives 32.4% JGA on SpokenWOZ and 21.8% on MultiWOZ, similar to the TTS results.

• Gains hold on test sets and across LLM sizes. With OLMo-1B, joint training with MultiWOZ text (E6: 34.9% SpokenWOZ / 18.2% MultiWOZ) closes 46% of the MultiWOZ gap between the MultiWOZ-speech no-text baseline (F5: 18.7% / 23.2%) and SpokenWOZ speech (E5: 34.1% / 13.9%); the converse (F6: 29.3% / 21.9%) recovers 79% of the SpokenWOZ gap. The authors' no-text OLMo-1B baselines (E5, F5) outperform those reported in the earlier work (E0: 32.1% SpokenWOZ; F0: 18.2% MultiWOZ), which they credit to more ASR pretraining data.

• Larger LMs benefit more from joint text training. Gemma-3-12B-it with MultiWOZ speech + SpokenWOZ text (F4: 42.2% SpokenWOZ / 31.6% MultiWOZ) is nearly on par with training on SpokenWOZ speech (E3: 42.6% / 16.6%) on the SpokenWOZ test set. The MultiWOZ degradation caused by adding SpokenWOZ text at 1B (B1 27.1% → B2 23.5%) shrinks at 12B (F3 32.4% → F4 31.6%) and disappears at 4B (F1 27.3% → F2 30.8%).

• Comparison with a prompted audio LLM. Prompting Gemini-2.5-flash to transcribe and predict dialogue states gives 26.5% SpokenWOZ / 20.8% MultiWOZ (G1), below the in-domain Gemma-3-12B-it no-text baseline (E3: 42.6% / 16.6%; F3: 20.7% / 32.4%) but above the out-of-domain validation results; the authors note this comparison is limited by Gemini's opaque training data and prompt sensitivity, though their joint-text models also have notably lower inference latency.

• Slot keys and slot values behave differently. Adding MultiWOZ text to SpokenWOZ speech training (A2) improves transcription WER by 35% relative to A1 (14.3% vs 15.5% on SpokenWOZ; 10.3% vs 15.7% on MultiWOZ) but not entity-level WER or slot-key F1. Conversely, adding SpokenWOZ text to MultiWOZ speech (B2 vs B1) improves slot-key F1 (81.4% vs 73.3%, and 80.0% vs 79.2%) and entity-level WER without a significant overall ASR improvement.

Methodology in Plain English

The starting point is an existing end-to-end spoken DST system: a WavLM speech encoder feeds a 4-layer Transformer connector (hidden size 1024, 4 attention heads, feedforward dimension 4096, sinusoidal positional encoding with maximum length 512, preceded by convolutional layers with strides 3 and 2 that bring the speech rate

Authors’ abstract

End-to-end spoken dialogue state tracking (DST) is made difficult by the tandem of having to handle speech input and data scarcity. Combining speech foundation encoders and large language models has been proposed in recent work as to alleviate some of this difficulty. Although this approach has been shown to result in strong spoken DST models, achieving state-of-the-art performance in realistic multi-turn DST, it struggles to generalize across domains and requires annotated spoken DST training data for each domain of interest. However, collecting such data for every target domain is both costly and difficult. Noting that textual DST data is more easily obtained for various domains, in this work, we propose jointly training on available spoken DST data and written textual data from other domains as a way to achieve cross-domain generalization. We conduct experiments which show the efficacy of our proposed method for getting good cross-domain DST performance without relying on spoken training data from the target domains.