CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

AI summary

Overview

Research area: Natural Language Processing / spoken dialogue models (speech-language modeling, tokenization, and inference efficiency).

Technical level: Intermediate — the core idea is explainable without deep background, but the paper assumes familiarity with autoregressive language models, speech tokenizers, and BPE-style subword compression.

Scope: A post-hoc adaptation framework (CoSTA) that lets an existing spoken dialogue model generate compressed speech tokens for faster autoregressive decoding while keeping its original speech tokenizer and decoder unchanged.

What This Paper Is About

Spoken dialogue models produce speech by generating long sequences of discrete speech tokens one at a time, and the same token space is used both for generation and for the speech decoder that turns tokens into audio. Because these sequences are long, responses are slow and consume the model's generation budget quickly. The paper asks whether the token space used for generation can be decoupled from the token space needed for synthesis — compressing the former losslessly while leaving the original tokenizer and decoder untouched.

Key Contributions

1. An empirical scaling regularity for speech-token compression. Across eight tokenizer configurations, with added BPE vocabulary scaled from 10K to 320K units in powers of two, the compression rate decreases approximately linearly with log2 of vocabulary size, with R² ≥ 0.98 (R² ≥ 0.994 across data sources and languages).

2. A formulation of generation–synthesis token-space decoupling. The model generates compact, losslessly expandable BPE speech units, which are deterministically expanded back to the original speech-token sequence before synthesis — the original tokenizer and decoder are never replaced or retrained.

3. CoSTA, a compatibility-preserving post-hoc adaptation recipe consisting of vocabulary extension with mean-based initialization, progressive pronunciation adaptation, and dialogue adaptation, requiring only about 6% additional parameters (embedding and LM-head rows) over the base model.

4. Evidence of efficiency gains without quality loss, including model-side speedups, system-level latency/throughput improvements, human evaluation, cross-architecture transfer, and complementarity with low-rate tokenization.

Main Findings

• Model-side speedup: On Step-Audio 2 mini (8B), CoSTA-ZH reaches a generated-token compression rate of 66.81% and 1.736× relative expanded speech-token generation speed; CoSTA-EN reaches 70.17% and 1.659×. CoSTA-MIX reaches 68.11% and 165.76% relative in Chinese, and 71.02% and 157.21% in English. Vanilla fine-tuning baselines stay at 100% compression with nearly unchanged speed (99.69% and 96.61% relative).

• System-level gains: Relative to the base model, CoSTA reduces full-response latency by 32.8–39.0% in Chinese and 22.4–26.0% in English across concurrency levels 1, 4, and 8, and increases throughput by 51.6–59.2% and 25.8–38.6% respectively, with at most 0.68 GiB additional peak memory.

• Example latency numbers (Chinese, concurrency 1/4/8): 10.73 / 13.63 / 15.56 s for Step-Audio 2 mini versus 7.14 / 9.16 / 9.49 s for CoSTA-ZH. Throughput rises from 0.103 / 0.316 / 0.515 req/s to 0.160 / 0.479 / 0.820 req/s.

• Quality preserved or improved: CoSTA-ZH averages 72.27 G-Eval with 1.51% ASR-CER and 3.75 UTMOS, versus 69.24 / 7.49% / 3.62 for Step-Audio 2 mini and 69.88 / 4.96% / 3.77 for Vanilla-FT-ZH. CoSTA-EN averages 69.97 G-Eval with 2.50% WER and 4.54 UTMOS, versus 63.73 / 6.49% / 4.53 for the base model.

• Human evaluation agrees: From six evaluators and 1,080 response-level evaluations (180 paired MOS ratings per language and metric), CoSTA wins more often than it loses against both the base model and Vanilla-FT, with win-or-tie rates of 86.1–96.1% for speech fluency and 83.3–87.7% for semantic quality.

• Dialogue adaptation is the critical stage: Removing it drops Chinese G-Eval from 72.27 to 24.35, while removing pronunciation adaptation shifts compression from 66.73% to 69.56% and worsens ASR-CER from 1.51% to 1.74%. Removing timbre conversion lowers G-Eval to 70.93.

• More sustained interaction under a fixed budget: Grouping queries into 10-query stress sessions (14 Chinese, 19 English), CoSTA sustains 7.21 (ZH) and 5.79 (EN) effective turns versus 3.71 and 1.84 for Vanilla-FT, with longer generated speech (150.67 s and 146.03 s).

• A practical vocabulary-size trade-off: Going from 10K to 80K added units improves compression and speed while holding quality; pushing to 160K lowers the compression rate from 67.50% to 63.62% but yields only a 1.1% speed gain while G-Eval falls from 81.36 to 78.87. The authors choose 80K.

• Cross-architecture transfer: On OpenS2S (a different Think–Talk architecture with a 12.5-Hz GLM-4-Voice tokenizer), CoSTA reduces speech-token steps by 21.17% in Chinese and 18.64% in English while improving G-Eval and text-speech alignment over matched Vanilla-FT, suggesting complementarity with low-rate tokenization.

Methodology in Plain English

The starting observation is that sequences of discrete speech tokens contain lots of repeated local patterns, so they can be compressed losslessly with byte-pair encoding (BPE) — the same trick used for text subwords. A larger BPE vocabulary yields shorter sequences, and the paper first measures this across eight tokenizer settings to confirm the trend is reliable and predictable.

CoSTA then inserts this compression into an already-trained spoken dialogue model in three steps. First, the model's output vocabulary is extended with the new merged speech units, and each new unit's embedding and output row are initialized as the average of the original tokens it expands into — a more stable start than random initialization. Second, "pronunciation adaptation" trains on compressed TTS data (300 hours each of Chinese and English from Emilia, converted to a single timbre with Seed-VC), in two stages: first only the embedding and LM-head layers at a learning rate of 1.6×10⁻⁴, then intermediate layers too at 2.0×10⁻⁵. Text tokens are left uncompressed; only speech tokens are BPE-compressed. Third, "dialogue adaptation" trains on roughly 50K compressed speech-to-speech dialogue examples per language, built following the OpenS2S pipeline with assistant speech synthesized by CosyVoice 2, updating only the LLM while the speech encoder, adaptor, and decoder stay frozen.

At inference, the model emits interleaved text and speech tokens from the extended vocabulary; a deterministic expansion step maps any compressed unit back to its original token sequence before the unchanged speech decoder produces audio. Nothing in the tokenizer–decoder interface changes, so the approach is a bolt-on rather than a retraining from scratch. Evaluation uses 12 URO-Bench datasets in aligned Chinese and English settings, with G-Eval, UTMOS, and ASR-CER/WER as quality metrics, and vLLM-based latency/throughput measurement on a single A100 80GB.

Why This Matters

Research impact. The paper reframes speech-token efficiency as a decoupling problem rather than a tokenizer-design problem. That opens a compatibility-preserving alternative to the usual route of training lower-rate tokenizers and re-aligning the whole pipeline, and the reported log-linear compression regularity gives other researchers a quantitative handle on how much sequence reduction is available at a given vocabulary budget.

Real-world applications:

• Real-time voice assistants and call-center agents, where full-response latency of roughly 7 s versus 11 s materially changes perceived responsiveness.
• On-device or edge spoken dialogue, where the paper's small memory overhead (at most 0.68 GiB additional peak memory) matters more than raw FLOPs.
• Long-form spoken content generation — audiobooks, narrated summaries, accessibility tools — where a fixed generation budget currently caps response duration.
• High-concurrency serving deployments, where throughput improvements of 51.6–59.2% (Chinese) translate directly into fewer GPUs per request.

Industry relevance. Because CoSTA is post-hoc and keeps the tokenizer and decoder intact, an organization with an already-deployed spoken dialogue model can adopt it without rebuilding its speech stack — the adaptation cost reported for CoSTA-ZH is 9:56:35 of training wall-clock time on eight A100 80GB GPUs, approximately 79.5 A100 GPU-hours, plus 1–2 days each for timbre conversion and dialogue synthesis during data preparation.

Future Directions

• Real conversational speech instead of synthetic dialogue. The authors flag that their dialogue-adaptation data is built from synthesized speech-to-speech examples, and ask how synthetic and real conversational data might complement each other.
• Alternative lossless segmentation schemes. BPE was chosen for being deterministic, lossless, and easy to integrate; comparing other reversible speech-token segmentation strategies in spoken dialogue modeling is left open.
• Finer-grained scenarios. Evaluation covers general speech-to-speech dialogue and efficiency, with emotional dialogue, personalized speaking styles, and diverse acoustic environments named as extensions.
• Pushing the compression–quality frontier further. The vocabulary sweep shows diminishing returns past 80K added units (160K gives a 1.1% speed gain but lower G-Eval), leaving open how to capture further compression without the quality drop — for instance by combining with low-rate tokenization, which the OpenS2S results suggest is complementary.

Target Audience

Researchers and engineers working on spoken dialogue models, speech-language model efficiency, or speech tokenization will get the most from this paper. It is also useful for practitioners who need to speed up an already-deployed speech model without retraining a tokenizer or decoder, and for readers interested in empirical scaling behavior of discrete speech representations. Readers without background in autoregressive decoding and subword tokenization will need to consult the cited background literature first.

Authors’ abstract

Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately $1.7\times$ expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.