Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization

AI summary

Overview

Research area: Speech and audio processing — neural audio codecs, speech tokenization, and generative/discriminative speech modeling for large language model pipelines.

Technical level: Advanced. The paper assumes familiarity with vector quantization, self-supervised speech encoders, CTC alignment, autoregressive speech language modeling, and speech evaluation metrics.

Scope: The paper introduces DyCAST, a variable-frame-rate speech tokenizer that aligns discrete tokens with characters in the transcript and models their durations explicitly, evaluated on speech resynthesis, voice conversion, discriminative probing, and text-to-speech.

What This Paper Is About

Most neural audio codecs convert speech into tokens at a fixed frame rate, spending the same number of tokens on information-poor regions (silence, steady sounds) as on rapidly changing speech. This wastes sequence length and leaves tokens poorly aligned with the underlying text. DyCAST instead learns soft character-level alignment so that each token approximately corresponds to a character, and predicts how long each token should last, allowing the token rate to be adjusted at inference time without any text input.

Key Contributions

1. DyCAST framework. A controllable, character-aligned, variable-frame-rate speech tokenizer built on soft character-level alignment and explicit duration modeling, extending a modular compressor-quantizer-decompressor design with dedicated dynamic pooling modules.

2. Retrieval-augmented decoding (RAD). A decoder-side mechanism that refines quantized latents by nearest-neighbor search against a pool of continuous latents, improving reconstruction fidelity without increasing bitrate.

3. Scalar spherical quantization (SSQ). A generalization of binary spherical quantization that preserves a spherical constraint while allowing each of the latent dimensions to take one of K scalar levels, decoupling bitrate from latent dimensionality.

4. Empirical evaluation. Competitive speech resynthesis, voice conversion, and downstream performance using significantly fewer tokens than fixed-frame-rate baselines, with code and checkpoints to be released at https://github.com/lucadellalib/dycast.

Main Findings

• Token efficiency at low frame rates: DyCAST variants operate at roughly 14.4 Hz (DyCAST-CA), 17.5 Hz (DyCAST-BP1), 9.0 Hz (DyCAST-BP3), and 6.2 Hz (DyCAST-BP5), with bitrates of approximately 0.92, 1.12, 0.57, and 0.40 kbps respectively. The paper reports using 3–8x fewer frames than strong fixed-rate codecs such as FocalCodec and Stable Codec.

• Competitive resynthesis quality: On LibriSpeech, DyCAST-CA reaches 3.99 UTMOS, 3.32 dWER, and 97.4 Sim; DyCAST-BP1 reaches 3.92 UTMOS, 3.61 dWER, and 97.6 Sim; DyCAST-BP3 reaches 3.95 UTMOS, 4.66 dWER, and 97.2 Sim; DyCAST-BP5 reaches 3.97 UTMOS, 8.84 dWER, and 96.5 Sim. The reference (uncompressed) values on LibriSpeech are 4.09 UTMOS, 0.00 dWER, and 100.0 Sim.

• Graceful degradation across frame rates: On MLS, DyCAST-CA achieves 2.85 UTMOS / 16.89 dWER / 98.3 Sim, DyCAST-BP1 achieves 2.90 / 16.46 / 98.5, DyCAST-BP3 achieves 2.92 / 21.61 / 98.3, and DyCAST-BP5 achieves 2.94 / 34.04 / 97.7. The MLS reference is 2.84 UTMOS, 0.00 dWER, 100.0 Sim.

• Generalization beyond English: Aside from the character aligner (which supports multilingual input), the codec itself is trained exclusively on English speech, yet it maintains robust multilingual performance with only moderate dWER increases as frame rate drops.

• Noise robustness: On VoiceBank, DyCAST-CA obtains 3.27 DNSMOS / 14.70 dWER / 91.5 Sim, DyCAST-BP1 obtains 3.22 / 12.60 / 92.1, DyCAST-BP3 obtains 3.23 / 17.96 / 91.4, and DyCAST-BP5 obtains 3.22 / 26.62 / 89.7. On Libri1Mix, the corresponding values are 2.92 / 30.95 / 91.8, 2.92 / 31.37 / 92.0, 2.91 / 36.96 / 91.4, and 2.90 / 46.42 / 90.0. Naturalness and speaker similarity remain stable even under aggressive compression.

• Voice conversion: On VCTK, DyCAST-CA obtains 3.19 UTMOS, 25.94 dWER, and 92.0 Sim, comparable to FocalCodec, with graceful degradation as frame rate decreases (DyCAST-BP5: 3.21 UTMOS, 35.59 dWER, 92.2 Sim).

• Retrieval-augmented decoding helps most at low rates: Using a pool of approximately 20M 32-dimensional latents drawn from LibriSpeech train-clean-100, dev-clean, and test-clean, moderate similarity thresholds (τ between 95 and 97) consistently reduced dWER and improved speaker similarity while keeping UTMOS stable. The largest dWER reductions appeared for the lowest-rate variant: DyCAST-BP5 went from 8.84 dWER without retrieval to 7.22 at τ = 95 and 8.08 at τ = 97. A very high threshold (τ = 99) essentially disabled the mechanism and matched the no-retrieval baseline.

• Strong ASR probing: DyCAST-CA achieved the best WER among all codecs evaluated in the discriminative probing setup, outperforming both fixed-frame-rate speech codecs and prior discrete tokenizers while using significantly fewer tokens. Specific WER/ER numbers are reported in the paper's Table 4, which is not included in the provided content.

• Speaker identification and emotion recognition: DyCAST achieves error rates comparable to strong baselines on speaker identification, though fixed-rate tokenizers optimized for speaker modeling (e.g., WavTokenizer) achieve lower SI error rates. On speech emotion recognition, performance across all codecs remains relatively close.

• TTS setup: Text-to-speech is evaluated on LibriSpeech with character-level text input and speaker embeddings for conditioning, using an autoregressive architecture over speech tokens and the same UTMOS, dWER, and Sim metrics; the numerical TTS results are not included in the provided content.

Methodology in Plain English

DyCAST follows a modular compress-quantize-decompress design. A frozen self-supervised encoder (WavLM-large, using representations from its 6th transformer layer) extracts acoustic-semantic features at a fixed rate. A lightweight compressor projects them into a lower-dimensional latent space. A chunker then groups consecutive frames into variable-length chunks using two parts: a boundary predictor, which decides where chunks start and end, and a downsampler, which takes the last frame of each chunk as its pooled representation. Each chunk-level representation is then discretized by a quantizer, producing one discrete token per chunk.

Training supervision for boundaries comes from a frozen character aligner — an MMS model with argmax CTC alignment — which provides character-level durations. Rather than treating boundaries as independent binary decisions, the boundary predictor uses a discrete-time hazard model that explicitly models the time until the next boundary, which is well suited to sparse boundaries and variable-length segments.

The reverse path, or dechunker, contains a duration predictor and an upsampler. Because only tokens (not boundaries) are typically transmitted, the duration predictor must reconstruct how many frames each token spans. It uses a negative binomial duration model, chosen over geometric or Poisson alternatives because it decouples mean and variance and can capture the heavy-tailed durations of characters in speech. Training combines a negative log-likelihood term with a normalized length regularization penalty that keeps total predicted duration consistent regardless of utterance length.

At decoding, durations can be produced in a free decoding mode when the total frame count is unknown, or in a budget-constrained decoding mode when the target length is known, renormalizing predictions so the rounded durations sum exactly to the target. The expanded features pass through a decompressor back to encoder dimensionality, and a Vocos decoder reconstructs the waveform.

Because character-aligned tokenization yields a very low frame rate (the paper notes 6–18 Hz), fine acoustic detail is easily lost. Retrieval-augmented decoding addresses this by maintaining a pool of continuous latents and, for each quantized latent, replacing it with a retrieved continuous latent when the nearest-neighbor similarity exceeds a threshold τ. The pool is indexed with an IVF index via FAISS (n_list = 4096, n_probe = 16, coarse clustering trained on 500k subsampled vectors). The quantizer itself, SSQ, uses L = 32 dimensions with K = 4 scalar levels each in [−1/√L, 1/√L], giving a nominal codebook of 4^32, made tractable by decomposing each token into 32 parallel streams with vocabulary size 4.

Training proceeds in four stages on LibriTTS resampled to 16 kHz: (1) reconstruction of WavLM features with L2 loss using teacher-forced character durations plus decoder training; (2) boundary predictor training against character aligner durations; (3) fine-tuning the compressor-quantizer-decompressor on predicted boundaries, with random sampling among aligner boundaries and greedily decoded boundaries (τ_h = 0.5, min_gap in {1, 3, 5}, no max_gap); and (4) freezing everything else and training the duration model.

Three decoding modes are offered: tokens plus durations, tokens plus a single global utterance length (near-zero overhead, used for resynthesis), and tokens only (useful for TTS and speech language modeling).

Why This Matters

Impact on research. The paper challenges the default assumption that speech tokens must arrive at a fixed temporal resolution. By grounding token boundaries in learned character-level alignment rather than heuristic frame merging, it connects speech tokenization more directly to linguistic structure, and it offers an explicit, controllable duration interface — a property most codecs lack. The SSQ quantizer and the retrieval-augmented decoding mechanism are also reusable ideas for other low-bitrate or retrieval-heavy generative pipelines.

Real-world applications.

• Speech language modeling and conversational assistants: shorter token sequences at comparable quality reduce the sequence length an LLM must model, lowering compute and latency.
• Text-to-speech: character alignment and the tokens-only decoding mode let systems stretch or compress individual characters, giving fine control over pacing and prosody.
• Low-bandwidth and on-device communication: bitrates as low as approximately 0.40 kbps (DyCAST-BP5) support transmission in constrained network or storage settings.
• Noisy-condition and multilingual speech pipelines: competitive VoiceBank and Libri1Mix results, plus multilingual generalization from an English-trained codec, suit real-world deployments with heterogeneous audio.

Industry relevance. The work targets the tokenizer layer shared by speech LLMs, TTS systems, and voice conversion products. Because DyCAST offers tunable frame rate and bitrate at inference — and a retrieval pool that can be constructed offline and updated over time to adapt to privacy, memory, speed, or efficiency needs — it maps onto deployment constraints where a single fixed-rate codec is a poor fit.

Future Directions

• Refined silence handling. The current approach aggregates silence segments into the following non-silence token; the authors explicitly leave more refined treatments to future work.

• Better duration and boundary modeling. The paper flags alternative pooling strategies (such as average pooling per chunk instead of selecting the last frame) and the analysis of alternative decoding strategies (in Appendix E.1) as areas where the current simple design could be extended.

• Retrieval pool design. The authors note that a speaker-specific pool would likely yield stronger results than the deliberately heterogeneous pool used here, leaving open the question of how to construct pools that maximize fidelity under real privacy and memory constraints.

• Text-to-speech and generative evaluation. The paper focuses generative evaluation on TTS in limited-data regimes; extending the assessment to broader generative tasks and to the tokens-only decoding mode is a natural next step.

Target Audience

Researchers and engineers working on neural audio codecs, speech tokenization, and multimodal or speech-based large language models. It is most useful to readers already comfortable with vector quantization, self-supervised speech representations, and speech evaluation metrics. Practitioners building TTS, voice conversion, or low-bandwidth speech systems will also find the inference-time control over bitrate and timing directly applicable.

Authors’ abstract

Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs. Code and checkpoints will be released publicly at https://github.com/lucadellalib/dycast.