RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

AI summary

Overview

Research area: Neural speech coding and discrete speech tokenization for speech generation (cs.SD), specifically variable-frame-rate codecs that feed language-model-based text-to-speech.

Technical level: Advanced. The paper assumes familiarity with residual vector quantization, self-supervised speech features (w2v-BERT 2.0), adversarial vocoders, and streaming/incremental decoding.

Scope: RubatoCodec is a variable-frame-rate speech codec built on X-Codec-2 that enforces an exact utterance-level event budget, places event boundaries by a deterministic "Adaptive Firing" rule, and supports causal, event-by-event streaming synthesis.

What This Paper Is About

Neural speech codecs turn audio into discrete tokens that a speech generator must then produce in sequence, so lower frame rates mean shorter sequences but more information packed into each token. Existing variable-frame-rate codecs allocate tokens based on content thresholds, which means the number of tokens an utterance needs varies with the content and the dataset rather than being predictable from audio duration. The paper's goal is to keep content-adaptive boundary placement while fixing the number of events per utterance, and to make the resulting code–duration representation decodable incrementally for streaming text-to-speech.

Key Contributions

1. Budgeted, content-adaptive tokenization. Adaptive Firing combines an exact utterance-level event budget K = round(R·T) with cumulative allocation driven by two signals: multi-scale latent dissimilarity (cosine comparisons at offsets (1,0), (1,1), (2,1)) and log-mel waveform spectral flux. Boundaries require no learned boundary predictor and no rate-penalty optimization; only a bisection search for a scaling constant and a linear-time position correction.

2. An onset-anchored interval aggregator. Each event's quantizer input is its onset frame feature plus a learned MLP correction computed from the onset feature, the mean of the enclosed frame features, and a learned duration embedding (indexed up to 64). Unlike Continuous Integrate-and-Fire, the firing signal sets boundaries only and does not weight the pooled features.

3. A code–duration interface for streaming generation. Each event carries 16 quantizer indices (layer 0 is finite scalar quantization with levels [8,8,8,8]; layers 1–15 are DAC-style vector quantizers with 4,096 entries each, 12 bits per index) plus an explicit duration. A causal decoder places event vectors on a sparse 50-Hz grid with learned offset embeddings and an empty-vector filler, and synthesizes audio as events arrive, with periodic history recomputation and crossfades.

4. Controlled empirical study plus a large multilingual model. Matched adaptive-versus-fixed-rate comparisons, a placement-only comparison against uniform four-frame events, and a multilingual model trained on approximately 375,000 hours, evaluated across five datasets totaling 10,563 utterances.

Main Findings

• Adaptive beats fixed-rate at low event rates: controlled experiments show adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates, at nominal rates of 6.25, 12.5, and 25 events/s.
• Boundary placement matters: a placement-only comparison in which both models share the 50-Hz encoder and decoder grids, the interval aggregator, and the hybrid residual quantizer indicates that content-adaptive boundaries contribute to the gains; the uniform control forms four-frame events at a nominal 12.5 Hz.
• Threshold-based codecs have unpredictable rates: one released threshold-based codec ranges from 46.1 to 81.5 events/s under a single threshold across the paper's five evaluation datasets, whereas RubatoCodec's grid-average event rate stays within 25/T events/s of 50R.
• Competitive multilingual reconstruction: a model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates.
• Better matched TTS: in a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity.
• The generator learns the budget: the downstream speech generator learns the codec's utterance-level event budget without an explicit generation-time rate constraint.
• Bitrate is controllable by quantizer depth: at 12.5 events/s, depths 1, 4, 8, and 16 correspond to content bitrates of 0.15, 0.60, 1.20, and 2.40 kbit/s, with timing cost added from the empirical entropy of event durations.
• Reported values not visible: the excerpt is truncated before the results tables, so per-dataset dWER, dCER, PESQ-WB, STOI, UTMOSv2, DNSMOS, and SIM numbers are not reported here.
• Two codecs excluded: TFC is excluded because public code or pretrained weights could not be located; VARSTok's evaluated operating point falls below the bitrate range considered.

Methodology in Plain English

The starting point is an existing codec, X-Codec-2, which already fuses a learned acoustic encoder with frozen w2v-BERT 2.0 semantic features at a 50-Hz frame rate. The authors keep that encoder and the spectral synthesis decoder but replace the temporal bottleneck.

First, they compute a per-frame "firing signal" that says how much the speech is changing. It has two parts: how dissimilar the fused features are across small temporal offsets (using cosine distance), and how much the log-mel spectrum changes from frame to frame (spectral flux). Both parts are normalized by their own sequence mean so neither dominates by raw scale. The waveform part helps early in training before the encoder has learned anything useful.

Second, they force the number of events. For T frames and a retention ratio R, the budget is K = round(R·T); the default R = 0.25 gives 12.5 events per second on the 50-Hz grid. The first frame always fires. The remaining K−1 starts are placed by turning the firing signal into small bounded increments, scaling it (via bisection) until the increments sum to exactly K−1, taking cumulative crossings of thresholds at half-integer values, and then applying a monotone correction so positions are strictly increasing and exactly K nonempty events result. Empty or degenerate signals fall back to uniform increments. Allocation is deterministic and non-differentiable, so the event count is enforced directly rather than through a rate loss, and boundaries drift during training as features change.

Third, they compress each event. The event's vector is its first frame plus a learned correction from an MLP that sees the onset feature, the mean of the event's frames, and a duration embedding. That vector goes through 16 residual quantizer layers plus a training-only semantic head that predicts the frozen w2v-BERT 2.0 layer-16 features at event onsets, keeping the first layer linguistically informative.

Fourth, they decode causally. The event vector is written at its start position on a sparse grid, with a learned empty vector and offset embeddings filling the frames in between, so event duration determines how many frames it occupies. A 12-layer causal Transformer, a factor-three upsampler (borrowed in spirit from Inworld TTS), and a Vocos-style spectral head with ISTFT produce 44.1-kHz audio, 882 samples per grid frame. Because attention excludes future frames and convolutions are left-padded, audio can be emitted event by event.

Training uses multi-resolution mel loss, semantic loss, quantization losses, HiFi-GAN and UnivNet-style discriminators, and a late perceptual loss from wav2vec2-XLS-R-300M, scheduled over three phases (10%, 75%, 15% of updates). The comparison models train on an approximately 5,000-hour English subset of HiFiTTS-2 for 100,000 steps; the placement-only study trains for one million steps; the multilingual model trains for 1.5 million steps on about 375,000 hours (English 44%, Japanese 27%, Chinese 13%, other 16%).

Why This Matters

Making sequence length predictable from audio duration is a practical requirement for speech generation: training and inference cost, batching, and streaming latency all depend on knowing how many tokens an utterance will need. RubatoCodec shows this can be done without giving up content-adaptive timing, and it packages the timing as an explicit duration symbol that a generator can predict alongside the codes.

Real-world applications:

• Streaming text-to-speech and voice assistants, where audio must start playing before the full utterance is generated and each completed event can be rendered immediately.
• Audiobook and long-form narration, where predictable token counts per minute of audio simplify cost estimation and batching.
• Dubbing and localization, where a multilingual codec preserving intelligibility across English, Japanese, and Mandarin supports cross-lingual pipelines.
• Low-bitrate speech communication and archival, where quantizer depth gives a direct knob from 0.15 to 2.40 kbit/s at 12.5 events/s.

Industry relevance: the code and weights are released (GitHub and Hugging Face), and the design deliberately reuses an existing open architecture (X-Codec-2) and a frozen public feature model, which lowers the cost of adoption by teams already building LLM-based speech systems.

Future Directions

• Joint generator design: the paper notes the downstream generator learns the utterance-level event budget implicitly; how best to condition or supervise that budget, and how it behaves under distribution shift, remains open.
• Boundary quality: allocation is deterministic and non-differentiable, so boundaries can only improve indirectly through feature learning. Whether a differentiable or learned boundary signal could improve placement without sacrificing the exact count is unresolved.
• Long-utterance streaming: the decoder periodically discards cached states and rebuilds them from a recent window starting at an event onset, with crossfades at each recomputation. The limits of this strategy on very long inputs are not established in the visible text.
• Broader languages and domains: evaluation covers English, Japanese, and Mandarin; other training languages (German, French, Korean, and Indic languages in the mixture) are not reported as evaluated, leaving generalization to those languages an open question.

Target Audience

Researchers and engineers working on neural speech codecs, discrete tokenization for audio language models, and streaming TTS systems. It is most useful to readers already comfortable with residual vector quantization, self-supervised speech features, and vocoder training, and to practitioners deciding whether a variable-frame-rate codec is worth adopting over a fixed-rate 12.5-Hz tokenizer. Readers wanting a gentle introduction to speech tokenization should treat the architecture section as advanced material.

Authors’ abstract

Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information. Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Its Adaptive Firing mechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor. Discrete codes and an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates. A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.