VoxServe: Streaming-Centric Serving System for Speech Language Models

AI summary

Overview

Research area: Machine learning systems — specifically, serving infrastructure for Speech Language Models (SpeechLMs) that generate audio in streaming settings.

Technical level: Advanced. The paper assumes familiarity with LLM serving systems (batching, KV caches, CUDA graphs), neural audio codecs, and GPU execution pipelines, though its core arguments are stated in accessible terms.

Scope: This paper presents VoxServe, a unified serving system that abstracts over diverse SpeechLM architectures and adds streaming-aware scheduling and an asynchronous inference pipeline to maximize throughput while meeting latency and playback-continuity constraints.

What This Paper Is About

SpeechLMs combine a text-LLM backbone with audio-specific modules such as detokenizers, producing multi-stage pipelines that must begin audio playback quickly and then keep generating chunks fast enough that playback never stalls. Today, each model family ships its own bespoke inference stack, so system-level optimizations like request batching, chunk-wise detokenization, and CUDA graph capture must be reimplemented for every new architecture. VoxServe's goal is a single framework that serves many different SpeechLMs while optimizing directly for the two metrics that matter in streaming speech: Time-To-First-Audio (TTFA) and streaming viability.

Key Contributions

1. A unified model execution interface that decouples SpeechLM architecture details from system-level optimizations, allowing one serving framework to cover diverse model designs.

2. A streaming-aware scheduling algorithm that separates a request's startup phase from its steady-state phase and dynamically prioritizes requests by their risk of breaking playback continuity.

3. An asynchronous inference pipeline that overlaps LLM backbone forward passes, detokenizer forward passes, and CPU-side work across device streams.

4. An implemented and evaluated system supporting seven modern SpeechLMs, with evaluation showing 10–20× higher request rates than existing implementations at comparable latency.

Main Findings

• 10–20× throughput improvement: Across three models with existing serving baselines (CosyVoice 2.0, Orpheus 3B, Step-Audio 2), VoxServe sustains 10–20× higher request rates at similar TTFA while keeping streaming viability high (10–20× is the range reported in the abstract and conclusion).

• CosyVoice 2.0: The baseline reaches 500 ms p90 TTFA at ≈0.4 req/s, whereas VoxServe maintains the same TTFA up to 4.0 req/s with 100% streaming viability.

• Orpheus 3B: p90 TTFA stays below 500 ms up to 10 req/s, but streaming viability drops past 8.0 req/s because of its high token rate (86 tokens/s). VoxServe still delivers more than 10× higher throughput for a given TTFA than the baseline.

• Step-Audio 2: Achieves the lowest request rate of the three due to its large size (9B), yet VoxServe still outperforms the baseline. The paper attributes baseline weakness here partly to infeasible detokenizer batching caused by the baseline's cache-management constraints, which VoxServe overcomes by maintaining cache state under batched inference.

• Scheduling ablation (CosyVoice, p90 TTFA): Under a fixed TTFA target, optimized scheduling reaches 3.5 req/s with comparable TTFA to only 1.5 req/s without optimization. At a fixed 2.0 req/s, TTFA is reduced by approximately 2.5×. Asynchronous pipelining adds further gains at high load — about 15% lower TTFA at 4.0 req/s.

• Multi-GPU data parallelism: With up to four H100 GPUs on CosyVoice, scaling is near-linear. Under a 500 ms TTFA constraint, DP=4 sustains approximately four times the single-GPU request rate (16 req/s versus 4 req/s).

• Disaggregated inference: On Step-Audio with the LLM backbone and detokenizer on separate GPUs (two H100s total), VoxServe maintains low TTFA at substantially higher request rates than a baseline modified for the same setup, despite added inter-device latency.

• Throughput-oriented mode (CosyVoice, 1,000 concurrent LibriTTS requests): Measured as total generated audio duration divided by execution latency, the baseline reaches approximately 10× real-time, VoxServe without scheduling optimization reaches 53×, and the optimized scheduler reaches approximately 134× real-time.

• Broader model coverage (Appendix B): Across Chatterbox TTS, CSM, GLM-4-Voice, and Zonos-v0.1, VoxServe maintains low p90/p99 TTFA and high streaming viability over a wide operating range, though absolute throughput differs by architecture.

• Input-distribution robustness (Appendix B): p90 TTFA for CosyVoice is consistently lower than the baseline across LibriTTS, the Hi-Fi Multi-Speaker English TTS dataset, and the LJ Speech dataset, with stable performance trends.

Methodology in Plain English

The researchers built a serving system organized around two processes: an interface process that accepts HTTP requests, and an execution process containing three components — a Scheduler that decides what to run, a Worker that manages GPU resources and runs inference, and a Model abstraction that each supported SpeechLM subclasses.

The Model abstraction divides inference into named stages: Preprocess (prompt formatting, tokenization, buffer allocation, optional audio encoding), LLM Forward (backbone generation), Sampling (temperature/top-k/top-p with optional repetition penalty), and Postprocess (running the audio detokenizer). A depth-wise model that samples multiple codebooks is handled as an optional extra stage. Because these stages share a common interface, the Scheduler can batch and reorder them generically for any model.

To support streaming, detokenization runs chunk-wise — a fixed number of tokens at a time — with per-request cache state persisted for stateful detokenizers such as Mimi and CosyVoice's. Detokenizers were implemented to support batch inference and avoid dynamic tensor shapes so they remain compatible with CUDA graph capture. Stable input contracts (input_tokens, input_features, input_masks) and fixed execution shapes raise the CUDA-graph hit rate under dynamic batching, with FlashInfer used as the attention backend; control-flow-heavy stages stay off the CUDA graphs to preserve sampling flexibility.

Scheduling treats requests in two phases. Newly admitted requests in the startup phase are prioritized until their first audio chunk is produced, subject to a bounded concurrency limit that prevents starvation of steady-state streams. Once in steady state, each request gets a soft deadline derived from its chunk duration and accumulated timestamp lag, and the scheduler prioritizes requests within 1 second of that deadline — exploiting the fact that streaming viability is binary, so slack can be spent on more urgent requests.

The asynchronous pipeline runs backbone and detokenizer forward passes as separate GPU tasks with explicit dependencies on per-request state, allowing GPU inference to overlap with CPU-side processing.

Evaluation used a single NVIDIA H100 GPU, with requests drawn from LibriTTS for TTS and VoiceBench (AlpacaEval subset) for STS, issued over a 60-second run at Poisson-distributed intervals. Baselines were each model's official serving implementation (for example, TensorRT-LLM plus Triton for CosyVoice, vLLM plus a custom PyTorch SNAC detokenizer for Orpheus, and a customized vLLM plus a custom PyTorch detokenizer for Step-Audio).

Why This Matters

Impact on research. The paper argues that SpeechLM serving has lagged text-LLM serving because of architectural heterogeneity, and that no prior system addresses high-throughput real-time streaming generation for SpeechLMs. VoxServe's contribution is as much conceptual as empirical: a model execution abstraction under which batching, chunk-wise detokenization, cache management, and CUDA graphs apply across multiple SpeechLM families at once. The authors state this unification is, to their knowledge, the first of its kind. That gives systems researchers a shared platform for cross-architecture optimization rather than one-off per-model work.

Real-world applications:

• Virtual assistants and other interactive speech interfaces, where TTFA and uninterrupted playback are the user-visible quality signals.
• Content generation such as audiobooks and podcasts, where the throughput-oriented mode applies and streaming metrics are irrelevant.
• Synthetic data generation for model training, which the paper lists among throughput-oriented scenarios.
• Language access services, which the paper names alongside virtual assistants and content generation as scaled deployments of SpeechLMs.

Industry relevance. The paper describes current practice as fragmented: new model releases ship bespoke inference libraries that are rarely optimized for many concurrent streaming requests and support only one architecture, and a common workaround is gluing an LLM serving system to a custom audio engine — which forgoes coordination between backbone and detokenizer and fails when the backbone is unsupported out of the box (for example, when multi-codebook prediction is needed). VoxServe targets exactly this pain: lower operational cost per device, reduced engineering cost when adopting a new architecture, and support for multi-tenant streaming. The code is released at https://github.com/vox-serve/vox-serve.

Future Directions

• Broadening baseline coverage. Only three of the seven supported models (CosyVoice 2.0, Orpheus 3B, Step-Audio 2) have official serving implementations to compare against; the rest are evaluated without baselines in Appendix B. Establishing fair baselines for the remaining architectures would strengthen the generality claim.
• Scaling beyond the tested configurations. Multi-GPU evaluation covers data parallelism up to four H100 GPUs and disaggregated inference on two GPUs. Behavior at larger cluster scales, and the relative merits of data parallelism versus disaggregation for different models, are left open.
• Tuning the scheduling policy. The startup phase uses a bounded concurrency limit to avoid starving steady-state streams, and steady-state prioritization keys off a 1-second deadline margin. How sensitive performance is to these choices, and whether they should adapt per model or per workload, is not reported.
• Combining with model-level efficiency techniques. The related work cites context compression, speculative decoding, and low-rank approximation as other routes to speech inference efficiency; integrating these with VoxServe's system-level abstraction is a natural next step that the paper does not explore.

Target Audience

Systems and ML infrastructure researchers and engineers building or deploying speech generation services — particularly those who work on LLM serving, GPU scheduling, or real-time audio pipelines. It is also relevant to model developers who want efficient serving without reimplementing batching and caching for each new SpeechLM, and to practitioners choosing between bespoke inference stacks for TTS and STS models. Readers without background in LLM serving internals or neural audio codecs will find the streaming-metrics discussion (TTFA and streaming viability) accessible but the optimization sections demanding.

Authors’ abstract

Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.