StepAudio 3 Gen Technical Report

AI summary

Overview

Research area: Audio generation and spoken-language modeling (cs.SD) — specifically unified audio language models that generate speech, singing, music, sound effects, and mixtures within a single system.

Technical level: Advanced. The paper assumes familiarity with residual vector quantization (RVQ), autoregressive language modeling, neural audio codecs, GAN-based vocoder training, and reinforcement learning from preference-style rewards.

Scope in one sentence: This technical report describes StepAudio 3 Gen, a discrete autoregressive audio generator built on a 12.5 Hz, 16 × 2048 RVQ tokenizer and a pretrained text LLM, and reports its performance on TTS, voice design, and audio-language benchmarks alongside the training recipe used to protect the backbone's text abilities.

What This Paper Is About

Speech, sound-effect, and music generation have historically been built as separate systems with incompatible representations and pipelines, which makes applications that need several kinds of audio difficult to assemble. This paper's goal is a single general-purpose audio generation model — covering zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, "vibe speech," and mixtures of these — that shares one representation and one backbone.

The core difficulty the authors target is interference: adding millions of new audio tokens and a heavy multi-codebook acoustic objective to a pretrained text LLM tends to degrade the language intelligence the model inherited. StepAudio 3 Gen is therefore as much an argument about how to train such a model as it is a description of one.

Key Contributions

1. Interference-aware progressive pretraining. A four-stage recipe that introduces audio alignment, understanding, generation, and end-to-end acoustic conditioning in sequence. Frozen-backbone alignment, 50% text replay from Stage 2 onward, and temporary gradient detachment at the residual code predictor are combined to acquire audio capability while retaining the LLM's textual ability.
2. RVQ Adaptor. A zero-initialized, token-wise stack of residual blocks (pre-normalization plus SwiGLU) that maps the summed embeddings of a full multi-codebook audio frame into the pretrained LLM's input space, without adding a separate sequence encoder or replacing the shared backbone.
3. Discrete autoregressive modeling across general audio. A shared LLM-based generator for speech, singing, music, sound effects, and mixed audio using one 12.5 Hz, 16-codebook RVQ representation. A time–depth architecture separates temporal prediction (main LM head, codebook 0) from within-frame codebook prediction (a lightweight causal Transformer for codebooks 1–15), generating audio entirely in discrete code space with no diffusion or flow-matching acoustic renderer.
4. A unified instruction format. Each request is organized into three fields — ROLE (speaker identity and voice characteristics), DIRECTOR (acoustic scene and generation intent), and SCRIPT (temporal arrangement of speech and sound events). Spoken segments are prefixed with a speaker tag and optionally a (description) marker, while sound effects and music are denoted as [description] entries, making the relative ordering of events explicit.

Main Findings

• The RVQ Adaptor improves audio-language capability across every reported metric. After pretraining, the system with the adaptor scored 3.00 CER on AISHELL-1 (vs 5.25 without), 3.41 WER on LibriSpeech test-clean (vs 6.00), 51.70 accuracy on MMAU (vs 40.70), CoVoST BLEU of 30.56 En→Zh and 18.59 Zh→En (vs 12.05 and 5.99), and 58.76 accuracy on SpeechMMLU in the T2S setting (vs 12.13).
• Interference-aware progressive pretraining better preserves text ability. Against a three-stage baseline (no RVQ Adaptor; ASR training, audio understanding and generation training, then cool-down), the interference-aware recipe scored higher on all eight text benchmarks: FinEval 71.68 vs 65.86, C-Eval 71.92 vs 66.34, MMLU 69.20 vs 64.99, CMMLU 71.73 vs 66.69, MATH 45.07 vs 36.62, GSM8K 74.05 vs 67.94, BBH 67.03 vs 59.61, and HumanEval Pass@1 54.27 vs 47.56.
• TTS human-likeness judged highest in an Arena-style comparison. On a Chinese human-likeness set built from text extracted from real human speech, with 1,500 pairwise comparisons across six systems, StepAudio 3 Gen achieved the highest Elo rating of 1755.33, above Qwen-Audio-3.0-TTS-Plus, Doubao-App, MiniMax-Speech-2.8-HD, Inworld-TTS-2, and StepAudio 2.5 TTS. Head-to-head against five commercial systems, with 100 comparisons per opponent, it reached an aggregate win rate of 82.0%, ranging from 73.0% to 90.0% per opponent.
• Voice design leads on InstructTTSEval. Evaluated on the full benchmark (1,000 Chinese and 1,000 English texts, each under Acoustic-Parameter Specification, Descriptive-Style Directive, and Role-Play, totaling 6,000 generated utterances per system, judged for binary style-consistency by Gemini 3.1 Pro), StepAudio 3 Gen scored an English average of 77.7% and a Chinese average of 85.2%, the best among the evaluated systems including Qwen3-TTS-12Hz-1.7B-VoiceDesign, MOSS-VoiceGenerator, Ming-omni-tts-0.5B, and VoiceSculptor. Its Chinese APS/DSD/RP scores were 88.5/91.5/75.5 and English 82.4/85.5/65.2.
• Voice design also leads in blind human preference. In an Elo-based battle evaluation with five voice-design-capable models and 490 Chinese-and-English head-to-head comparisons, StepAudio 3 Gen ranked first with an Elo score of 1668.5 and an aggregate win rate of 75.5% (per-opponent win rates begin at 65.3%; the upper end of that range is cut off in the available text).
• The tokenizer design concentrates semantics in early codebooks. Training uses quantizer dropout of 0.5 (a larger ratio was found to strengthen coarse layers but slightly degrade full-depth reconstruction) plus a distillation branch regressing frozen SSL teacher features from the quantized latent, so semantically aligned information concentrates in the earliest codebooks. The tokenizer was pretrained on roughly 700k hours of speech, music, and sound events, then refined decoder-only with the encoder, quantizer, and semantic branch frozen.
• Evaluation deliberately avoids objective TTS metrics. The authors argue CER and speaker similarity capture limited aspects of TTS: flat prosody can still yield low CER, and speaker similarity reflects identity and timbre rather than expressive delivery. They therefore rely on subjective comparison for TTS.
• Not reported in the available content: the model's parameter count or size, inference latency or throughput figures, and quantitative generation results for music, sound effects, vocals, and speech-to-vocal beyond the description of training data construction. TTS objective scores (CER, speaker similarity) are not reported for the final model.

Methodology in Plain English

Representation. StepAudio Tokenizer converts audio from all supported domains into one shared discrete space at 12.5 Hz, using 16 codebooks of 2,048 entries each, and reconstructs 24 kHz waveforms. A frozen self-supervised encoder supplies semantic features while a convolutional encoder operating on the raw waveform at 50 Hz supplies acoustic features; the two are fused along the channel axis, compressed in time to 12.5 Hz, and quantized by a single shared quantizer so each layer carries both kinds of information at once. The decoder is fully causal — a Vocos-style Transformer with rotary embeddings and 25-frame sliding-window attention feeding an inverse STFT head — so audio is generated incrementally with no look-ahead, supporting streaming.

Generation. Text and audio share one autoregressive stream. Codebook 0 is added to the main vocabulary as 2,048 contiguous tokens, so one LM head predicts both text tokens and the top-level audio code. Each of the 16 codebooks has its own input embedding table; the 16 vectors for a frame are summed and passed through the RVQ Adaptor, whose output is added only at audio positions. A four-layer causal Transformer (the RVQ Code Predictor) then takes the backbone hidden state and codebook 0 as prefixes and generates codebooks 1 through 15, feeding each decoded code back in. The frame's factorization is exact for the joint within a frame, because the backbone hidden state already encodes past frames through the summed frame embedding.

Training. The tokenizer is trained adversarially with a multi-period waveform discriminator and a multi-resolution spectral discriminator, with feature-matching, multi-scale mel-spectrogram, and quantization commitment losses. The LLM is initialized from a pretrained text LLM and trained in four stages consuming approximately 2.7T tokens: (1) modality alignment with a frozen backbone, training only the input-side audio embedding and adaptor on ASR and speech-to-text translation where only text output is supervised, with a 0.1× multiplier on the token embedding; (2) audio understanding with the full model unfrozen, mixed one-to-one with text; (3) the main generation stage, with the code predictor's conditioning input detached from the backbone so the 15-codebook loss cannot reshape the representation built earlier; (4) a long-context joint cool-down that removes detachment and lowers the acoustic weight from 1.0 to 0.1 while extending sequence length from 16,384 to 32,768. The loss is a token-level term plus a weighted sum over the 15 residual codebooks, with λ the only weight changed between stages.

Post-training. Full-parameter supervised fine-tuning on roughly 5,000 hours of audio (music, sound effects, voice design, singing, speech) at sequence length 16,384 and batch size 64. Then GRPO: each instruction is sampled into 16 candidate audio sequences; an audio understanding model captions each output, a text LLM scores instruction consistency from 0 to 100 (averaged over four independent scorings), and for speech an ASR transcript yields CER or WER. The reward multiplies the consistency score by an exponential penalty on recognition error when error is at most 0.5 and is zero otherwise, with τ = 3 — so a linguistically wrong output cannot score highly on acoustic style alone. Groups with reward standard deviation below δ = 0.02 are discarded.

Why This Matters

The paper argues a position rather than only reporting a system: audio generation can be done entirely in discrete code space, and a pretrained text LLM can absorb general audio generation without sacrificing its language ability if the training curriculum is designed around the specific interference channels. If the reported retention numbers hold, that has implications for anyone who wants one model that can both understand language and generate audio, instead of stitching a text LLM to a separate diffusion or flow-matching renderer.

Real-world applications suggested by the capabilities described:

• Conversational and assistive voice interfaces, where the same backbone handles text reasoning and speech output, and the causal tokenizer decoder supports low-latency streaming.
• Character and role-play voice production, using the ROLE/DIRECTOR/SCRIPT format to specify timbre, speaking style, emotion, accent, and paralinguistic behavior such as laughter, breathing, and pauses, with multiple characters and their relationships in a single instruction.
• Scene-level content creation such as audio drama, podcast, or game audio, where dialogue, sound effects, ambience, and background music must be generated and placed on a shared timeline.
• Music and singing content production, including speech-to-vocal conversion that preserves a target speaker's timbre.

Industry relevance: the evaluation targets commercial TTS systems directly, and the paper reports a state-of-the-art claim on both TTS human-likeness and voice design. The instruction format, the explicit separation of speech from sound effects and music on a timeline, and the streaming-capable decoder all point at deployable content and voice pipelines rather than benchmark-only research.

Future Directions

• Report the cost side of the system. Parameter count, inference latency, and throughput are not reported in the available content, which makes it hard to judge the deployment trade-offs of the streaming tokenizer and the detached-then-jointly-trained code predictor.
• Extend quantitative evaluation to non-speech domains. The evaluation section is organized around the RVQ Adaptor, text retention, TTS, and voice design. Music, sound-effect, vocal, and speech-to-vocal quality are described in terms of data construction rather than reported benchmark numbers.
• Reconcile subjective and objective measures. The authors deliberately set aside CER and speaker similarity for TTS. Whether the human-likeness advantage is compatible with competitive objective intelligibility and speaker-similarity figures — for zero-shot voice cloning in particular — remains open.
• Stress-test the interference claim at larger scale and longer context. The cool-down stage covers roughly 44 minutes of audio at 12.5 Hz, which the authors say suffices for long-form synthesis and multi-turn dialogue. Whether the same recipe holds for longer horizons, additional audio domains, and larger backbones is not addressed.

Target Audience

Researchers and engineers working on unified audio generation, audio language models, and neural speech codecs; practitioners building TTS, voice design, or mixed speech-and-sound content systems who need to know what training interventions preserve a text LLM's abilities; and reinforcement-learning practitioners interested in a concrete GRPO setup for audio, including the multiplicative instruction-consistency and recognition-error reward, the 16-sample group size, and the low-variance-group filtering heuristic borrowed from DAPO. Readers without a background in

Authors’ abstract

We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared $16 \times 2048$ residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.