StepAudio 3 Music Technical Report

AI summary

Overview

Research area: Audio and music generation (eess.AS), specifically long-form text-to-music synthesis with symbolic planning and autoregressive language modeling.

Technical level: Advanced. The paper assumes familiarity with residual vector quantization, flow-matching diffusion Transformers, Mixture-of-Experts language models, and preference optimization.

Scope: A single technical report describing the architecture, data pipeline, training curriculum, and evaluation of StepAudio 3 Music, a system that generates songs and instrumentals up to 5 minutes and 30 seconds using a 50-Hz single-codebook tokenizer, an ABC-notation planning stage, and a flow-matching DiT renderer.

What This Paper Is About

Generating complete songs—not just short excerpts—requires coordinating long-range musical decisions (melody, section structure, lyrics-to-music relationships) with fine acoustic detail (timbre, vocal texture, transients). This paper asks how to split that labor across components: a discrete tokenizer that a language model can predict stably, an autoregressive model that organizes musical structure, and a continuous renderer that supplies acoustic fidelity. It also introduces an explicit intermediate plan written in ABC notation, so harmony, rhythm, melody, and form become readable, inspectable context before audio is synthesized.

Key Contributions

1. A generation-oriented tokenizer. Controlled comparisons at a common 25-Hz rate across single-codebook VQ, Semantic RVQ, and Acoustic RVQ show that the best-reconstructing representation is not the best-generating one. This motivates a 50-Hz, 65536-entry single-codebook StepAudio Music Tokenizer trained with semantically informed self-supervised and multi-task objectives, paired with a flow-matching DiT and a StepAudio VAE that render 48-kHz audio.

2. Explicit musical planning via ABC-CoT. A Mixture-of-Experts autoregressive model first emits an arrangement plan in ABC notation—chords, tempo, meter, key, bar structure, and melody—then predicts music tokens conditioned on that plan through a two-pass factorization, making musical decisions part of the generation context.

3. A progressive multi-task training curriculum plus preference alignment. Large-scale pre-training (600B tokens), multi-task mid-training (500B tokens), high-quality annealing (80B tokens), supervised fine-tuning (approximately 20B tokens), and Direct Preference Optimization produce a system supporting song generation, instrumental generation, vocal-to-mix accompaniment generation, and cover-song synthesis.

4. Reported benchmark standing. The final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among evaluated systems, with competitive SongBench results; on the preliminary Artificial Analysis Music Arena Vocals leaderboard it obtains a Quality Elo of 1105.

Main Findings

• Reconstruction quality does not predict generation quality. At a common 25-Hz rate, both Semantic RVQ and Acoustic RVQ reconstruct substantially better than single-codebook VQ from ground-truth tokens, but this advantage does not carry over to generation from predicted tokens.

• Acoustic RVQ is hard for the language model to predict. Because its acoustic residual features are not pre-clustered, FastLM struggles with them, and internal music experts judged the resulting generations poor in both audio fidelity and musicality.

• Semantic RVQ predicts reliably but degrades in musicality. Internal experts attributed this to error accumulation across residual codebook streams, since errors are combined when embeddings are aggregated and earlier codebook errors affect later predictions.

• Single-codebook VQ trades modest fidelity for musicality. It incurs only a modest reduction in generated audio fidelity relative to Semantic RVQ while maintaining good musicality, which is why it was selected.

• 50 Hz improves musicality over 25 Hz with no noticeable fidelity change. Using the same dataset, the 50-Hz configuration was hypothesized to help because it provides twice as many training tokens for the same amount of music, doubling the prediction targets from the existing corpus.

• The renderer hits an information ceiling imposed by the tokenizer. The native 50-Hz StepAudio VAE reconstructs better than the 25-Hz variant, but scaling the DiT from 0.9B to 4B or 8B parameters did not give consistent gains: the 0.9B model gave the best MCD, MS-Mel-L1, MS-STFT-L1, SI-SNR, and UTMOS, while the 4B model gave the best SDR. The 0.9B DiT was adopted.

• Token bitrate. With 65536 entries, each token can be represented by 16 bits, giving a nominal bitrate of 800 bit/s from 50 frames/s × 16 bits/frame, excluding entropy coding and auxiliary side information.

• Chunked long-form rendering. Rendering proceeds in 30-second chunks; the first chunk starts from an all-zero two-second VAE-latent context, and each subsequent chunk is conditioned on the final two seconds of the previous chunk's generated latents.

• Training scale. The pre-training corpus comprises more than 100 million songs, over 5 million hours of audio, and approximately one trillion tokens.

• Planning adherence is untested. The paper states that adherence to individual notes, chords, and bars is distinct from the caption alignment measured in the current evaluation and remains an empirical question.

Methodology in Plain English

The researchers first worked backwards from the output. They asked whether a stronger renderer—a bigger VAE latent space or a larger DiT—would improve final audio, and found it would not consistently, which pointed to the discrete tokens as the limiting factor. They then compared three ways of turning audio into tokens at the same frame rate, keeping the encoder backbone and training objectives fixed so the bottleneck structure was the only variable. Each tokenizer was frozen and paired with its own trained renderer, and the team measured three things: how well ground-truth tokens reconstruct audio, how accurately a language model predicts the tokens, and how good the end-to-end generation sounds to internal music experts. That combination led them to a single-codebook design at 50 Hz, prompted by a separate ablation showing better musicality without a noticeable fidelity change.

The tokenizer itself is trained in three stages: a self-supervised BEST-RQ-style stage where contiguous waveform spans are replaced with Gaussian noise before Mel extraction, a multi-task stage adding lyric recognition (CTC), Mel reconstruction, and Chroma reconstruction, and a final stage that inserts the vector-quantization bottleneck between the 12th and 13th Conformer blocks. Only then is the renderer trained, on ground-truth tokens and matching VAE latents, with the VAE decoder frozen.

For the language model, the team cleaned raw audio through an automated pipeline: bandwidth analysis, audio event detection with FireRedVAD, Demucs source separation into vocals and accompaniment, MMS-LID language identification, lyrics transcription routed to FireRed ASR for Chinese and English and WhisperX for other languages, and SongFormer section labeling. The MoE model was then trained in three curriculum stages—large-scale pre-training for lyrics-to-music alignment, mid-training that introduces ABC-CoT planning and reference-conditioned tasks, and annealing on high-quality examples from the core creation tasks. Finally, supervised fine-tuning on roughly 20B tokens and DPO against expert pairwise preferences specialized the model and aligned it with human judgment.

Why This Matters

This work argues that how you tokenize audio determines what a music language model can learn, and it gives concrete evidence that optimizing a codec for reconstruction fidelity can actively harm generation. It also positions symbolic notation as a practical middle layer between a text prompt and an audio waveform, giving creators something readable to inspect before synthesis. The reported leaderboard position (Quality Elo 1105 on the preliminary Artificial Analysis Music Arena Vocals leaderboard, behind only Suno V5.5 and Mureka) places the system among leading commercial offerings.

Real-world applications implied by the supported tasks:

• Cover-song production — preserving a reference song's melody while changing its style.
• Accompaniment and arrangement for vocalists — generating an instrumentally arranged full mix from a dry vocal track.
• Text-to-song creation for independent musicians and content creators — producing full songs or instrumentals from lyrics and a prompt, up to 5 minutes and 30 seconds.
• Assistive composition editing — using the ABC-CoT plan as an intermediate artifact that can be reviewed or revised before committing to audio.

Industry relevance: the paper targets the same market as commercial music generation services, and its central design conclusion—that a predictable, compact, single-stream token target beats a higher-fidelity multi-codebook one for autoregressive generation—is directly actionable for anyone building hierarchical music systems.

Future Directions

• Verifying plan adherence. The paper explicitly leaves open how closely generated audio follows the ABC-CoT decisions (specific notes, chords, bars), noting this is distinct from the caption alignment currently measured.
• Breaking the tokenizer ceiling. Since scaling the DiT from 0.9B to 8B parameters did not consistently help, future work would need a discrete representation that preserves more source-specific information without sacrificing predictability.
• Data scarcity and scaling. The 50-Hz choice was justified partly by doubling prediction targets from an existing corpus given scarce music training data, raising the question of how much further density or corpus growth would help.
• Extending duration and task coverage. Current support tops out at 5 minutes and 30 seconds across song generation, instrumental generation, vocal-to-mix, and cover synthesis; longer forms and additional control modalities are natural extensions.

Target Audience

Researchers and engineers building generative music or audio systems, particularly those working on neural audio codecs, hierarchical language-model-plus-renderer architectures, or preference alignment for audio. It will also interest practitioners evaluating codec design trade-offs, and technically literate music-technology readers who want to understand why symbolic planning interfaces such as ABC notation are being integrated into generation pipelines. Readers without a background in vector quantization, diffusion or flow matching, and transformer architectures will find the architecture and ablation sections demanding.

Authors’ abstract

We introduce StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation. The StepAudio Music Tokenizer represents audio as a 50-Hz stream from a 65536-entry single codebook, using semantically informed self-supervised and multi-task training to preserve musical structure and reconstruction-relevant information. A flow-matching diffusion Transformer (DiT) predicts continuous StepAudio VAE latents, which our VAE decoder converts into 48-kHz audio. This discrete-continuous design is guided by comparisons of single-codebook VQ, Semantic and Acoustic RVQ, and different DiT configurations. For explicit planning, a Mixture-of-Experts autoregressive model uses ABC notation to produce an intermediate arrangement plan (ABC-CoT) before predicting music tokens, making harmony, rhythm, and melodic structure part of the generation context. A progressive training curriculum and supervised fine-tuning support song and instrumental generation, accompaniment generation from dry vocals, and cover-song synthesis for up to 5 minutes and 30 seconds. With reinforcement learning via direct preference optimization (DPO), the final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems, with competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, it obtains a Quality Elo of 1105, behind only Suno V5.5 and Mureka and ahead of Suno V5, MiniMax models, and other systems. Audio demonstrations are available at https://stepaudiollm.github.io/step-audio-3-music.