Fish Audio Drama 3 Technical Report

AI summary

Overview

Research area: Speech generation (cs.SD) — expressive text-to-speech and audio language models, combining large-scale mixture-of-experts language modelling with neural audio codecs and preference-based reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with autoregressive language models, mixture-of-experts routing, linear attention, residual vector-quantized audio codecs, speculative multi-token prediction, and reinforcement learning from human feedback.

Scope in one sentence: The report describes Fish Audio Drama 3, an open-weight 35B-parameter speech generation model (3B active per token) built to produce directed, multi-turn, multi-character, editable and streamable performances, and details its architecture, three-stage training pipeline, reward-model-only RL, and a purpose-built inference engine.

What This Paper Is About

Existing large text-to-speech models read text with near-human intelligibility, but the applications that matter — audio drama, dubbing, interactive characters — demand a performance shaped by natural-language direction, refined across takes, placed inside a scene, and correctable word-by-word. The authors' previous model, Fish Audio S2, took a first step with inline instructions and multi-speaker generation, but remained single-shot in control, all-or-nothing in generation, architecturally ill-suited to long directed sessions, and optimized during RL against proxies such as word error rate and speaker similarity that expressive delivery legitimately violates. Drama 3 redesigns the architecture, task formulation and post-training objective around directed performance, adding multi-turn revision, scene dubbing, editing by infilling and streaming text input within a single model.

Key Contributions

1. Hybrid backbone with a codebook multi-token-prediction (MTP) decoder. Drama 3 replaces S2's per-frame Fast AR module with a 13-layer decoder (four shared layers plus nine per-codebook layers), in which each residual codebook is predicted by a dedicated full-width layer that attends causally over the entire sequence, including the fine acoustic detail of the reference audio. The backbone itself is a 40-layer hybrid of Gated DeltaNet and gated full-attention layers with a sparse MoE feed-forward network routing each token to 8 of 256 experts plus a shared expert, activating about 3B of 35B parameters per token.

2. Multi-turn revision, scene dubbing, editing and streaming text input in one model. Tasks S2 did not support — revising a take from a note in the next turn, dubbing a whole multi-character scene, editing speech by infilling, and speaking text as it streams in — are all written as conversations over the same token stream as cloning and directing, with loss applied only to the model's own turns.

3. A single learned reward model, with every rule-based reward eliminated. Because expressive delivery breaks transcription and similarity proxies, RL optimizes one learned model of human preference — a smaller speech language model fine-tuned on human preference judgments, with a calibrated and bounded reward — using GRPO without a KL penalty and on very large batches.

4. A rewritten inference engine for the new generation structure. The engine runs the codebook-MTP decoder once per frame rather than once per codebook, stores routed experts in block-scaled FP8, deletes auxiliary kernel launches that CUDA graphs leave in place, tunes kernels to live serving shapes, and separates prefill from decode so arriving requests do not stall running streams.

Main Findings

• Decode rate despite seven times the parameters. At a concurrency of 1, Drama 3 decodes 271 frames per second on one NVIDIA H200 GPU, against 177 for S2 measured the same way, although Drama 3 has more than seven times as many parameters. On one NVIDIA B300 GPU it decodes 295 frames per second.

• Weights read per frame drop sharply. A frame of Drama 3 reads about 6.0 GB of weights, less than two fifths of the roughly 15.6 GB S2 reads. The breakdown: backbone without routed experts 2.9 GB for Drama 3 versus 7.3 GB for S2; eight active routed experts 1.0 GB for Drama 3 (S2 not reported in the table); codebook-MTP decoder 2.1 GB versus 8.3 GB for S2's Fast AR. Layer passes in sequence after the backbone step fall from 40 in S2 to 13 in Drama 3, and the number of layers whose KV cache grows with context falls from 36 to 23.

• FP8 experts help at scale. On the H200, the FP8-format experts raise the per-stream decode rate at a concurrency of 32 by 36% over BF16 experts.

• Revision rounds improve control accuracy. On the Step-Audio-Edit benchmark, each revision round raises the model's emotion and style accuracy.

• Lowest cloning error rate across five languages. On the MiniMax Multilingual Testset, Drama 3 has the lowest error rate of all compared systems in Chinese, Dutch, English, German and Turkish.

• Editing beats a commercial infill API on error rate. On the Ming-Freeform-Audio-Edit benchmark, Drama 3 has a lower error rate than the infill API of Cartesia Sonic-3 in English and Chinese, and in Chinese a higher speaker similarity than dedicated editing models on insertion and substitution.

• Kernel census shows launch overhead dominates. With CUDA graphs, PDL and piecewise prefill graphs enabled on a B300, the median frame took 4.0 ms and ran 1,171 kernels, with only 0.09 ms of gaps between them. Moving all the data the frame needs would take 0.76 ms at the GPU's memory bandwidth, one fifth of the frame; the other four fifths went mostly to starting and draining kernels — about 830 kernels moved almost no data yet took 2.7 ms.

• Accepted kernel-level optimisations (per-stream decode rate gains). Auxiliary decode kernels removed: 1,429 to 1,278 kernels per step, +1.7% at B300 concurrency 32. Gated DeltaNet decode fused and sampler counter folded in: minus 100 kernels per step, +4.2% at B300 concurrency 1. Gated DeltaNet prefill launches merged: 23 to 13 kernels per layer, +4.4% at B300 concurrency 128. First-frame sampling captured into CUDA graphs: 30.9 to 14.7 ms host time per round, +15.3% on H200 at concurrency 32 (measured on an earlier development model). Execution plan built from host-side lengths: 9.75 to 0.86 ms GPU idle per round, approximately +7% at B300 concurrency 128.

• A rejected fusion. A kernel fusing the router's GEMM, softmax and top-8 selection into one launch took 14.5 μs, against 7.5 μs for the kernels it would have replaced, so it was rejected. Under PDL, changes that saved 0.218 ms of kernel time per step gained only 0.14–0.16 ms in serving.

• Training data and scale. Pre-training processes several hundred billion tokens with the language coverage of S2, about 80 languages and dialects. Stage 1 and Stage 2 sequences are at most 8K tokens; Stage 3 raises the maximum sequence length to 16K tokens; SFT trains on sequences of up to 32K tokens. Samples are packed with document masking into rows of 128K tokens.

• RL configuration. Each RL step uses 1,536 prompts times G = 8 takes sampled at temperature 1 with no top-p or top-k truncation. Length is bounded by a sampling budget of about 2.5 times the expected duration of the script, with no take exceeding 4,096 frames (about 190 s). Groups whose reward standard deviation is below 0.05 are dropped, and the DAPO-style asymmetric clip is set at ε_low = 0.2 and ε_high = 0.28.

• Audio tokenizer unchanged from S2. Built on the Descript Audio Codec, it encodes 44.1 kHz

Authors’ abstract

We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.