StepAudio 3 Realtime Technical Report

AI summary

Overview

Research area: Audio-language foundation models for realtime spoken dialogue (speech recognition, audio understanding, full-duplex conversation, and voice agents). Listed under cs.SD.

Technical level: Advanced. The report assumes familiarity with mixture-of-experts language models, streaming/full-duplex audio architectures, speculative decoding, and multi-stage post-training.

Scope: One sentence — the paper describes StepAudio 3 Realtime, a continuous listen-converse-think-act audio-language model that combines deep acoustic perception, full-duplex turn management, parallel reasoning during speech ("Think-While-Speaking"), and asynchronous tool use, and reports benchmark results for it and for its ASR-specialized sibling, StepAudio 3 ASR Max.

What This Paper Is About

Realtime spoken interaction is hard because a system has to keep listening while it is speaking, decide whether a pause is a completed turn or just a breath, and tell a backchannel ("right") apart from an interruption — all while still reasoning carefully enough to answer hard requests and running tools that may outlast the spoken exchange. The paper's goal is to build one model that coordinates perception, reasoning, speaking, and action as a conversation unfolds, without trading deep deliberation for low latency.

Key Contributions

1. A unified conversational loop architecture. StepAudio 3 Realtime organizes four coupled functions — Deep Perception, Seamless Duplex, Think-While-Speaking, and a streaming Voice Agent — over a shared conversational context that includes acoustic evidence, dialogue history, the current turn, reasoning progress, and tool-execution status. Functions run concurrently as new user input arrives.

2. Think-While-Speaking for parallel reasoning and speech. Building on the two-process design of Mind-Paced Speaking, a Formulation Brain produces a private reasoning trace while an Articulation Brain emits short response segments conditioned on the reasoning available so far, with playback-aware scheduling releasing segments as the streaming audio progresses.

3. Adaptive Thinking plus MTP acceleration. Adaptive Thinking decides per turn whether explicit reasoning is warranted (using a blind judge that compares original and no-think trajectories against a target answer), and MTP3 with three prediction heads drafts up to three future tokens per target-model step to speed up the remaining private reasoning.

4. A quality-over-quantity data and training recipe. A three-stage pretraining run (32K sequence length, 1.2T tokens), midtraining that extends context to 128K and adds over 10,000 hours of synthetic full-duplex interaction data, and a merging step that averages four complementary teacher checkpoints with a normalized 3:1:1:1 weighting.

Main Findings

• ASR accuracy leads on standard and context-rich benchmarks. StepAudio 3 ASR Max scores 1.18 WER on LibriSpeech test-clean, 2.28 on test-other, 0.49 CER on AISHELL-1, 3.99 on WenetSpeech test-net, and 4.35 on test-meeting. On ContextASR-Bench (Contextless setting) it is best on all four subsets: 7.91, 3.43, 1.43, and 1.02. Its macro-average error rate is 5.67% on English subsets and 1.23% on Mandarin subsets, versus 6.60% and 1.69% for HY3.0 ASR Preview.

• Audio understanding is broad, with one clear weak spot. StepAudio 3 Realtime leads the reported baselines on four of eight benchmarks, with a macro average of 81.3 (Doubao 2.0 Lite 77.7, Gemini 3 Flash 77.1, Gemini 3.1 Pro 81.8). Its largest margins are MMSU (90.6 vs Gemini 3.1 Pro's 83.6, a 7.0-point gain) and MMAR (86.5 vs 81.7, a 4.8-point gain). It also leads on Step-Caption (78.2) and MTalk-Bench (91.7). It trails Gemini 3.1 Pro by 17.7 points on AudioMultiChallenge (49.3).

• Data quality beats data volume in a controlled ablation. With SFT data as the only changed factor, roughly 100K quality-controlled examples versus about two million randomly sampled examples raised MMSU from 78.78 to 89.70 and MMAR from 74.70 to 84.50, WildSpeech from 74.20 to 77.11, and the MTalk-Bench macro average from 88.83 to 90.84 — despite using roughly one twentieth as many examples.

• Top reported full-duplex score. On the Artificial Analysis subset of Full Duplex Bench v1 and v1.5, StepAudio 3 Realtime achieves 98.9 Overall, beating the strongest baseline Qwen Audio 3.0 Realtime Plus at 98.4, with GPT-realtime-2 (High) at 95.3 and Grok Voice Think Fast 2.0 High at 95.1. Category scores are 98.9 pause handling, 100.0 turn taking, 99.0 user interruption handling, and 98.0 backchannel handling.

• Dialogue and reasoning are competitive but not best-in-class. In reasoning mode, StepAudio 3 Realtime reaches 73.0 macro average on StepAudioChat (eight dimensions), above Doubao 2.0 Lite at 70.5 and DeepSeek-V4-Flash at 71.4, but below Kimi K3 at 77.1. It is second on reasoning, memory, knowledge, conversational pragmatics, and persona and role consistency.

• Adaptive Thinking helps selectively and has a calibration flaw. It invokes explicit reasoning at rates from 51.5% to 82.0% across categories. Full thinking's largest gains over forced no-think are Reasoning (11.37 points), Persona and Role Consistency (8.37 points), and Knowledge (5.94 points). Relative to Direct SFT, Adaptive Thinking improves Dialogue Pragmatics (63.59 to 65.87) but reduces Reasoning (71.89 to 66.80). Reasoning has a think rate of only 59.5% despite benefiting most from full thinking, while Faithfulness has a higher rate (79.2%) but gains only 1.72 points from full thinking.

• MTP speeds up private reasoning with diminishing returns. Under typical (Medusa-style) acceptance, MTP5 accepts 2.153 drafts per step versus 1.801 for MTP3, with wall-clock speedup of 1.72x and 2.05x respectively (MTP3 strict: 1.231 accepted/step, 1.76x; MTP5 strict: 1.353 accepted/step, 1.49x). The fourth and fifth heads have strict marginal acceptance rates of 10.7% and 5.7%. Tested MTP configurations improve Reasoning and Memory over baseline while Instruction Following declines.

• Model merging integrates complementary teachers cheaply. Four teachers trained on different data compositions are merged by direct parameter averaging with a 3:1:1:1 weighting. The merge reaches a macro average of 81.3 on audio understanding (tying the best teacher), the highest text macro average of 76.5, and a dialogue macro average of 73.0, exceeding three teachers but below the strongest dialogue teacher at 74.2.

• A stated open gap. The report explicitly identifies multi-turn constraint following and retail tool-use tasks as remaining weaknesses, and states that keeping strict verification for spoken output does not eliminate errors from incomplete private reasoning.

• Also reported in the abstract but not detailed in the provided content: a 56.0% macro task-success rate on τ-Voice, and a 90.6 MMSU score. The paper points to Section 8 and Tables 1 and 10 for the full cross-capability comparison and τ-Voice domain-level analysis; those details and Table 10 are not present in the supplied content.

Methodology in Plain English

The authors start from a shared audio-language foundation. Audio enters through an Audio Transformer (AuT) encoder taken from Qwen3-Omni, passes through an adapter into the language model's representation space, and text tokens enter the decoder through a separate path so the model can condition jointly on sound and text. The model is a mixture-of-experts architecture, and its generated audio is fed back into the model-audio stream so it can hear itself while the user is talking.

Training happens in stages. Pretraining runs modality alignment, then large-scale multimodal mixed training, then a cooldown stage weighted toward high-quality data. Midtraining then extends context to 128K and adds audio-understanding, synthetic conversational, and voice-agent data — including over 10,000 hours of synthetic full-duplex interaction and supervision for streaming ASR, voice activity detection, and predicting whether an utterance is complete. Only at supervised fine-tuning do the two released models diverge: one branch is specialized for transcription (with time-frequency masking, a frozen audio encoder, up to 32K-token packed sequences, and optional context like dialogue history or terminology), the other for spoken interaction. Long-tail terminology is handled by generating synthetic sentences around homophone-rich names and technical terms, keeping only those whose pronunciation matches the target text.

For turn-taking, audio is organized into 320 ms blocks each followed by a state or text token, and the model uses both audio streams plus dialogue history to decide whether to keep listening, start speaking, continue, or yield. For reasoning, the model uses two concurrent calls to the same audio model (Formulation and Articulation Brains), with Adaptive Thinking deciding when to reason at all and MTP speculative decoding speeding up the private thinking; strict verification is kept for the spoken response while typical acceptance is used only for private reasoning. Finally, four teachers trained on different data mixtures are combined by averaging their parameters with weights chosen against held-out evaluations.

Why This Matters

Impact on research. The paper attacks a real tension in speech AI — you normally either reason deeply (and respond slowly) or respond fast (and reason shallowly). Demonstrating that a reasoning trace can run in parallel with spoken delivery, and that a merged multi-teacher model can hold strong capabilities across audio, text, and dialogue, gives the field a concrete architectural and training template. The negative findings are equally useful: the Adaptive Thinking ablation shows that a reasonable selection policy can still misallocate reasoning to the wrong turns, and the MTP tables show that deeper draft heads give diminishing acceptance.

Real-world applications (bullets):

• Voice assistants and customer-service lines that must handle interruptions, backchannels, and pauses without awkwardly talking over users.
• Transcription and captioning products, driven by the ASR Max branch, that need accuracy on long, multi-domain, entity-rich speech such as meetings and technical discussions.
• Voice agents that kick off a tool call (order lookup, booking, account change) mid-conversation and fold the result back into an ongoing spoken exchange.
• Accessibility and hands-free interfaces where users speak over the system or in noisy, multi-speaker environments.

Industry relevance. Full-duplex behavior, realtime latency, and tool execution are the practical barriers to shipping spoken assistants. This report benchmarks against named commercial systems (Doubao 2.0 ASR/Lite, Seed 2.0 Lite, HY3.0 ASR Preview, Gemini 3 Flash and Gemini 3.1 Pro, GPT-realtime-2, Qwen Audio 3.0 Realtime Plus, Grok Voice Think Fast 2.0), which makes the trade-offs legible to product teams. The model-merging recipe is also an operational win: complementary data mixtures can be developed independently and recombined in parameter space without retraining on the union of all data.

Future Directions

1. Fix reasoning allocation. Adaptive Thinking's own numbers show Reasoning getting only a 59.5% think rate despite benefiting most from full thinking, while Faithfulness gets 79.2% for a 1.72-point gain. Better per-turn, and possibly per-token, decisions are an obvious next step; the paper notes the aggregate comparisons do not establish the optimal decision for individual turns.

2. Close the multi-turn constraint gap. AudioMultiChallenge is the standout weakness (49.3, 17.7 points behind Gemini 3.1 Pro), and the paper flags maintaining and revising constraints over natural multi-turn audio as a clear area for improvement.

3. Improve tool-use task success. The report identifies retail tool-use as a remaining gap and reports a 56.0% macro task-success rate on τ-Voice, leaving substantial headroom.

4. Make deeper drafting pay off. MTP heads four and five add accepted drafts but at 10.7% and 5.7% strict marginal acceptance. Whether more heads, different acceptance rules, or different draft structures can deliver net efficiency — the paper cautions that wall-clock ratios are configuration-specific and not a controlled comparison of draft depth — is unresolved. The paper also notes that strict verification for spoken output does not remove errors caused by incomplete private reasoning.

Target Audience

This report is most useful to speech and audio-language model researchers and engineers building realtime conversational systems: full-duplex dialogue, streaming ASR, speculative decoding, and multi-teacher model merging. Product and infrastructure teams evaluating latency-versus-reasoning trade-offs for voice assistants will find the benchmark comparisons and the honest gap analysis directly actionable. It is not an introductory paper — readers should already be comfortable with mixture-of-experts LLMs, streaming audio architectures, and post-training pipelines.

Authors’ abstract

Realtime spoken interaction demands deep reasoning, prompt responses, and fluid turn-taking. We present StepAudio 3 Realtime, an audio-language foundation model organized around a continuous listen-converse-think-act loop. Deep Perception captures rich acoustic cues to interpret user intent, while Seamless Duplex models synchronized audio streams to handle pauses, backchannels, and interruptions naturally. Crucially, we resolve the tension between deep deliberation and latency via Think-While-Speaking, executing private reasoning in parallel with spoken delivery. In reasoning mode, StepAudio 3 reaches a 73.0 macro average on StepAudioChat. With Think-While-Speaking, it achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time. Furthermore, an integrated Voice Agent handles asynchronous tool execution without disrupting the dialogue flow. StepAudio 3 Realtime achieves top-tier performance across key dimensions: an exceptional 90.6 on the MMSU benchmark, 98.9 Overall on the Artificial Analysis Full-Duplex Bench, and a 56.0% macro task-success rate on $τ$-Voice.