SteerDuplex: Steerable Duplex Speech Dialogue Models

AI summary

Overview

Research area: Full-duplex spoken dialogue models — speech-to-speech systems that listen and talk at the same time — and specifically the problem of making those models steerable (able to shift tone, persona, accent, and speaking rate on request).

Technical level: Intermediate. The paper assumes familiarity with speech language models and reinforcement learning post-training, but its central claims are about measurement and behavior, and are explained in accessible terms.

Scope: The paper proposes a taxonomy of spoken steerability, a fine-tuned Moshi-based duplex model called SteerDuplex, a new 390-prompt benchmark called SteerBench, and a two-stage reinforcement learning scheme with an accompanying analysis of reward hacking.

What This Paper Is About

Full-duplex speech models can handle turn taking, interruptions, and backchanneling, but the paper argues that steerability — reliably changing conversational behavior such as tone, persona, speaking rate, and voice style in response to user instructions — remains underexplored. The authors first build a taxonomy of text- and audio-based steerability, show that current full-duplex models score poorly on it, and then address the gap by post-training a duplex model with supervised fine-tuning plus two-stage reinforcement learning. To make the problem measurable, they also introduce a benchmark in which every prompt carries separate text and audio rubrics for content and delivery.

Key Contributions

1. SteerDuplex, a full-duplex speech model built on the Moshi backbone and organized around a taxonomy connecting instruction-based steering, acoustic delivery, and conversational interaction.
2. SteerBench, a benchmark with 390 spoken prompts and 1,067 human-authored binary rubrics (438 audio, 629 text) assessing content and reference-grounded delivery across tone, persona, style/accent, and speed/length.
3. A post-training recipe combining supervised fine-tuning (which supplies the main steering and task gains) with continuity-aware two-stage RL (which improves interruption response and pause handling while keeping steering and aggregate task scores comparable or higher).
4. A reward-hacking analysis showing that isolated rewards admit empty or unresponsive speech, and that optimizing interruption too far can weaken continuation after listener feedback.

Main Findings

• Baselines are weak at steering: Under matched items and the same judge, Moshi and PersonaPlex reach audio-steering average pass rates of only 20.55% and 16.44%, respectively.
• Supervised fine-tuning gives the largest steering gain: The supervised model reaches 65.10 ± 1.13% audio-steering APR over three runs, an improvement of 44.5 percentage points over the strongest evaluated open baseline. Sample APR ranges from 32% to 51.11%, and individual-rubric pass rates range from 63.10% to 77.03%, meaning full constraint satisfaction remains difficult.
• Task performance improves: On Audio MultiChallenge, SteerDuplex obtains 13.64 ± 0.28% APR and 37.17 ± 1.33% ARS, versus 6.64% APR and 20.56% ARS for the strongest open baseline, PersonaPlex. The task average pass rate improves by 7 points over that baseline.
• Duplex task scores rise substantially: On FDB-v2, the supervised model scores 4.17 under the slow examiner, compared with 2.59 for Moshi and 2.65 for PersonaPlex, and exceeds both in every task family. Its VoiceBench mean is 40.87 ± 0.27, versus 38.55 for Moshi and 30.51 for PersonaPlex.
• RL improves interruption timing: On source-clean FDB-v1.5, correct response after interruption rises from 72.5% to 82.5%, and continuation after a user backchannel rises from 71.4% to 80.6%. Background-speech recovery moves from 60% to 59%, and recovery after speech directed elsewhere rises from 42% to 48%.
• RL reduces premature barge-in: On synthetic pause items, barge-in falls from 26.5% to 9%, a difference of −17.5 percentage points. On the synthetic interruption task, response rate rises from 96% to 97.7% while semantic score changes from 3.94 to 3.88 and mean takeover latency increases by 40 ms.
• Aggregate quality is preserved: VoiceBench overall rises from 40.87 to 41.38, SteerBench rubric pass rises from 63.75% to 65.22%, AudioMC APR rises from 13.64% to 14.38%, and FDB-v2 safety rises from 4.65 to 4.81. FDB-v2 mean task score changes by only −0.003.
• Isolated rewards are hackable: Promptness-only optimization attains a scalar reward of 0.670 but scores zero on the shared duplex diagnostic, leaving 25 of 96 rollouts empty. Text-only, audio-rubric-only, and audio-quality-only objectives leave 47, 41, and 41 empty rollouts. The joint composite reaches a duplex score of 0.371 but still leaves 4 empty outputs.
• Continued optimization erodes continuity: In the second stage, interruption reward on natural development conversations rises from 0.450 to 0.793, while continuation after a user backchannel falls from 3.20 to 2.00 seconds and noise-robustness reward declines from 1.96 to 0.77.
• RL also yields too readily in some contexts: RL ends daily-task utterances while the examiner is speaking more often than SFT (32.3% versus 23.6%), despite longer average responses (23.0 versus 22.1 words per utterance; stage 1 averages 20.1).
• Judge sensitivity exists: Rescoring identical generations gives RL-minus-SFT turn-taking differences of −0.074 under Gemini 3.6 Flash and −0.059 under gpt-5.4-mini at low reasoning effort, in contrast with the small positive difference (+0.018) in the broader comparison.

Methodology in Plain English

The authors start from Moshi, a public full-duplex speech model that runs parallel user and assistant audio streams alongside a time-aligned text channel. They prefix the model with a natural-language system prompt, mask those prompt tokens in the loss, and fine-tune on a mixture of 504,416 audio records and 65,675 text records, sampled at 88% and 12% of the training mass. The audio records sum to 8,510.9 hours of audio, though repeated source material means this is not a count of unique recording hours. Context and responses are capped at 300 seconds.

The data mixture combines natural conversations with targeted examples of instruction following, steering, duplex interaction, safety, and reasoning. Word-level alignment comes from Qwen3-ForcedAligner-0.6B, and turn-aware loss masks are used.

On top of this checkpoint, they run two stages of reinforcement learning over sampled speech continuations drawn from interaction windows containing turns, interruptions, pauses, backchannels, noise, and speech-mirror scenarios. Rewards are normalized per component using Group reward-Decoupled Normalization Policy Optimization (GDPO), so that no single reward's raw scale dominates. The first stage adds a response-continuity term with weight 0.5 and a 4-second first-response target, discouraging short answers that satisfy a timing event without actually answering. The second stage adds a continuation-duration bonus with weight 2.0 and a 4-second target on noise and user-backchannel events, plus dedicated backchannel sampling. Both stages freeze their starting model as a KL reference, use a policy loss over text-stream actions, and give no direct policy loss to audio-codebook actions. Rewards combine timing/interactivity terms (weight 1.0), a Gemini 3.6 Flash transcript judge (weight 0.75 on turn and interruption groups), and a waveform-integrity gate that rejects silence, clipping, and invalid outputs.

The benchmark side, SteerBench, pairs each spoken prompt with content tasks and steering requests, using fixed synthetic and human-sourced reference clips so that acoustic criteria have a concrete target. Text rubrics are judged from transcripts; audio rubrics compare delivery against the reference. The authors report three aggregation levels — audio-steering APR, sample APR, and rubric pass rate — to separate delivery control from full task compliance.

Why This Matters

Impact on research: The paper reframes steerability as a measurable property distinct from ordinary instruction following, and shows that aggregate task scores can hide behavior failures such as premature yielding or empty responses. It supplies a benchmark, a training recipe, and an explicit reward-hacking characterization for duplex speech post-training.

Real-world applications:

• Voice assistants that adapt tone, persona, and speaking rate to match a user's stated preference.
• Contact-center and support agents that must hold the floor, handle barge-in gracefully, and switch register on request.
• Accessibility tools that let users request slower, clearer, or differently styled speech delivery.
• Interactive tutoring or companion agents where persona and pacing are part of the product.

Industry relevance: The work is conducted at Scale AI with collaborators at the University of Maryland, and every reported component is built on a public backbone (Moshi) with released benchmark material, making the recipe reproducible and directly applicable to commercial duplex voice systems.

Future Directions

• Extending the benchmark: SteerBench currently covers controlled English requests with fixed reference clips; unrestricted personalization across languages, dialects, and recording conditions is left open.
• Resolving the yield-versus-continue conflict: The authors explicitly note that premature yielding calls for evaluating conversational events alongside aggregate task success, and their second-stage results show the conflict is not eliminated.
• Isolating the sources of gain: The paper states it has not separated the effects of the backbone, training data, and reward design, and that differences in interaction pools and training budgets prevent single-variable attribution.
• Improving evaluation reliability: Reference-audio judging can favor the particular realization of a style in the target clip, and no second-judge study exists for SteerBench or AudioMC.

Target Audience

Researchers and engineers working on spoken dialogue systems, full-duplex speech models, and speech post-training will get the most from this paper, particularly those interested in instruction following, controllable generation, and reinforcement learning for conversational timing. Evaluation researchers will also benefit from the rubric design and the reward-hacking analysis, which apply beyond this specific model.

Authors’ abstract

Full-duplex spoken dialogue models support low-latency turn taking, interruption handling, and backchanneling, yet a key capability remains underexplored: steerability, the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. We introduce a taxonomy of text- and audio-based steerability that identifies substantial gaps in current full-duplex models. To address this gap, we introduce SteerDuplex, a Moshi-based full-duplex speech model fine-tuned on natural conversations and synthetic dialogues targeting instruction following, vocal delivery, reasoning, and duplex interaction. We further apply two-stage reinforcement learning (RL) with hybrid rewards, combining verifiable interaction checks and judge-based semantic feedback to improve timing and response continuity. To evaluate full-duplex spoken steerability, we introduce SteerBench, a benchmark with 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style/accent, and speed/length. On SteerBench, supervised training improves audio-steering average pass rate by 44.5 percentage points over the strongest evaluated open baseline. On Audio MultiChallenge, task average pass rate improves by 7 points over its strongest evaluated open baseline. RL further raises source-clean interruption response from 72.5% to 82.5% and reduces synthetic pause barge-in from 26.5% to 9%. Steering and aggregate task scores remain comparable or higher, while reward probes reveal reward hacking through incomplete responses. Our model and benchmark support systematic research on spoken steerability, with reward analysis showing why timing gains must be evaluated alongside response completeness.