Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models

AI summary

Overview

• Research area: Efficient inference for full-duplex speech language models (speech/audio, LLM systems, speculative decoding).
• Technical level: Intermediate — the core idea is intuitive, but the paper assumes familiarity with autoregressive decoding, KV caches, and speculative decoding.
• Scope (one sentence): The paper shows that chunk-interleaved full-duplex speech models waste compute while listening, and exploits that idle time to draft the next chunk's output before its audio arrives, verifying the draft when the audio shows up.

What This Paper Is About

Chunk-interleaved full-duplex speech language models process a conversation in fixed-length time chunks, and each chunk must finish before the next one's audio arrives — the chunk duration is a hard deadline. The workload is lopsided: listen chunks leave the GPU idle for 70–89% of the chunk duration, while speak and tool-call chunks need long serial generation that blows the deadline (every evaluated tool call missed it, and one model missed the deadline on 95% of speak chunks). The goal is to convert that idle waiting time into useful work so responses and tool calls start almost as soon as the user stops talking, without retraining the model or changing its outputs.

Key Contributions

1. Two empirical properties of chunk outputs. Across four models, replacing a chunk's real audio with silence leaves the output unchanged in 95.3% of chunks, and 88% (87.7%) of the cases that do change diverge at the very first output token.
2. Speculative Listening. A training-free, lossless framework for "input-time self-speculative decoding": the target model drafts the next chunk under a silence hypothesis during the idle window, adds a runner-up branch from the second-best first token, keeps all drafts in a cross-chunk hypothesis pool, and verifies the draft inside the forward pass that prefills the real audio.
3. TickTock-Bench. A Spec-Bench-inspired benchmark and replay harness for measuring decoding methods under chunk deadlines, covering eight tasks, 760 sessions, and 15,883 one-second chunks, with per-chunk-type latency, deadline-miss rate, and maximum lag as metrics.
4. Empirical validation on four models. Speedups of 2.6–2.8× on speak chunks and up to 23.41× on tool calls, with the maximum lag behind real time cut by 94.4% and the delay before a response starts cut by 73%.

Main Findings

• Idle time is large and systematic. Listen chunks idle for 70–89% of the chunk duration; tool calls take tens of serial forward passes and finish 1.8–2.6 s after their audio arrives.
• Silence is a good guess. The silence-hypothesis output matched the real-audio output token for token in 95.3% of chunks pooled over four models; per-model rates ranged from 83.3% (Realtime-Venus) to 99.9% (LWS).
• Mismatches start at the first token. 87.7% of mismatched chunks diverge at the first output token, which in most models encodes the listen-versus-speak decision.
• Large speedups on speak chunks. Speculative Listening reached 2.6–2.8× average speedup over autoregressive decoding across the four models, versus 0.96–1.23× for the training-free baselines (PLD, Token Recycling, lookahead decoding).
• Even larger speedups on tool calls. 23.41× on Streaming SLM and 3.45× on Gander for FDB v3 tool-call chunks.
• More tokens committed per forward pass. #MAT (output tokens per target forward pass) of 2.15–2.63 for Speculative Listening versus 1.01–1.23 for the baselines.
• Real-time behaviour improves sharply. Maximum lag per session fell from 2,619 ms to 0 ms on Streaming SLM and from 3,966 ms to 2,168 ms on LWS; 35% of Streaming SLM sessions lagged more than 1 s under AR, versus 0% with Speculative Listening (the paper reports at most 1% of sessions lagging for LWS).
• User-perceived latency drops. Response onset fell from 417–1,089 ms under AR to 74–241 ms, a 73% average reduction close to the roughly 200 ms human turn-taking gap; tool-call latency fell by 72–96% (e.g., 2,623 ms to 112 ms on Streaming SLM).
• Post-audio work becomes nearly flat in output length. 1.0–1.4 post-audio forward passes for outputs of 2 to 128 tokens, versus AR growing from 2.0 passes for one token to 73.6 passes for 65–128 tokens.
• The hypothesis pool is what unlocks tool calls. In ablation, adding the pool raised tool-call speedup from 1.42× to 22.04× on Streaming SLM and from 1.21× to 2.04× on Gander; adding the runner-up branch then raised Gander to 3.45×.
• Sampling degrades but does not break it. Speedups of 2.1× (speak) and 13.3× (tool) at temperature 0.3, falling to 1.4× and 5.4× at temperature 1.0, using speculative sampling to preserve the target distribution.
• Quality is preserved. Under float32 the method reproduces AR output token for token on nearly all sessions; under bfloat16, AlpacaEval judge scores (1–5, judged by DeepSeek-V4.1-Flash) differ by less than 0.1 point on all four models.
• Gains persist under GPU contention. With 4 concurrent sessions on one A100, speedups were 2.0× (speak) and 10.5× (tool), and the tool-call deadline-miss rate fell from 100% to 2.9%.

Methodology in Plain English

The researchers first profiled four chunk-interleaved models chunk by chunk and classified each chunk as listen, speak, or tool-call, measuring how long each took after its audio arrived. They noticed that listening chunks leave the GPU mostly idle, waiting for the next audio.

They then ran an "oracle probe" with no time limit: for every chunk, they decoded the output twice — once with the real audio and once with the audio replaced by silence — and checked how often the two matched. That gave the two key statistics (95.3% match, and most changes at the first token).

The method itself reuses standard speculative decoding, but moves the drafting earlier in time. After chunk t−1 finishes, the model greedily decodes a guess for chunk t using silence instead of audio, spending only the idle window. If it finishes early, it starts a second guess from the second-most-likely first token. All guesses from the whole session are stored in a pool indexed by short token sequences, so a partially drafted tool call from earlier chunks can be found and reused later. When the real audio arrives, the KV cache is rolled back and a single forward pass over (audio + draft) verifies the entire guess; accepted tokens are committed and the first mismatch is corrected, so every forward pass commits at least one token and the output is identical to ordinary decoding.

They built TickTock-Bench to measure this fairly: a replay harness feeds each session chunk by chunk through each model's own runtime on a single NVIDIA A100 GPU, labels chunks by type, and records latency, deadline misses, and how far the conversation falls behind the incoming audio. Baselines were three training-free speculative methods (prompt lookup decoding, Token Recycling, lookahead decoding) that all draft only after the audio arrives.

Why This Matters

The paper reframes latency in full-duplex speech models: the bottleneck is not just per-token decoding speed but when computation happens relative to the audio stream. It shows that idle listening time is a usable resource, and that a model's own predictions under a plausible-but-wrong input can be verified cheaply later — without retraining, extra draft models, or output changes.

Real-world applications:

• Voice agents and assistants where sub-200 ms turn-taking is the difference between natural and sluggish conversation.
• Agentic voice interfaces that call tools (booking, search, APIs), where a multi-second tool call currently stalls the whole conversation.
• Real-time translation, tutoring, and call-centre agents that must stay synchronized with a live speaker rather than falling behind.
• On-device or single-GPU low-concurrency deployments, which the authors name as the primary target setting.

Industry relevance: the approach is training-free and lossless, so it can be dropped into existing chunk-interleaved runtimes (the paper evaluates four, including two public agentic models) as an inference-scheduling change rather than a new model. The released benchmark gives a common yardstick that was previously missing for this class of models.

Future Directions

• Drafting under better input hypotheses than digital silence, such as streaming acoustic features from the incoming speech, which the authors name as their main next step.
• Handling higher concurrency. Speedup shrinks as sessions share a GPU (2.9×→2.0× on speak chunks from 1 to 4 sessions), so scheduling drafts across contended hardware is open.
• Extending beyond the Thinker. Only the text backbone was measured; how drafting interacts with different speech decoders (Talkers) is untested.
• Adapting to parallel-stream architectures such as Moshi, which decode every 80 ms frame and have no chunk-level idle window, requiring a different formulation.

Target Audience

Researchers and engineers working on real-time speech language models, full-duplex dialogue systems, and LLM inference optimization. It is most useful to those building or serving chunk-interleaved models who care about latency budgets, and to anyone benchmarking speculative decoding outside text generation. Readers need some background in autoregressive decoding and speculative decoding to follow the verification mechanics, but the motivating statistics and results are accessible to a general ML audience.

Authors’ abstract

Chunk-interleaved full-duplex speech language models listen and speak in fixed time chunks, and each chunk must be processed before the next one arrives. Their workload is asymmetric across chunks: listen chunks leave compute idle while awaiting audio, and speak and tool-call chunks require serial generation, which delays responses and tool calls. Using the idle compute for the next chunk requires decoding it before its audio arrives, under an assumed input. Taking silence as the assumed input and comparing the output with the output under real audio on four models, we observe two properties. First, the next audio chunk does not change the output in 95.3% of chunks. Second, 88% of changes begin at the first output token. Building on these properties, we propose Speculative Listening, a training-free, lossless framework for input-time self-speculative decoding. During idle time, the model drafts the next chunk under silence and, for changes at the first token, a second branch from the runner-up first token. It keeps all drafts in a cross-chunk hypothesis pool and verifies them when the audio arrives. On our eight-task benchmark TickTock-Bench, Speculative Listening speeds up speak chunks by 2.6-2.8x and tool-call chunks by up to 23.4x. On average, it reduces the decoding delay before a response starts by 73%, to 74-241 ms, close to the 200 ms gap between human turns, and the maximum lag behind real time by 94.4%.