Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents

AI summary

Overview

Research area: Realtime voice agents — specifically, spoken language understanding combined with speculative (early) reasoning and tool/function calling during user speech.

Technical level: Advanced. The paper assumes familiarity with cascaded speech pipelines (ASR → LLM → TTS), voice activity detection (VAD), function calling, and the ReAct tool-use pattern.

Scope (one sentence): The paper proposes and evaluates Heartbeat Thinking, a scheduling architecture in which periodic "Thinker" invocations and asynchronous tool calls run continuously while the user is still speaking, and a separate "Responder" reuses whatever preparation has completed when speech ends.

What This Paper Is About

Voice agents that use external tools typically wait until the user stops talking before they start reasoning or calling tools, which produces an audible delay. Prior speculative systems start work earlier, but they either cut off a generation episode at a speech-chunk boundary or interrupt ongoing generation when new input arrives — either of which can truncate a tool call before its arguments are complete.

Heartbeat Thinking's goal is to let reasoning and tool execution span multiple speech updates without being restarted, so that listening time becomes reusable preparation, while keeping the final answer grounded in the complete request.

Key Contributions

1. A non-interrupting speculative scheduling policy ("heartbeat thinking"). A Thinker is invoked every fixed interval (Δ = 500 ms in the experiments) with at most K = 2 concurrent invocations; new speech and newly returned tool results enter subsequent invocations rather than interrupting ongoing generation. Completed reasoning is written to shared state before tools are dispatched asynchronously, so model computation and tool execution overlap.
2. A decoupled Thinker/Responder handoff. At the VAD-detected end of speech, new invocations stop and unfinished Thinker and speculative tool tasks (including queued calls) are cancelled. The Responder receives the final transcript, history, tool schemas, and only the snapshot of preparation complete and available at context-assembly time, then uses a ReAct process to check that preparation against the full request, reuse applicable results, and issue any remaining calls itself. Because the interface is external records rather than model-internal state, the two roles can use independently selected models.
3. A speculative tool coverage metric (STC) and a three-benchmark evaluation. STC measures the fraction of reference calls covered by completed matching speculative calls whose results enter the Responder context without the Responder repeating them (on AudioCRAG it instead measures the fraction of samples reusing completed speculative results without additional retrieval). Experiments cover SpeechFCEval (1,914 samples), FDBench v3 (100 recordings, 79 scenarios, 12 speakers), and AudioCRAG (1,312 samples), against Speech-to-Speech, Pipecat, LiveKit, and a SHANKS-style Window Thinking baseline that follows the same update schedule but discards late-arriving outputs.
4. An analysis of when speculative execution helps and when it misleads. The paper groups FDBench recordings by information timing and explicit intent revision, and separately identifies cases where correct tool calls nonetheless yield answers built on erroneous or outdated results.

Main Findings

• Coverage rises on all three benchmarks. STC improves over Window Thinking by 10.56 percentage points on SpeechFCEval (33.13% → 43.69%), 16.00 points on FDBench v3 (34.67% → 50.67%), and 42.37 points on AudioCRAG (20.05% → 62.42%), with DeepSeek-V4-Flash as Responder and Qwen3.6-35B-A3B as Thinker.
• FDBench v3 latency improves substantially. Mean final-response latency falls from 6.90 s to 4.39 s (36.4%), which the paper reports as also 45.7% below Pipecat (8.08 s), the fastest evaluated cascaded baseline. Mean Correct Tool Call Latency moves to −4.37 s relative to speech end, versus −2.74 s for Window Thinking and 2.45–3.52 s for the cascaded baselines (negative values mean the call starts before speech ends).
• FDBench v3 task quality rises. Quality goes from 72.0% to 77.0% and Pass@1 from 57.0% to 65.0%, although Pass@1 remains below Speech-to-Speech (75.0%) and LiveKit (67.0%).
• Task quality does not always improve. Accuracy falls slightly on SpeechFCEval (67.95% → 66.85%) and AudioCRAG correct-answer rate falls (57.24% → 55.49%), while AudioCRAG missing-answer rate rises from 14.33% to 16.84%. The paper's stated conclusion is that more retained preparation does not consistently yield better final task performance.
• The coverage advantage holds across Responders. With Qwen3.5-397B-A17B, SpeechFCEval STC is 41.20% versus 32.49% for Window Thinking and accuracy 69.70% versus 67.27%; with Qwen3.8-Flash-Next, STC is 55.27% versus 44.05% and accuracy 64.30% versus 57.14%. On AudioCRAG, Heartbeat STC is 66.46% and 40.70% for those two Responders versus 20.43% and 11.05%; the paper notes the highest reuse rate and the highest correct-answer rate occur with different Responders.
• Higher coverage does not imply higher accuracy across Thinkers. On SpeechFCEval with DeepSeek-V4-Flash as Responder, Gemma-4-26B-A4B-it reaches 68.58% accuracy at 42.71% STC, versus 66.85% accuracy at 43.69% STC for Qwen3.6-35B-A3B. On FDBench, GLM-4.7-Flash reaches 84.0% Quality at 43.33% STC while Gemma reaches 80.0% Quality at 52.00% STC.
• Early information helps, late information still gets partial coverage. On FDBench v3, Heartbeat initiates a call consistent with the final request before speech end in 73.3% of early-information recordings, 49.2% of late-information recordings, and 70.6% of explicit-revision cases.
• Correct calls can still be misused. In 4.0% of FDBench recordings, the final answer uses erroneous or outdated speculative results despite correct tool-call arguments; the paper's example is an item added to a cart at $99.99 before a $50 budget constraint was heard, where a corrected search cannot undo the earlier state-changing action.

Methodology in Plain English

The system splits one assistant turn into two jobs. During listening, a "Thinker" model wakes up on a timer — every 500 ms, with no more than two running at once — and looks at the latest partial transcript plus a shared notebook of what earlier wake-ups already reasoned and which tools already returned results. Each wake-up can do nothing, write a short reasoning note, or request one or more tool calls with structured arguments.

Crucially, a new wake-up never interrupts an old one. If a tool call needs several seconds to assemble its arguments, it gets that time; the newer transcript is handled by a newer wake-up instead. Tools run in the background and their results are filed back into the notebook, so a later wake-up can use a returned result to fill a missing argument and fire a follow-up call — even if the user has not said anything new.

When voice activity detection says the user has stopped speaking, the system freezes the notebook: it stops starting new wake-ups and cancels anything unfinished, including queued tool calls. A separate "Responder" model then gets the complete transcript, the dialogue history, the tool schemas, and only the preparation that had finished by that moment. It re-reads the full request, discards speculative interpretations that later speech contradicted, reuses any completed results that still match, and runs whatever calls are left itself. Because the handoff is an explicit record exchange rather than shared internal state, the Thinker and Responder can be different models.

Evaluation compares this against ordinary cascaded pipelines and against a SHANKS-style "Window Thinking" baseline that uses the same update rhythm but throws away outputs that arrive after their window. Measurements cover tool coverage, task quality, and timing relative to the aligned speech endpoint on FDBench v3, with the speech pipeline built from Qwen3-ASR-1.7B and Qwen3-TTS-12Hz-1.7B-Base, VAD settings of 200 ms minimum speech, 800 ms minimum silence and 500 ms padding, and all experiments on NVIDIA H200 GPUs.

Why This Matters

Research impact. The paper isolates a specific design flaw in prior speculative voice systems — coupling generation lifetime to input-update boundaries — and shows that decoupling them changes both coverage and latency. Its negative results (coverage up, accuracy flat or down on two of three benchmarks) reframe the open problem from "start earlier" to "start earlier and interpret the results correctly," and the STC metric separates call-level correctness from answer-level correctness in a way earlier work did not.

Real-world applications:

• Order and delivery support: identifying an order while the customer is still describing the problem, so the lookup is already done when they finish.
• Shopping with constraints: starting a product search early, then revising it when a budget or size constraint arrives late in the utterance.
• Travel and booking agents: pre-fetching availability or fare data during a spoken request that includes several fields.
• Knowledge-grounded spoken assistants: pre-retrieving passages so the answer is composed from already-returned results rather than starting retrieval at end of speech.

Industry relevance. The latency figures are the practical argument: 4.39 s mean final-response latency on FDBench v3, described as 45.7% below the fastest cascaded baseline tested. Because the Thinker and Responder are separate, independently selectable models talking through an external record interface, the design also allows a small fast model to do speculative work and a larger model to answer — a deployment pattern industrial voice-agent teams can adopt without retraining a single monolithic model.

Future Directions

• Adaptive rather than fixed scheduling. The authors use a fixed heartbeat interval and a bounded number of concurrent Thinker invocations, and state they have not explored scheduling driven by changes in the partial transcript or by tool-call completion — which could cut redundant inference when little new information has arrived.
• Incremental delivery of late results. The Responder receives a snapshot at context assembly, so tool results returned afterward are excluded. Delivering those late results incrementally is flagged as future work, particularly for tools with variable execution times.
• Broader latency characterization. Latency analysis is confined to FDBench v3; the paper calls for evaluation across different tool workloads to characterize the trade-off between speculative coverage, inference cost, and response latency.
• Making speculative results trustworthy. Given that quality dropped on SpeechFCEval and AudioCRAG and that 4.0% of FDBench recordings used outdated results despite correct calls, the paper explicitly leaves "interpreting and using speculative results" as a separate open challenge — including the problem illustrated by state-changing actions that a later correction cannot undo.

Target Audience

Researchers and engineers working on realtime voice assistants, speech-to-speech systems, and tool-augmented LLM agents — particularly those building or benchmarking low-latency cascaded or duplex pipelines. It is also useful for readers interested in speculative execution and scheduling under streaming input, and for practitioners who need to reason about the gap between "the right tool call was made" and "the right answer was given." The paper is not beginner-friendly: it presumes comfort with function calling, VAD-based endpointing, and benchmark-driven evaluation.

Authors’ abstract

Tool-using voice agents often leave users waiting after speech ends, even when actionable information arrives earlier in the request. Speculative approaches begin work during speech, but chunk-bounded or interrupt-on-update generation can truncate a tool call before its arguments are complete. This couples computation time to speech updates, limiting useful work completed in advance. We introduce Heartbeat Thinking, which handles updated input through new Thinker invocations while allowing ongoing invocations and dispatched tools to finish. Periodic invocations read the latest partial transcript and a shared record of completed reasoning and tool results. These results help resolve missing arguments and initiate dependent calls, allowing preparation to progress across speech updates. Across SpeechFCEval, FDBench v3, and AudioCRAG, Heartbeat improves aggregate speculative tool coverage or retrieval reuse over a SHANKS-style windowed baseline, with benchmark-dependent effects on final task quality. On FDBench v3, it increases tool coverage by 16 percentage points and reduces mean final-response latency by 36.4\%. The results show that heartbeat thinking paradigm preserving ongoing computation during speech can turn listening time into reusable preparation for faster and more capable voice-agent responses.