Real-Time Voice Agents, Duplex Interaction, and Turn-Taking

A voice agent is a chain of parts that all run while the others are still running: capture, VAD, ASR, dialogue state, reasoning or policy, tools, TTS, echo cancellation, interruption, and memory. What makes the interaction feel cooperative is not the quality of any single part. Neither is what keeps its actions under control. It is the timing, and the state transitions between the parts.

Two kinds of question get asked about that chain, and they are not the same question. Time to first audio, end-to-end response latency and interruption reaction are clock questions — how fast. Turn-taking errors, overlap recovery and user repair rate are conversation questions — whether the exchange worked. Neither set, on its own, is the release decision.

Neither is enough because a fluent voice hides the distinctions the system has to keep straight internally. Inside the agent there are provisional hypotheses, confirmed user decisions, tool proposals, authorized actions and durable memory. Those are five different things. Human-like timing confers none of them. A human-sounding voice confers none of them either: no understanding, no authority, no guarantee of safe intent.

Every one of those boundaries has been measured by somebody. The interval a human turn actually allows was measured across ten languages. The rate at which a streaming recognizer takes back words it has already emitted was measured and named by the engineers who built one. How often a smart speaker wakes up to a television was counted in two countries. What it costs to leave retention at its default was settled by a court, in dollars and in months. The rest of this lesson runs on those numbers instead of on the reader's intuition about what a fluent voice must understand.

Take the sequence every voice team dreads. The user cuts in with "no, cancel that" while the agent is speaking. Playback cancellation works exactly as designed and the voice stops. But the tool call has already been committed. And the dialogue manager files the interruption as a new topic rather than as a correction of the thing it was already doing. No measurement records a fault. Playback stopped on time, and the wrong action still went through.

Neither half of that sequence has to be taken on trust. Both are documented quantities rather than anecdotes. The first half is that the agent had to start early. Conversation leaves gaps of roughly 200 ms, while producing language takes over 600 ms. Levinson and Torreira put the arithmetic in one line: “the gaps between turns are short (of the order of 200 ms), but the latencies involved in language production are much longer (over 600 ms)”. So anything that answers in time, human or machine, is already building its response out of a sentence that is still being spoken.

The second half is that what it starts from is revisable. A streaming recognizer emits partial results and then takes them back, at a rate its own authors count with two named metrics. The next two sections give both measurements in full.

Put the halves together and the commitment point stops being a design sin somebody was careless about. It is a place in the pipeline that the arithmetic guarantees will exist. Something will be executed from an unfinished sentence unless a named state decides otherwise. That is why this trace is the one the rest of the lesson keeps returning to.

Where those boundaries sit is decided by the interaction model, and there are three to choose between. Half-duplex interaction alternates listening and speaking with explicit turns, so the point of commitment is never in doubt. The price for that clarity is paid on the clock the previous section described. Every explicit turn boundary is dead time inside a 200 ms window.

Full-duplex interaction is not a promise waiting for hardware. Moshi, published in 2024 by a team at Kyutai, is one: “Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice”. That is inside the human turn-transition window. It is worth being clear about how the window was reached. Moshi removes explicit speaker turns altogether, and abandons the VAD → ASR → dialogue → TTS pipeline that the rest of this lesson names its states along. Speed came from dissolving the very stages where a commitment point could be given a name. The gain and the cost are the same architectural decision.

A transactional voice workflow makes the opposite bargain: confirmations, audit and rollback around consequential tools, paid for in latency. The three promise different things about timing and commitment. So each one is checked in a different way, and no single test settles all three.

Whichever of the three you pick, the same pipeline runs underneath it, and the middle stage is the one doing the safety work. The agent senses continuously and minimally, taking incoming audio as it arrives. It maintains provisional state, holding what it has heard as a hypothesis rather than as a decision. It plans and authorizes, separating a conversational response from an executed tool call. Then it speaks and repairs the turn, including when the user cuts in.

That middle stage is not an academic nicety, and a production system shows what it is made of. Duplex Conversation decomposes smooth turn-taking into exactly three subtasks: user state detection, backchannel selection and barge-in detection. Lin and five colleagues published it in 2022, and reported what happened when it was put into service: “We deploy the Duplex Conversation to Alibaba intelligent customer service and share lessons learned in production. Online A/B experiments show that the proposed system can significantly reduce response latency by 50%.” Note which way the number ran. Adding an explicit layer that decides whether the user has finished speaking did not cost latency. It halved it, because the system stopped waiting out silences that were never the end of a turn.

Remove that middle stage and the two ends touch. The system speaks straight from raw sensing. Every fragment it has picked up becomes something it is prepared to act on, including the ones the recognizer is about to revise. That is not a subtle bug. It is the cancelled tool call, written out as an architecture.

The way to catch that missing middle before a user does is to draw the state machine: idle, activated, listening, thinking, confirming, acting, speaking, interrupted and failed. Draw it well enough that someone from another team can point at one transition and say: that is executing from an unstable partial transcript. If no transition can be named that way, the drawing is not finished yet. Name the cancellation semantics on the same diagram — what stops playback, what stops inference, what stops queued work, and what stops an external action already in flight. Those are four different stops, and only the first one is audible.

Then walk it, and not only along the path where a clean request gets a clean answer. The adversarial dialogues worth walking are the ones the literature already measures: a pause that is not the end of a turn, a backchannel that is not a request, an interruption in mid-sentence, a partial that gets revised after the tool call was assembled, and an activation nobody in the room asked for. Keep a short record of three things while you do. What sensing continuously and minimally assumes. One counterexample that breaks the assumption. And what speaking and repairing does when it breaks.

A finished state machine tells you what to measure. It takes a portfolio rather than a headline number, because the two halves can move in opposite directions at once. That is no longer a hypothetical. Full-Duplex-Bench is a public benchmark built for exactly this: “we introduce Full-Duplex-Bench, a benchmark that systematically evaluates key interactive behaviors: pause handling, backchanneling, turn-taking, and interruption management.” It appeared in 2025. Its evaluation set is enumerated rather than gestured at: 216 Candor pause-handling samples, 119 Candor smooth-turn-taking, 55 ICC backchannel, 200 synthetic user-interruption and 137 synthetic pause-handling.

Read two of its columns together and the lesson's warning becomes a table. The end-to-end models are fastest at turn-taking: Moshi at 0.265 s and dGSLM at 0.352 s average latency, against Freeze-Omni at 0.953 s and Gemini Live (gemini-2.0-flash-live-001) at 1.301 s. The same two models have the highest takeover rates during natural speaker pauses — Moshi 0.985/0.980 and dGSLM 0.934/0.935, against Gemini Live's 0.255/0.310, lower being better. The fastest system on the clock is the one most likely to start talking while you were only drawing breath. A release decision that reads only the latency column records that as progress.

The lifecycle column is the one nobody reads until a court does. Alexa kept voice recordings by default. The complaint the Department of Justice filed on behalf of the FTC in May 2023 alleged: “Since launching its first Echo smart speaker, Amazon has set its default settings to retain users' voice recordings indefinitely.” The stipulated order entered by Judge Tana Lin on 19 July 2023 imposed a $25,000,000 civil penalty and mandated deletion, including of what the order defines as “Inactive Alexa Child Profile” data — a child profile unused for 18 months or more. A retention default became a dollar figure and a deadline measured in months. Neither of them would ever have surfaced in a latency percentile.

So four kinds of evidence have to travel together, and the portfolio owes one more thing besides. It should say when an unstable partial transcript requires the agent to abstain or fall back rather than act.