People Making Conversation: Evaluating How Naturally Speech Models Converse

AI summary

Overview

Research area: Evaluation of real-time conversational speech systems, specifically turn-taking behaviour (when a system should speak versus stay silent) and conversational memory in spoken dialogue models.

Technical level: Intermediate. The paper assumes familiarity with speech model families (full-duplex models versus hosted real-time APIs), voice activity detection, endpointing, and standard text-similarity metrics, but the core idea is stated in plain terms and the results are reported as simple rate and latency vectors.

Scope: The paper builds a new evaluation that places six real-time speech systems at 427 pause points and 482 turn-end points drawn from 176 unscripted conversations in the Seamless Interaction corpus, and scores what each system generates against what the human actually did.

What This Paper Is About

Existing evaluations of conversational speech models test boundary detection, adherence to a stated turn-taking policy, or aggregate dialogue statistics; none measures the decision a real system faces at every silence, namely whether to speak at all. The authors ask whether a system does what the person did at that moment: silent where the person stayed silent, replying where the person replied, with comparable timing and length, plus whether it can answer a question about what was said 30 to 120 seconds earlier. Their claim is that no human listener rates the outputs; the reference is the human's own behaviour at the same boundary in a conversation nobody staged.

Key Contributions

1. A continuation-based benchmark for turn-taking. Rather than asking whether a system emits a boundary event at the right time, the benchmark scores what a generative system does at a boundary a person actually faced in an unscripted conversation, including what it retains from the preceding minutes. Point types are pauses (silence inside a speaker's turn where the listener did not speak) and turn ends (where the floor passed and the person replied).

2. A curated point set built on rule-based labels with multiple gates. From roughly 46,000 candidate boundaries in 176 conversations, deterministic rules and a GPT-4o judge produce 427 pause points and 482 turn-end points across 157 and 171 conversations respectively; the reported run draws a seeded random 300 of each. A separate recall probe set gives 184 conversation-recall points across 46 conversations and 669 fact-recall control points.

3. A metric vector rather than a single score. The benchmark reports shares of silent, takeover, and after-resumption behaviour at pauses; takeover latency; turn-end latency to first word; response duration and word-count ratios to the human reply; token F1 and cosine similarity (all-MiniLM-L6-v2); and judged recall by history length. Metrics are never pooled across point types, and no combined score is reported.

4. Full release of artefacts. Labels, evaluation point identifiers, the harness, and every system output are released at https://github.com/People-Make-Things/people-conversation-bench under the same licence as the source corpus (CC BY-NC 4.0), with no audio or participant metadata redistributed.

Main Findings

• Hosted systems almost never stay silent at a pause. The hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) were silent on 0 to 2% of pause points because their silence-based endpointers request a response once speech ends. Full-duplex models (PersonaPlex, Moshi) stayed silent on 22% and 39% of points respectively, the only systems that regularly held.

• Latency, not a decision, determines whether hosted words land inside the pause. Hosted speech began a median 1.0 to 1.1 s after the pause began for GPT Realtime, 2.7 s for Gemini, and 3.9 s for Grok over all rollouts, against a median labelled pause of 0.97 s. Whether the words fall inside the pause (a takeover on 7 to 26% of points) or after the person resumed (72 to 86%) follows from that timing. Grok's low takeover rate is its slowness, not restraint: its words arrived after the person had resumed on 86% of points.

• Only full-duplex models interrupt within a human reaction time. Median takeover latency was 216 ms for PersonaPlex and 373 ms for Moshi; three in ten of PersonaPlex's takeovers and a quarter of Moshi's were speech already under way at the pause.

• The endpointer is the policy. Raising GPT Realtime's silence setting from 300 ms to 500 ms cut its takeover rate from 26% to 17% and delayed its speech by 80 ms. Running the same model with semantic turn detection at low eagerness produced no words on 5% of the 173 shared pauses, took the floor inside the pause on 6% (versus 17% at 500 ms and 26% at 300 ms), and spoke after the person resumed on 88%, with first words a median 6 s after the pause ended; its median takeover latency is reported as 1339 ms. The abstract describes this configuration's latency as about 8 s.

• Takeover rises with pause length for every system. GPT Realtime took 5% of pauses under 0.8 s and 69% of pauses over 2.5 s; the full-duplex models started higher, at 20 to 28% on the shortest pauses, and rose less.

• Responses run long at turn ends. Median duration ratios to the human reply were 1.4 for Grok, 3.3 to 3.6 for Gemini, Moshi, and GPT Realtime, and 7.8 for PersonaPlex; word ratios were 1.1, 2.6 to 3.0, 0.8 for Moshi, and 3.9. No system says as little as the person did.

• The full-duplex models do not stop. 90% of PersonaPlex rollouts and 92% of Moshi rollouts were still speaking at the 30 s cap, against 24% for Gemini, 2 to 3% for GPT Realtime, and 1% for Grok. Moshi produced no words on 19% of turn-end points, and median turn-end latency was 86 ms for Moshi (often speech already under way), 0.5 s for GPT Realtime, 0.9 s for Grok, 1.1 s for PersonaPlex, and 1.7 s for Gemini.

• Content similarity sits near its floor. Median token F1 was 0.10 to 0.12 for every system that answers, against a measured floor of 0.08 (a human reply from a different point in the same conversation) and a ceiling of 0.88 (a human reply against its own label transcript).

• Hosted systems read history well; full-duplex models hear it badly. Judged by Claude Opus 5.5 and given the history as text, Grok answered 46 to 70% of conversation-recall questions, GPT Realtime 43 to 49%, and Gemini 38 to 42%, with no trend across 30 to 120 s of history. Hearing the same history as audio, PersonaPlex answered 13 to 20% and Moshi 17 to 28%. The abstract reports the pooled ranges as 41 to 60% (Claude Opus 5.5 judge, hosted), 19 to 26% (strict word matcher, hosted), 17 to 22% (full-duplex, judge), and 12 to 15% (full-duplex, strict matcher).

• The fact-recall control separates memory failure from inattention. GPT Realtime answered the general-knowledge control question 95 to 98% of the time, Grok 84 to 89%, and Gemini 62 to 73%, so their conversation-recall misses are misses of memory; PersonaPlex and Moshi answered it 16 to 40% of the time, so most of their misses are failures to answer the question at all. Rescoring the same transcripts with GPT-4o moved no system by more than 9 points and preserved the ordering.

• Wordless phonation is a full-duplex phenomenon. Above-gate audio with an empty transcript accounted for 6% of PersonaPlex's pause points and 1% of Moshi's.

Methodology in Plain English

The authors start from the naturalistic subset of the Seamless Interaction corpus, 1,395 prompted casual sessions of at least seven minutes in which two participants were given a topic card and talked. They score each conversation on six "liveliness" traits computed from released transcripts and voice activity (speech-time balance, speaker alternations per minute, short-utterance rate, question rate, one minus the longest single-speaker share, overlaps per minute) and keep the 176 conversations at or above the 0.75 mean-percentile cut, totalling 35.9 hours with a mean of 12.2 minutes and a median of 8.7 minutes.

Labels come from a deterministic rule engine over the corpus's own voice activity and transcripts, with gap-fill transcription by Qwen3-ASR and alignment by Qwen3-ForcedAligner for voiced stretches of at least 350 ms with no transcript. Gaps of at most 250 ms inside one speaker's activity are closed; a listener segment of at most 1 s and two words during the other speaker's turn is a backchannel. At every offset the engine looks ahead 5 s to decide whether the event is a pause, an end of turn, or a failed take.

Evaluation points pass hard gates first: pauses must last at least 600 ms with no listener words inside, active turns must exceed 1 s and three words, turn ends must be followed by a reply starting within 3 s with at least five words, and the audio must pass a speech gate. Survivors are ranked heuristically, capped at three per type per conversation, then re-ranked by a GPT-4o judge that rejects incoherent or unresponsive windows.

Each system gets the conversation history (capped at 164 s, PersonaPlex's context window) plus a short identity prompt, then a live stream of the active turn, the recorded pause, and the speaker resuming, delivered as real-time frames so the pause is heard as real silence. Hosted models read completed turns as text plus trailing audio and rely on their server-side endpointer; full-duplex models receive two audio channels and must decide for themselves. At turn ends the trigger differs by family: hosted models get an explicit end-of-turn signal with the endpointer off, while full-duplex models must detect the end from audio and then receive silence frames until they stop or hit the 30 s cap. The person's reply is never played back. Speech intervals are detected by an energy gate calibrated against Whisper word timestamps, and a response only counts as speech if it also transcribes to words.

Why This Matters

The paper reframes turn-taking evaluation from "did the system detect the boundary" to "did the system make the same decision the person made," which makes the endpointer configuration, not the model's judgement, the visible variable in hosted systems. It also separates memory from attention using a general-knowledge control, showing that some recall failures are failures to answer at all rather than failures of long-context memory.

Real-world applications:

• Voice agents and call-centre systems that must decide when to yield the floor rather than replying at every silence.
• Accessibility and assistive listening tools, where interrupting a speaker's mid-turn pause is more damaging than a slower response.
• Meeting transcription and summarisation systems that need to distinguish a hesitation inside a turn from a genuine end of turn.
• Spoken conversational memory features, such as assistants that must recall what was said minutes earlier rather than seconds earlier.

Industry relevance: the results give providers a concrete configuration lever (300 ms versus 500 ms silence, semantic detection at low eagerness) and show that endpointer choice moves takeover rates from 26% to 17% and then to 6%, while the full-duplex family that stays silent most often is also the family that never stops talking at 90 to 92% of turn ends. The paper reports that OpenAI and xAI assign output ownership to the customer and Google disclaims it, and that none of the terms restricts research publication.

Future Directions

• Stream audio history to the hosted models. The authors state directly that the cross-family recall gap is partly a gap between channels, because hosted models read the history as text while full-duplex models hear it; streaming audio history to hosted systems is listed as future work.
• Test more turn-detection settings. Semantic turn detection was run at one setting (low eagerness) on pause points only, so the space of eagerness and endpointer configurations remains unexplored, and the paper notes its hosted pause results measure one configuration's behaviour rather than whether a model could hold under a different turn detector.
• Fix the harness timing problems and rerun. A harness audit found that scoring could stall the shared audio stream and that the drain could outlast queued playback, so reaching the 30 s cap does not establish continued speech, full-duplex durations can include later resumptions, and latencies can reflect stalls. Turn-end comparisons also depend on family-specific response triggers, and reply playback remains untested.
• Replace rule-based labels with human-annotated ones and remove the shared-provider judge. GPT-4o both selects the points and extracts recall facts while sharing a provider with two systems under test; the corrected labels from the ongoing full human re-annotation are intended to regenerate every point.

Target Audience

Researchers and engineers building or evaluating real-time speech systems, particularly those working on turn-taking, endpointing, and full-duplex dialogue models; benchmark designers interested in how to score generative systems against human behaviour rather than against staged stimuli; and product teams deploying voice agents who need to decide how aggressive their silence thresholds should be. The paper's plain-language framing of pause versus turn end and its per-system reporting also make it readable for applied practitioners who are not speech-processing specialists, though interpreting the metric vector requires some familiarity with the underlying model families.

Authors’ abstract

A conversational speech model has to decide, at every pause, whether to speak. Existing evaluations test boundary detection, policy adherence, or aggregate statistics, not this decision. We place real-time speech systems at 427 pauses and 482 turn ends from 176 unscripted conversations in Seamless Interaction, give each the conversation so far, and score what it generates against what the person did: whether and how soon it speaks at a pause; its latency, length, and wording at a turn end; and whether it can answer a question about the preceding 30 to 120 seconds. Across six systems, the hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause whichever endpointer they use; only their latency, about 1 to 4 s with silence-based endpointers and about 8 s with semantic turn detection, decides whether the words land inside the pause. The full-duplex models (PersonaPlex, Moshi) are the only systems that regularly stay silent and the only ones that interrupt within a human reaction time. Hosted systems reply longer than the person. Given the history as text, hosted systems recall 41 to 60% of conversation facts under a Claude Opus 5.5 judge and 19 to 26% under a strict word matcher; hearing it as audio, full-duplex models recall 17 to 22% and 12 to 15%. We release the labels, points, harness, and every output.