Voice Activity Detection, Endpointing, and Segmentation

Two jobs are being confused in that gap, so separate them. Voice activity detection estimates where speech-like energy occurs. Endpointing decides when an interaction segment is complete enough to commit. Both depend on noise, overlap, language, speaking style, device behavior, and above all on the cost of waiting weighed against the cost of cutting someone off. The detector reports energy shaped like speech. It never reports a person choosing to address the device, finishing a thought, or agreeing to be recorded. The endpointer lives in that gap and does the converting: frames of acoustic activity in, a claim about the state of the interaction out.

Because they are two jobs, the two numbers usually quoted answer two questions. False alarm and missed speech by acoustic condition asks how often the detector is wrong. Endpoint latency after the true end of speech asks how long the user waits. Neither is the release decision on its own.

Deciding that speech has stopped is an old engineering problem with shipped answers, and the older ones state the trade without embarrassment. Telephone standards got there first. G.729 Annex B, a silence compression scheme approved in October 1996, stops sending frames once it concludes nobody is talking. Sixteen years later a different standards body solved the same problem and wrote the trade down as a number: “Discontinuous Transmission (DTX) reduces the bitrate during silence or background noise. When DTX is enabled, only one frame is encoded every 400 milliseconds.” — RFC 6716, the Opus codec specification, September 2012.

Both standards are buying bandwidth. Both pay for it in the same coin: the risk of clipping a quiet onset, a syllable that began while the encoder had already concluded nobody was talking. Every 400 ms of silence that Opus declines to transmit is 400 ms in which a returning voice has to be noticed again. An endpointer in a voice interface makes that identical trade. What it spends on the other side is a user who is still thinking. Two standards bodies were standing at that dial thirty years before voice assistants.

The study is short enough to tell whole, and the design is the argument. Two rounds. 134 hours of content from 12 TV shows. Popular smart speakers, in the US and in the UK, sitting where a customer would put them. No adversarial audio, no synthesised wake words, no laboratory noise generator: television, played into a room, for as long as television is normally played. The result was 0.95 misactivations per hour and 1.43 for every 10,000 words spoken. On some devices, 10% of misactivation durations lasted at least 10 seconds.

The failure that produces is not “bad ASR”. It is an endpointing policy with no explicit latency–truncation tradeoff. The dial can be moved to the other end, where the device stops answering the television and starts clipping the quiet ends of real sentences instead. Nobody has written down what a segment is supposed to mean at either setting. Neither direction shows up in a review that only counts how often the detector was wrong per acoustic condition. Both settings break the interaction, in opposite directions, while the detector goes on doing exactly the job it was given.

The thing that goes unwritten has a name. It is the segment contract, and it is three numbers: how much audio is kept from before a detected start, how much is kept after, and how long the system waits before giving up. Between them those three decide what a segment is. All three are fixed before any measurement happens.

Which is why a single score is the wrong instrument. Detecting speech and closing a segment each involve four distinct moves, and one number folds all four into itself. First, define the segment contract: specify pre-roll, post-roll, minimum duration, maximum duration, overlap, and timeout behavior. Second, estimate speech activity: use acoustic evidence with thresholds, smoothing, and state appropriate to the device. Third, apply the endpoint policy: combine activity, pause length, partial transcripts, turn cues, and the latency budget. Fourth, measure downstream effects: clipped words, false starts, waiting time, privacy capture, and user correction.

The third move is where the contract is spent, and it has published operating points. Google published one in 2017, for an endpointer built on a grid LSTM, and stated the result plainly: “Overall, we achieve 16% (100 ms) endpointer latency improvement relative to our previous best model on a Voice Search Task.” — Chang and colleagues, 2017. Note what surrounds that figure. The same work reports a 32% relative false-alarm improvement. It takes the false-alarm comparison at a fixed false-reject rate of 2%. The fixed 2% is not decoration. It is what makes the false-alarm number mean anything. Without a pinned operating point, either number can be bought from the other.

So the contract stands where it belongs. It is a choice made in advance — a latency budget in milliseconds, and a false-reject rate you agreed to hold constant. It is not something inferred later from results it had already shaped.

Two of the contract's terms come disguised as vocabulary, and that is the trouble with them. Pre-roll and hangover look like words to learn. They are in fact durations that somebody chose. The other two, endpoint and barge-in, name the moments where a product commits and cannot take it back.

The durations are on the record. A 1997 paper on a G.729-compatible detector opens by naming what can go wrong: “Voice activity detectors can exhibit four types of error and these are Noise Detected as Speech (NDS), Hang Over (HO), Front End Clipping (FEC), and Mid Speech Clipping (MSC).” — Watson and colleagues, 1997. Four names, and two of them are clipping at different places in an utterance. The front and the middle are separate defects with separate causes. That is already more resolution than a single accuracy figure carries. Their own hangover setting was fifteen 10 ms frames of overhang, against the five overhang frames used by the GSM VAD. Three times the tail, chosen, written down, and defended in a paper. That is what a hangover value is.

What the numbers should be is settled by none of that, because the same threshold is charged for differently everywhere it is used. In a kitchen it is paid for in seconds of waiting and in truncated commands. Move the identical policy into a call center, a meeting room, or the paragraph of a privacy notice, and the currency changes to turns, to compute, to retained audio. In one documented case it changed to a federal complaint.

On 31 May 2023 the FTC and the DOJ sued Amazon. The allegation was about what happens after activation: “Alexa's default settings save these voice recordings and transcripts indefinitely, even if the user stops using the account for months or even years.” — the complaint in United States v. Amazon.com, 2023. The case concerned children's voice recordings and transcripts. Amazon agreed to a $25 million civil penalty, and the court entered the stipulated order on 19 July 2023. Activation and deactivation are not a matter of preference in that document. They are the boundary of what a company is holding, for how long, about whom.

A policy tunes better against a charge than against a target, and there are two charges on the record. The device answers to a television. The device cuts a person off mid-sentence. Put the policy on trial for both.

The second charge has a published example. Google replaced its VAD segmenter in 2022 and said why: “VAD segmenters, however, may be sub-optimal for real-world speech where, e.g., a complete sentence that should be taken as a whole may contain hesitations in the middle ("set an alarm for... 5 o'clock").” — Huang and colleagues, 2022. Their alternative predicts segment boundaries while decoding rather than before it. They tested it on real-world YouTube audio up to 30 minutes long, on a Conformer RNN-T. Against the VAD baseline it improved relative WER by 8.5% and cut median end-of-segment latency by 250 ms. Both directions moved at once. That is the point: a segmentation change is a transcription change and a latency change in the same act.

So work the loop deliberately. Collect difficult turns — hesitations, long pauses, quiet endings, noise, overlap, and playback. A pause in the middle of "set an alarm for... 5 o'clock" is the canonical item, and 134 hours of television is the canonical distractor. Sweep thresholds and delays to produce latency–truncation curves rather than choosing one accuracy point. Inspect downstream traces, linking segmentation errors to ASR, intent and user retries; 8.5% of relative word error rate is a quantity only visible after the segmenter. Then define a safe fallback: manual push-to-talk, clarification, cancellation, and timeout recovery.

Then write three lines. What your segment contract assumes, since pre-roll, post-roll and timeout each encode a belief about how people talk to this particular product. One counterexample — a room in which that belief is false and the contract commits anyway. And what the downstream measurement can tell you afterwards and what it cannot. It will be measured inside the contract you have just written.

“False alarm and missed speech by acoustic condition” is easy to demand and rarely printed, so here is one printed. AVA-Speech is a Google dataset from 2018. It provides 45 hours of densely labelled speech activity, taken from 15-minute clips of 185 YouTube movies, split into clean speech, speech with music and speech with noise. Comparing detectors at their own preferred settings is meaningless, so the authors pin the operating point: “Since the WebRTC VAD has a false-positive rate (FPR) of 0.315 on our NoSpeech segments, we report true-positive rates (TPR) for all models tuned to the same FPR.” — Chaudhuri and colleagues, 2018.

At that fixed false-positive rate of 0.315, the WebRTC VAD reaches a true-positive rate of 0.786 on clean speech and 0.706 on speech with noise. A ResNet-50 acoustic model, at the identical operating point, reaches 0.992 and 0.944. Read the spread rather than the winner. The same detector loses eight hundredths of its true-positive rate when noise arrives. The two models are separated by more than two tenths on clean speech alone. One averaged accuracy figure would have shown none of that. A detector can hold a respectable per-condition number while a device is answering to a television, because a television emits exactly the kind of energy the detector was built to find.

So the figure needs provenance and it needs company. Give the unit. Say whose speech was measured. Say how uncertain the number is, and what operating point the system was held at. Then put three other kinds of number next to it.