Keyword Spotting, Wake Words, and Command Recognition

Keyword spotting is the job of finding a small vocabulary, or a single wake phrase, inside a stream of audio that never stops. Everything about the task follows from one word: continuous. A low-power front end runs all the time. Temporal context is carried across frames. A threshold is crossed or it is not. Then the decision is taken again, and again, for as long as the device has power — often with a second-stage verifier behind the first to catch what it waved through. False-alarm control is not a feature bolted onto this system. It is most of the system.

Which is why a clip accuracy figure predicts almost nothing about a day. A benchmark asks a fixed set of questions once. A room asks the same question across thousands of windows a day, and almost every one of them has the same correct answer, which is no. A model that is right on nearly every clip can still be wrong often enough per hour to make the device unusable. The accuracy column will not show it.

Two more things that column does not settle. A detected phrase does not establish that it was aimed at the device rather than at the room. Nor that the speaker is authorized to be giving it instructions.

So the thing worth designing first is not the model. It is a trigger contract: what counts as a trigger, at what cost in false accepts, with somebody owning the writing of it and somebody owning the tuning that follows. The tuning never stops. Write that contract without a per-hour number in it and the cost has already been measured, on shipped hardware from four manufacturers with a television playing: 0.95 misactivations per hour.

That contract is written once, and the two steps after it inherit whatever it says. Its terms decide what gets collected: which utterances count as positives, and — the part that does the real work — which sounds are worth collecting as hard negatives. That is where the television goes, and the children, and every near-miss the room has already produced. Then the system meets a live stream and is tuned against it continuously, for as long as it ships.

Step two is usually described as a wish list. It can be done as a procedure instead. Hunting for the phrases that wake a speaker by accident can be mechanised. In 2020 Schönherr and five colleagues mechanised it. Their abstract says how: “More specifically, we automate the process of finding accidental triggers and measure their prevalence across 11 smart speakers from 8 different manufacturers using everyday media such as TV shows, news, and other kinds of audio datasets.” The candidate near-misses were crafted with a pronouncing dictionary and a weighted phone-based Levenshtein distance — a count of how few phone substitutions separate a harmless phrase from the wake word. The work ended in a released dataset of more than 1,000 verified triggers. That is what a finished step two looks like. Not the words television, children, music in a planning document. A set with a number attached to it that somebody else can rerun.

The assumptions from the first step either hold in the live stream or they do not, and the tuning has no way to tell you which. It optimizes toward the terms it was handed. A term set well is pursued; a term set badly is pursued just as hard.

This failure does not have to be imagined. It has been measured, on hardware anyone can buy. Six researchers put popular smart speakers in front of a television and left them there: Google Home Mini, Apple HomePod, Amazon Echo Dot 2nd and 3rd generation, and the US-only Harman Kardon Invoke — four manufacturers. The material was 134 hours of content from 12 TV shows: 217 episodes, 1.057 million words, played through twice, in the US and in the UK. Their own summary of what came back, in a peer-reviewed paper in 2020: “After playing two rounds of 134 hours of content from 12 TV shows near popular smart speakers in both the US and in the UK, we observed cases of 0.95 misactivations per hour, or 1.43 times for every 10,000 words spoken, with some devices having 10% of their misactivation durations lasting at least 10 seconds.”

Read the three figures as three different units, because that is the lesson in them. Per hour is the room's unit: leave the television on through an evening and the device answers a question nobody asked it. Per 10,000 words is the exposure unit. It is 1.43 times, and it scales with how much talking happens near the device rather than with how long the device is switched on. The third figure is not a rate at all but a distribution. On some devices, 10% of the misactivation durations lasted at least 10 seconds — the difference between a light blinking and ten seconds of a household's conversation leaving the room.

No clip accuracy column reports any of the three. The dataset behind such a column has usually been balanced: a sensible thing to do to a classification problem, and a fatal thing to do to a rare event. A review that counts only clip accuracy leaves that gap unchallenged, because the balance is invisible in the score it produces.

A smoke detector has exactly one thing to do once it fires. A wake word does not. What happens after the phrase is detected is what separates three systems that look identical right up to the moment the threshold is crossed. A wake word hands control to a broader speech or assistant pipeline, so its errors are paid for downstream, by whatever it woke. Which is why the durations matter as much as the rate: on some of the devices in front of that television, 10% of misactivation durations lasted at least 10 seconds. What a woken pipeline does with those seconds is a separate question from whether the threshold should have moved. A keyword spotter and a command classifier are being asked for other things entirely. The three cannot share evidence. Each has to be evidenced in its own terms, and a number earned by one of them settles nothing about the other two.

Through all of that, four terms have been doing quiet work. Each is an operational commitment rather than a definition: a unit somebody has to report, a set somebody has to collect, a stage somebody has to pay for, a session somebody has to run. Similar-sounding words here imply different evidence, different units, or different decision rights, so they are worth documenting one at a time.

So what has to be in the report? Begin with the pair that describes the trigger in the room's units rather than the clip's: false accepts per hour, and misses per target utterance. The published example of exactly that shape is the Hey Snips work from 2019. It fixes the operating point first — 0.5 false alarms per hour — and only then quotes a miss rate. It quotes that twice. A 222K-parameter dilated-convolution model gives 0.12% false rejection on clean positives and 1.60% on positives at 5 dB SNR. An LSTM baseline gives 2.09% and 11.21%. Same model, same threshold, noisier room, and the number moves. Their conclusion states the discipline plainly: “We have shown that the presented WaveNet model significantly reduces the false rejection rate at a fixed false alarm rate of 0.5 per hour, in both clean and noisy environments, on a crowdsourced dataset made publicly available for research purposes.” The underlying set is Hey Snips: about 11K wake-word utterances and 86.5K negatives, roughly 96 hours, from over 2.2K speakers. Note the shape of the negatives — the hours, not the clips.

Those numbers say what the detector does and nothing at all about what using it feels like. So detection latency and partial-word behavior belong next to them. Then the slices, because an average taken over a mixed corpus is the balanced dataset again in another costume. Rooms where the keyword is genuinely rare. Performance broken out by language, accent, age, device, distance, and playback. Children, distance and a television playing are not edge cases; the television is the condition under which the four manufacturers' speakers reached 0.95 misactivations per hour. And the language slice has a prerequisite. It can be evidenced against 50 languages and 6,601.39 hours. It cannot be evidenced against 27.92 hours of one language.

Clips cannot measure any of that. Continuous audio is the only setting in which a team other than your own can demonstrate a dataset balanced while ignoring real-world keyword rarity. Hand them recordings from the places the device will actually live, and let them count the accepts per hour themselves. The reference protocol already exists and it is not exotic: 217 episodes drawn from a dozen series, 1.057 million words, 134 hours of it, played twice on both sides of the Atlantic. One evening of a television is enough to start. Two rounds is enough to publish.

The last step of that evaluation, testing privacy behaviour, is the one teams treat as advisory. It is not. Alexa kept what it heard, and on 31 May 2023 the United States sued Amazon over it on the FTC's behalf. The complaint states the mechanism plainly: “Alexa's default settings save these voice recordings and transcripts indefinitely, even if the user stops using the account for months or even years.” The alleged violation was of the COPPA Rule, over children's voice recordings and transcripts. Amazon agreed to a $25 million civil penalty. It agreed to delete inactive child accounts, certain voice recordings and geolocation data, and to stop using that retained data to improve its products. A retention default is not a settings screen. It is the thing a filed federal complaint turns on.

Three notes go with the audio, and they are the notes the next team will need. Write down what your trigger contract assumes about the room. Write down one room where that assumption is false. Write down what continuous tuning then does, because that is the step nobody records.