ASR Serving, Latency, Privacy, and Monitoring

The same recognizer can be deployed in three shapes, and they are not variants of one another. Offline batch ASR works on stored recordings. It can hand the model the complete file before committing to a word, and optimize throughput while it does. A streaming service has to answer while the speaker is still talking. On-device recognition has to answer inside a battery and a thermal envelope. The quantized RNN-T that He and colleagues ran on a Google Pixel phone in 2019 occupied 120 MB. No server-side deployment ever has to answer for a number like that.

Each shape is built against a different constraint. The evidence that clears one does not clear the others. A throughput figure says nothing about what a caller waits for. Neither figure says anything about what a handset does to its own power draw. The rest of this lesson works mostly on the middle case. That is where the measurement most reliably goes wrong.

The three columns below name, for each shape, the decision it turns on, the evidence that speaks to that decision, and the habit that most often substitutes for the evidence.

Take a number that a real streaming system actually publishes. He and colleagues, reporting an on-device recognizer in 2019, say exactly what theirs measures: “In Table 4, we report WER and RT90, i.e. real time factor (processing time divided by audio duration) at 90 percentile, where lower values indicate faster processing and lower user-perceived latency.” Their symmetrically quantized RNN-T reaches RT90 = 0.51 — twice as fast as real time — at 7.3% voice-search word error rate. The float word-piece model sits at 1.43. The quantized CTC baseline sits at 0.86.

Notice what that figure is and is not. It is a 90th percentile, which is already better discipline than an average. And it is scoped to processing time divided by audio duration: model compute. A service can carry an RT90 of 0.51 and still put the caption on the screen two seconds after the words were spoken. Those two seconds accumulate across network buffering, voice activity detection, queueing, decoding and punctuation. Then comes the interface's own commit policy — the moment the UI is willing to put a caption up and stop revising it. Nothing in a real-time factor accounts for the buffer, the queue, or that moment. The published figure is correct and the service is late. Both statements stay true at the same time. This is the reference point for everything below.

Put the two seconds, the 150-millisecond budget and the FCC's synchronicity paragraph side by side. The shape of the problem is a chain rather than a component. ASR serving includes capture, transport, buffering, segmentation, model inference, decoding, post-processing, delivery, storage, and feedback. Model compute is one item on that list for latency. It is one item for cost, for privacy exposure, and for operational risk too.

Watching such a chain is harder than watching a classifier, and there is a measured defect that shows why. A 2024 study of Whisper transcripts reports: “While many of Whisper's transcriptions were highly accurate, we find that roughly 1% of audio transcriptions contained entire hallucinated phrases or sentences which did not exist in any form in the underlying audio.” Of those hallucinations, 38% included explicit harms. They concentrated on speakers with longer non-vocal durations, such as speakers with aphasia. A defect at roughly 1% of outputs is invisible to every latency and throughput dashboard in the building. The pipeline was fast, the queue was shallow, the transcript arrived, and the sentence was not in the audio. Detecting it means re-running audio and comparing transcripts. Somebody has to correct a transcript before anyone learns it was wrong. And because the hallucinations concentrate on particular speakers, the sample you review decides what you can find. The raw material is a recording of a person speaking, which is why logging everything is neither necessary nor safe. The proxy metrics that fill the gap in the meantime can drift on their own.

What you are left with is two views of one service, answering different questions. Per-user percentiles say how long one person waits. Fleet metrics — throughput, real-time factor, cost, queue depth — say how much work the service is absorbing. The RT90 of 0.51 in the case above was the second kind of number, used to answer the first kind's question. Neither view answers the other's. A release decision needs both.

Live captions, call transcription, on-device assistants, and enterprise ASR run much the same pipeline against four different budgets. It is the budget rather than the pipeline that fixes the architecture. The case above belongs to the first of the four. The other three would have failed differently, and would have needed different units to show they had not.

An operations dashboard earns its place when it can settle an argument about where the delay came from instead of starting one. The test is the case above. On the morning after, could this dashboard say which of those six stages the two seconds came from, when the model's own RT90 was 0.51?

Step 2 has a documented precedent. It includes the sampling rate a real deployment used, and what it had to change once the sampling became public. Apple's Siri "grading" programme had human reviewers listen to a sample of Siri audio. It was suspended after press disclosure in July 2019. On 28 August 2019 Apple stated: “Before we suspended grading, our process involved reviewing a small sample of audio from Siri requests — less than 0.2 percent — and their computer-generated transcripts, to measure how well Siri was responding and to improve its reliability.” Apple announced that by default it would no longer retain audio recordings of Siri interactions. It made audio sampling opt-in and restricted listening to Apple employees. A related class action, Lopez v. Apple, had been filed on 7 August 2019. Less than 0.2 percent is a small number, and it was still the thing that had to change. The design question is not only how much you sample. It is whether the sampling survives being described in public.

Write down what your end-to-end latency budget assumes. Then write one case that breaks it — the 60-millisecond Opus frame, a queue that grows under load, a commit policy tuned for stability rather than speed. Then write what the on-call response is when that case appears.

That dashboard has to carry both of the views the chain forces apart, and then a third kind of evidence that only accumulates over time. The four layers below are that list.

Why the per-user percentile and the fleet average cannot substitute for each other has a number attached to it. Take a server that responds in 10 ms typically but has a 99th-percentile latency of one second. On its own it makes one request in 100 slow. Dean and Barroso, writing in 2013, worked out what that server does inside a fan-out: “If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second”. At a real Google service, the 99th-percentile latency for a single request measured at the root was 10 ms, while the 99th percentile for all requests to finish was 140 ms. A modest tail at one stage becomes the common case once a request crosses many stages. Capture, transport, buffering, segmentation, inference, decoding, post-processing and delivery are exactly that.

The same argument applies to accuracy. There the slice that matters is usually a group of speakers rather than a group of servers. A 2020 PNAS study tested five commercial ASR systems — Amazon, Apple, Google, IBM and Microsoft — on 19.8 hours of matched interview audio from 42 white and 73 black speakers. It measured “an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers”. That is a gap of 0.16 absolute WER between speaker groups, across five deployed services at once, and an aggregate accuracy number hides it completely. Report the difficult slices with whichever layer you are reporting — the calls where the service is slowest, the speakers for whom it is least accurate — not the average taken across all of them. The case above passed comfortably on the average. Read the four layers together and no single number is left carrying the release claim on its own.