Transducers and Streaming Recognition

A transducer is three parts working at once. An acoustic encoder turns audio into a representation. A prediction network summarizes the output labels already emitted. A joint network puts the two together and either emits the next label or emits blank. The arrangement was defined in 2012. Alex Graves named the two RNNs the transcription network and the prediction network, and extended the output space with a null symbol whose "intuitive meaning" is 'output nothing'. The alignment lattice then advances through acoustic time and through output history. That is what lets it recognize incrementally, without having to hand a label to every frame.

That is a real capability. It is not the same purchase as speed. Streaming architecture guarantees neither low perceived latency nor stable partials. Chunking, lookahead, endpointing, beam search, hardware, commit rules and revision policy jointly determine what the user experiences. The model is one contributor out of seven. Google's own paper on shipping a streaming recognizer to phones states the target in user terms rather than model terms. The system must be at least as accurate as a conventional one “without increasing latency (i.e., the delay between the user speaking and the text appearing on the screen), thus running at or faster than real-time on mobile devices”.

Two families of numbers describe the result, and they measure different things. First-token, partial, final and endpoint latency ask how quickly words arrive. Partial-result stability and revision distance ask how much they move once they have. Neither stands in for the release decision on its own. A system can look good on one and fail the other. Both families have been measured on a shipped recognizer. That measurement is where this lesson goes next.

Eleven words moved on screen to produce a nine-word transcript. That is one utterance from a shipped on-device recognizer, measured by Shangguan and colleagues in 2020. Their worked example is the whole problem in one row. The unstable partial word ratio is 11/9 = 1.22. Five segments were unstable over that single utterance, an unstable partial segment ratio of 5/1 = 5.0. More words moved than ended up in the transcript. The sentence was rewritten five times in front of whoever was reading it. Nothing there is a broken recognizer. It is a product that never decided the point at which a word on screen becomes a word the user is allowed to act on.

The same paper breaks the instabilities down, and the breakdown contradicts the intuitive story that partials move because the decoder heard more audio. Only 52.4% of observed instabilities are streaming instability. The remaining 47.6% come from text normalization: capitalization 24.7%, punctuation and spacing 21.2%, numerals 1.7%. Nearly half the rewriting a reader sees has nothing to do with acoustics at all. No amount of extra lookahead removes it.

The transport then adds a budget that no comparison of models is counting. A numbered standard fixes it rather than estimating it. RFC 6716 states it: “The Opus codec scales from 6 kbit/s narrowband mono speech to 510 kbit/s fullband stereo music, with algorithmic delays ranging from 5 ms to 65.2 ms.” A latency figure that names only the model is not the figure a user feels.

Four terms carry that story, and two of them mean less than they appear to. Partial stability is a promise made to a user rather than a property of a model — UPWR and UPSR had to be defined before it could be counted at all. A blank transition is an alignment step rather than a silence. Graves is explicit about what the prediction network does, and about the option the null symbol adds: “The prediction network attempts to model each element of y given the previous ones; it is therefore similar to a standard next-step-prediction RNN, only with the added option of making 'null' predictions.” Words that pass for near-synonyms here can owe different proof, come in different units, and place the decision somewhere else entirely.

Label history is the axis these streaming designs differ on, and the prediction network is only one answer to it. CTC streaming works from acoustic context, with framewise label and blank predictions and no memory of what it has already written. Transducer streaming keeps that memory. Chunked attention reaches for context a third way, with bounded blocks and lookahead. Both buy more history than CTC does, at a different price.

The first of those differences was priced in 2012, on identical inputs. On the TIMIT core test set, Graves reports a phoneme error rate of 25.5% for CTC, 23.2% for the RNN transducer, and 72.9% for the prediction network alone. Label-history conditioning bought 2.3 percentage points over CTC. The label history by itself, with no audio, gets 72.9% of the phonemes wrong. Neither component is the system. The combination is. Graves states what each column measures: “Phoneme error rate, which is customarily presented as a percentage, is recorded for both the transcription network and the transducer. The error recorded for the prediction network is the misclassification rate of the next phoneme given the previous ones.”

That is 2.3 points, on one corpus, in 2012. It is not a constant. None of the three designs inherits the evidence of another — the same reason the Parakeet, Whisper and FastConformer rows settle nothing outside the long-form track they came from.

A system that returns text while the speaker is still talking makes three separate decisions, and the shipped recognizer above made all three. It encoded the audio it had so far. It committed words to the screen. It revised them afterwards — eleven unstable words for a nine-word transcript. Only the first two reach the user in time.

Keeping encoding, committing and revising apart matters because they are not decisions of the same kind. Encoding is a modeling choice. Committing is a product choice about what a reader may rely on. Revising is what the second choice costs when it is made too early. The cost is measurable in both directions: at a 200 ms partial emission interval, the more stable model shows a 71.6% improvement in UPWR and 68.8% in UPSR. Stability was bought. The next section says with what.

The commit rule is a document, and a short one. For anything running in a browser, part of it already exists as a normative default rather than a team convention. The Web Speech API specification marks the boundary on the result itself: “The final boolean must be set to true if this is the final time the speech service will return this particular index value. If the value is false, then this represents an interim result that could still be changed.” MDN Web Docs states the same promise from the reader's side — “if not, then this result is an interim result, and may be updated later on”. And SpeechRecognition.interimResults defaults to false, so interim results are not returned at all unless the application asks for them. Step 1 below is therefore a decision to opt in, not an invention.

Steps 2 and 3 have a published price. “Changing the partial emission interval from 50ms to 200ms results in 75ms increase in mean partial delay,” the 2020 paper reports. Seventy-five milliseconds of mean partial delay bought 71.6% off UPWR and 68.8% off UPSR. That is the shape of the trade a contract has to state, in a unit someone else can recompute. Write it to be used against you. Another team ought to be able to pick it up and argue that your delay was measured from the wrong starting point.

And remember where nearly half the movement came from — capitalization 24.7%, punctuation and spacing 21.2%, numerals 1.7%. A contract that only governs the decoder governs 52.4% of the problem.

A latency number cannot, on its own, tell you where the clock was started or what it was allowed to leave out. A team can report an excellent first-token figure and be telling the truth the whole time, while “the delay between the user speaking and the text appearing on the screen” is something else entirely. So write down the unit, the sessions it was measured over, how uncertain it is, and the network, hardware and numeric precision it ran on. RT90 1.43, 1.03 and 0.51 came from one model. Then set stability beside it.

Stability has defined units, so there is no excuse for reporting it as an impression: “UPWR is the ratio of total number of unstable words in a test corpus to the total number of words in the final hypotheses. UPSR captures the ratio between the aggregated number of revised segments and the total number of utterances in a dataset.” Both range over [0, ∞), with 0 the most stable, and a reviewer can recompute them from a log.

Four kinds of evidence make that report readable, and each covers a gap the others leave open.