Conformer and Transformer Speech Encoders

Three encoders sit behind this lesson, and it is worth seeing at the start how little they have in common.

A transformer encoder is built mostly out of self-attention and feed-forward blocks. Every position can look at every other one. Nothing in the design says how far away is too far. A conformer encoder keeps those blocks and adds convolutional modules beside them. The convolutions pull local acoustic structure back into a model that otherwise treats a sequence as positions to be mixed. Gulati and colleagues published that architecture in 2020, as a convolution-augmented transformer for speech recognition.

A streaming variant goes the other way. It restricts the context around the same blocks, because it has to produce words before the utterance is finished. In the Transformer Transducer, published the same year, the restriction is literal and countable. A mask of left 512 / right 512 frames becomes left 10 / right 0.

Adding context and restricting it are different moves, made for different reasons. They do not land on a common scoreboard. Each of the three has to be measured on its own. A number that ranks one of them has said nothing about the other two.

Skip that measurement and this is what follows. On a single A100 GPU at batch size 1, an unmodified Conformer can transcribe at most 15 minutes of audio in one pass. A Fast Conformer reaches 25 minutes. Put limited-context attention into the plain Conformer and it reaches 135 minutes. Put it into Fast Conformer and it reaches 675. Those four numbers were measured in 2023, by the team that built Fast Conformer. The wall is measured, not an anecdote about an unnamed team.

The paper states the exchange rate directly: “By switching to limited context attention, we extend the maximum duration that the model can process at once on a single A100 GPU by 45x: from 15 minutes for unmodified Conformer to 675 minutes for Fast Conformer with limited context”.

Read the two ends of that sentence together and the shape of the failure is obvious. Global attention was the change made for accuracy. At hour-long durations it is the change that makes the model unrunnable. Fifteen minutes is less than one hour, on an A100. The 45x is recovered by restricting context — putting back, deliberately and by hand, the limited context that a recurrent encoder would have imposed for nothing.

A review that compares word error rates at matched parameter, data, and decoding budgets sees none of this. The review is not wrong. The cost simply landed where it does not look: in memory and in maximum admissible duration, not in the error rate.

The team that has to put limited context back is not dealing with a bad architecture. Transformer encoders model content-dependent global interactions. That is a real capability, and the reason the swap looks worth making. Conformer blocks keep it and add convolutional modules that emphasize local acoustic structure alongside attention. What that combination scored is why the architecture spread as fast as it did, and the 2020 abstract reported it in one sentence: “On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.”

The small model is the interesting row, and it is the row that gets skipped. It is the one that fits on a device. The headline 2.1%/4.3% says what the architecture can do when size is not the constraint. The 2.7%/6.3% at 10M parameters says what is left when it is.

Years later the ranking has held under independent measurement. The Open ASR Leaderboard compares 86 open-source and proprietary systems across 12 datasets, in a paper first posted in October 2025 and revised on 30 March 2026. Its conclusion: “We observe that Conformer-based encoders paired with transformer-based decoders achieve the best average WER, while connectionist temporal classification (CTC) and token-and-duration transducer (TDT) decoders offer superior RTFx, making them better suited for long-form and batched processing.” A Hugging Face engineering post on 21 November 2025 describes the same leaderboard, as of that date, as comparing 60+ open and closed-source models from 18 organizations across 11 datasets.

Notice that the leaderboard's own sentence refuses to settle the architecture choice. Best average WER and superior RTFx land on different decoders in the same finding. The second half is the half that a citation of “the top of the leaderboard” drops.

Neither architecture is inherently optimal for all three of our objects. Full attention cost grows with sequence length, which is what the 15-minute ceiling was charging for. Limited context changes how much evidence is reachable, which is what the 675 minutes bought. Convolution buys locality; it does not automatically guarantee robustness. So the block with its name on the architecture is only part of what is under comparison. Subsampling, positional encoding, normalization, residual paths and context masks matter as much as the headline block. Compare a transformer encoder, a conformer encoder and a streaming variant on the audio you will actually serve, before the architecture is fixed.

Those less glamorous parts have an ordering problem, and the frame rate is the first of them. It is set before anything else runs: “The encoder starts with a sub-sampling module, which increases the frame rate from 10 ms to 40 ms.” That is a 4x reduction, taken before any attention or convolution block sees the signal. Fast Conformer takes it to 8x — 10 ms to 80 ms — using three depth-wise convolutional sub-sampling layers. That cuts the compute of the subsequent attention layers by 4x. It makes the encoder 2.8x faster, with 2.9x fewer multiply-add operations.

So the choice made in the first hour is a real one, with published numbers on both sides: 40 ms per frame or 80 ms, and 2.8x of encoder speed riding on it. Reducing the frame rate is close to the first thing an encoder does. Benchmarking the system is the last thing anyone does. Between the two, nobody goes back. Modeling local and global context is the step teams assume rather than test. When it is assumed, the subsampling factor is never re-examined, and the benchmark at the end quietly reports on it as though it were a property of the audio — 80 ms of it at a time.

The far end of that chain is also where the 15-minute ceiling turned up. Memory and maximum duration belong to a whole system running at real lengths, not to any one block. So they stay invisible until the model has already been built around them.

Most of that argument rides on four words, and two of them are prices rather than parts. Lookahead is latency spent in advance. Subsampling is resolution spent for compute. Neither is a component you can point to in a diagram. Both are paid before any benchmark runs.

The Transformer Transducer priced the first of them in one table. Full attention, left 512 / right 512 frames, gives 2.4%/5.6% WER on LibriSpeech test-clean/test-other. The streamable configuration, left 10 / right 0, gives 4.2%/11.3%. Between those two extremes the exchange rate is in seconds. A right context of 6 frames per layer costs roughly 3.2 seconds of aggregated latency. Two frames per layer — about 1 second of latency — recovers about 30% relative over the streaming model.

In 2020 Zhang and colleagues summarised the left-hand side of that bargain: “We present results on the LibriSpeech dataset showing that limiting the left context for self-attention in the Transformer layers makes decoding computationally tractable for streaming, with only a slight degradation in accuracy.” One word standing in for another can change the proof owed, the unit, and often who decides.

Who decides is easier to see when the same encoder block is sent to four jobs whose context budgets have nothing in common. A global-attention encoder is a plain success in the first row below and a plain failure in the fourth. It does not change in between. The same design that reaches 2.4%/5.6% with left 512 / right 512 is the one that stops at 15 minutes of audio on an A100. Each of the four needs its own units and its own demonstration. The maximum-duration table carries none of the accuracy figures. The accuracy table carries none of the durations. That is why they are worth holding apart rather than collapsing into one number for the block.

Only an evaluation able to contradict itself is worth running, and there is a published instance of one contradicting the average it was reported inside. Five commercial ASR systems — Amazon, Apple, Google, IBM and Microsoft — were tested on 19.8 hours of interviews with 42 white and 73 black speakers across five US cities. Koenecke and colleagues published the result in PNAS in March 2020: “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers.” The gap was equally large on identical phrases, which removes the easy explanation that the two groups were simply saying different things. A slice that comes apart while the average holds is not noise around the measurement. The 0.35 against 0.19 is the measurement.

It is worth being exact about what the headline numbers earlier in this lesson were measured on instead. All of them are LibriSpeech figures. LibriSpeech is a 2015 corpus, distributed as OpenSLR SLR12 with train-clean-100, train-clean-360, train-other-500 and the dev and test clean and other splits. The corpus page describes it plainly: “LibriSpeech is a corpus of approximately 1000 hours of 16kHz read English speech, prepared by Vassil Panayotov with the assistance of Daniel Povey. The data is derived from read audiobooks from the LibriVox project, and has been carefully segmented and aligned.” Read audiobooks are not the interview audio in which 0.35 and 0.19 diverged. A 2.1% on test-clean is a true statement about read English speech. It says nothing yet about the speakers a deployment will meet.

The evaluation should also mark the place where short acoustic events have already been lost to 40 ms or 80 ms frames. Then say what the system does there — whether it abstains, falls back, or carries on emitting words for sounds it no longer has.

The exercise is the smallest version of all of this: an encoder architecture ablation that changes one thing at a time. One thing means one — the block, or the subsampling factor, or the lookahead, and never two together. The whole difficulty in the case above was a single change that moved accuracy one way and the maximum admissible duration the other. The three knobs come with published settings to sit between: 4x subsampling at 40 ms per frame or 8x at 80 ms; global attention or limited context; and a right context measured in frames per layer, where 6 frames costs roughly 3.2 seconds of aggregated latency and 2 frames about 1 second.

Three notes are enough, and they are the ones another team will need. What does reducing the frame rate to 40 or 80 milliseconds assume — which sounds is it betting are not load-bearing? Then one counterexample, a clip where that bet is wrong. Then what benchmarking the system does with the result, once latency, memory and the longest recording you must accept in one pass are back in the picture.