CTC and Monotonic Alignment

A transcript can be exactly right and its timestamps quietly wrong. That gap has been published, measured and given units.

The 2024 paper that measured it separates the behavior from the defect in two sentences: “Connectionist temporal classification (CTC) models are known to have peaky output distributions. Such behavior is not a problem for automatic speech recognition (ASR), but it can cause inaccurate forced alignments (FA), especially at finer granularity, e.g., phoneme level.”

Read that twice. Only the second sentence describes something broken. The transcripts are right. The model is not sick. What is wrong is a reading — frame-by-frame posteriors taken as the moments labels begin and end. The size of that wrongness is a number nobody in a transcription pipeline is collecting.

Huang and colleagues collected it. Their fix was label priors, and it “outperforms the standard CTC model and a heuristics-based approach for obtaining CTC's token offset timestamps by 12-40% in phoneme and word boundary errors (PBE and WBE) measured on the Buckeye and TIMIT data.”

So between 12% and 40% of the boundary error in a standard CTC aligner was removable. Two named corpora, one target: peakiness. Every one of those systems would have reported the same sequence error before and after. And the story does not end in triumph. With the peakiness reduced, the method still falls behind the Montreal Forced Aligner on TIMIT.

Not picking is what the objective is built to do. Connectionist Temporal Classification defines a distribution over frame-level paths containing labels and blanks. It then sums the probabilities of every path that collapses to the target sequence.

The collapse rule is not folklore. The 2006 paper states it as a one-line operation: “We do this by simply removing all blanks and repeated labels from the paths (e.g. B(a − ab−) = B(−aa − −abb) = aab)”. Nobody has to mark where each label starts. Monotonic order survives because the paths themselves only run forward, and the sum handles the rest. Repeated labels are the one place the rule is strict. Two identical labels in a row need a blank between them, or the collapse merges them into one. The same example shows exactly that.

Two consequences follow, and both come due later in this lesson. The first is written into the definition itself. Graves and his co-authors state it plainly: “Implicit in (2) is the assumption that the network outputs at different times are conditionally independent, given the internal state of the network.” Each frame is scored as though the others were not there.

The second is that the alignments are latent explanations under the model, not guaranteed phonetic boundaries. The paper's own figure caption says what the output looks like: “The CTC network predicts only the sequence of phonemes (typically as a series of spikes, separated by ‘blanks’, or null predictions), while the framewise network attempts to align them with the manual segmentation (vertical lines).” Spikes is the authors' own word. And note which of the two networks was ever asked about the manual segmentation. Model size settles little here either. A bigger encoder buys better sequences, not more honest boundaries.

That sixty-second unit is where most of the confusion starts. It carries an ordinary English name and an unordinary meaning. Blank is a token in CTC, not a period of silence. A model can spend blank frames on loud audio and label frames on something nearly inaudible. Nothing in the 2006 definition ties it to the acoustics. It is one of the 62 categories in the softmax layer, and the collapse rule removes it along with repeated labels.

A path is not a route through a graph either. It is one frame-level sequence, one of the ticker's display histories. The collapse function and forward–backward are what connect the two.

Get either word wrong and a diagnostic changes what it has to prove — and changes who is entitled to act on the result.

Collapsing is the last thing that ever happens to a blank, and by then every interesting decision has already been made. There are three stages: encode the acoustic sequence, predict labels and blank at every frame, then marginalize and decode. Blanks exist only in the middle stage. They are produced there and cleared at the end. Nothing further down the chain sees one.

That middle stage is also the last point at which the first stage can be questioned. What the encoder kept, and what it quietly dropped — a frame rate too coarse for a short sound, say — arrives at marginalization as settled fact. The sum is faithful to the paths it was handed. It has no opinion about the ones it never received.

The 2006 network did a single job: phoneme label sequences on unsegmented speech, 61 categories and a blank. The same objective now does at least four. They do not all want the same evidence.

Forced alignment is the one that touches the label-prior result head-on. It can use posterior structure, but it only survives doing so if the calibration is done separately, as its own piece of work. That is exactly what label priors are for. Keyword spotting searches an open vocabulary through CTC prefixes. Streaming transcription and low-resource ASR take the objective somewhere else again.

The low-resource case has a named shape. The wav2vec 2.0 paper describes its recipe: “Pre-trained models are fine-tuned for speech recognition by adding a randomly initialized linear projection on top of the context network into C classes representing the vocabulary of the task... Models are optimized by minimizing a CTC loss”. For Librispeech that vocabulary is 29 character tokens plus a word-boundary token. Thirty units and a CTC loss on top of a pretrained encoder.

What it buys is in the abstract: “Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER.” With all the labels, the same system reaches 1.8/3.3 WER on Librispeech test-clean/test-other. None of those four jobs is discharged by the sequence error and alignment diagnostics that were supposed to discharge the case above.

Sixty-one phoneme categories cannot be enumerated by hand. Two labels over four frames can, and that is the exercise. Write out every path of labels and blanks. Collapse each one with the paper's own rule — remove blanks and repeated labels — and see which survive to the target. Use the 2006 worked example as your check: B(a − ab−) and B(−aa − −abb) both give aab. A tiny CTC example enumerated by hand is the fastest way to see where the peaks actually fall, and why they fall there rather than on a boundary.

Then write down three things. What encoding the acoustic sequence assumes. One counterexample to that assumption. What marginalizing and decoding then does with the counterexample once it is on the table.

Do the exercise and the timing problem stops being arguable. It becomes measurable. The instrument is an independent aligner, and there is a published one with error bars.

The Montreal Forced Aligner, published in 2017, reports absolute differences from hand annotation on Buckeye. Word boundaries: 24.1 ms mean, 15.8 ms median. Phone boundaries: 17.0 ms mean, 11.2 ms median. The comparisons in the same table are FAVE at 24.7/16.6 and 19.3/12.0, and PLA-LS at 30.5/15.6 and 24.0/13.9.

Those averages are the trap, and the authors say so themselves: “While phone and word-level alignment is comparable to human annotators on average, the force-aligned boundaries do contain more medium-to-large alignment errors (>25 msec).” The distribution behind the 24.1 ms is right-skewed. 68% of word-boundary differences fall under 25 ms and about 90% under 50 ms, while 2–5% of tokens differ by at least 100 ms. A tenth of a second is visible in a caption.

So an evaluation reports the tail, not the mean. And it reports it next to sequence error, not instead of it. The two will sometimes conflict. The conflict is the point.

The portfolio also has to name where the reading of the peaks stops holding, and say what the decoder does when it arrives there: abstain, or fall back.