TTS Acoustic Models, Alignment, Duration, and Prosody

Three attention mechanisms went into the same listening test and came out with the same score. Location-sensitive attention 4.31 ± 0.06. GMMv2b 4.32 ± 0.06. Dynamic convolution attention 4.31 ± 0.06, all on Lessac. About 4.3 for every one of them. Google Research published the comparison in 2020. On that evidence there is nothing to choose between the three.

Then the utterances got long. The same paper describes that test in one passage: “Now we evaluate our models on long utterances taken from two chapters of the Harry Potter novels. We use 1034 utterances that vary between 58 and 1648 characters (10 and 299 words). Google Cloud Speech-To-Text is used to produce transcripts of the resulting audio output, and we compute the character errors rate (CER) between the produced transcripts and the target transcripts.” The Tacotron 2 location-sensitive attention baseline's CER shot up at roughly 3× its maximum training length. The location-relative mechanisms held across the whole range, 58 characters to 1,648. Same voice. Same in-domain MOS of about 4.3. One of them stops delivering the text.

That is not a defect of one listening test. It is the shape of every headline naturalness number in this literature. Tacotron 2 reported a mean opinion score of 4.526 ± 0.066 in 2018, against 4.582 ± 0.053 for ground-truth recordings. Its abstract puts it this way: “Our model achieves a mean opinion score (MOS) of 4.53 comparable to a MOS of 4.58 for professionally recorded speech”. Now look at what produced that number. “We randomly selected 100 fixed examples from the test set of our internal dataset as the evaluation set.” One hundred short clips. They came from the test set of a 24.6-hour single-speaker internal US English corpus. Each was rated by at least 8 raters, on a 1-to-5 scale in 0.5-point increments.

One hundred short clips, rated one at a time, is the instrument. It cannot see a phrase dropped in the middle of a 1,648-character passage, because it was never shown one. Alignment — the running map from text to time — is what drifts there. Nothing in the audio betrays it. The wrong words are delivered as convincingly as the right ones.

FastSpeech and Glow-TTS are two answers to a single question — where does the timing come from — and there are three places to get it. An attention-based acoustic model learns a soft text–audio alignment while it generates acoustic frames. That is the arrangement whose CER shot up at roughly 3× its maximum training length in the Harry Potter chapters. A duration-based non-autoregressive model predicts token durations and generates frames in parallel. That is how FastSpeech reached 0 repeats and 0 skips on the 50 hard sentences. An end-to-end latent model jointly learns alignment, acoustic representation and waveform generation in one trained system.

The third family has its own numbers. VITS, published in 2021, scored MOS 4.43 ± 0.06 on LJ Speech against 4.46 ± 0.06 for ground truth. That put it above Glow-TTS + HiFi-GAN fine-tuned at 4.32 ± 0.07 and Tacotron 2 + HiFi-GAN at 3.77 ± 0.08. What makes it a third family rather than a tuning of the second is where its durations come from. The consequence is directly observable: “While Glow-TTS generates only fixed-length utterances due to the deterministic duration predictor, samples from our model follow a similar length distribution to that of Tacotron 2.” Swap the stochastic duration predictor for the deterministic one used in Glow-TTS and the score falls to 4.39 ± 0.06. A small MOS cost, for a system that can no longer say the same sentence two different lengths.

They are three families, not three settings of one dial, and that changes what a passing test result means. Evidence that clears one of them leaves the others untested. 0 word errors on 50 hard sentences is a FastSpeech result. 4.43 on LJ Speech is a VITS result. Neither transfers.

Underneath all three families the job is the same. An acoustic model transforms linguistic conditions into intermediate acoustic or latent structure, and two distinct things happen inside it. Alignment determines which text element corresponds to which time region. Duration, pitch, energy and style controls shape rhythm and prosody. The long-form baseline lost the first and kept the second. That is exactly why it kept sounding like a 4.31.

Neither of the main answers is free, and both halves of the trade have been measured. Attention learns flexible alignment and can skip or repeat text. Duration predictors provide more explicit timing but may produce flat or incorrect prosody. Flatness here is a statistic, not a complaint. FastSpeech 2, in 2021, compared the pitch of synthesized speech with the recordings. FastSpeech's synthesized pitch had a kurtosis of −0.041 where ground truth had 0.977. FastSpeech 2's own came out at 0.996. The repair was more predictors rather than fewer: “The variance adaptor aims to add variance information (e.g., duration, pitch, energy, etc.) to the phoneme hidden sequence, which can provide enough information to predict variant speech for the one-to-many mapping problem in TTS.” On LJSpeech that bought MOS 3.83 ± 0.08 against FastSpeech's 3.68 ± 0.09. Ground truth sat at 4.30 ± 0.07, and FastSpeech's CMOS was −0.885.

The other dependency is quieter and just as measurable. FastSpeech 2 takes its phoneme durations from the Montreal Forced Aligner, not from a teacher model's attention map. So the timing a duration predictor learns is only as good as that aligner. The aligner's own paper put numbers on it in 2017. On the Buckeye corpus its word boundaries differed from hand annotation by 24.1 ms on average, against 26 ms reported inter-transcriber reliability. Its phone boundaries differed by 17.0 ms — identical to the 17 ms reported for human agreement. The averages are the flattering part. “While phone and word-level alignment is comparable to human annotators on average, the force-aligned boundaries do contain more medium-to-large alignment errors (>25 msec).” Only 68% of its differences fell under 25 ms, against 90% inter-transcriber agreement at a 26 ms tolerance. And 2–5% of tokens were off by at least 100 ms. So weigh the three families — attention, duration prediction, end-to-end latent — against the timing the product needs. Do it before the architecture is fixed.

Whichever family you pick, the work runs in four stages. Only the last of them can tell you the first three were wrong. Representing the linguistic sequence comes first — encoding phones, text, language, speaker and style conditions. It fixes the assumptions everything downstream inherits. Learning or supplying alignment then works from those assumptions without questioning them, whether through attention, monotonic alignment, forced durations from the Montreal Forced Aligner, or latent variables. Predicting prosodic structure estimates duration, pitch, energy, rhythm and spectral representation on top of that. Validating content and control is the stage that finds out whether any of it held: coverage, repetitions, timing, style consistency, long-form behaviour.

That is the stage an in-domain listening panel never really runs. The three Google mechanisms passed the panel identically at about 4.3. They were told apart only when 1,034 book utterances were back-transcribed and scored. A naturalness score is not the fourth stage. Character error rate against the target transcript is.

That argument has been carried by four terms, and in a review meeting they blur together. They are not interchangeable. Each implies a different measurement, and a different owner when the timing goes wrong.

Long-form narration is only one of the demands a synthesizer gets asked to meet. The other three want the timing to behave differently. What follows is narrow but firm. A coverage result earned on 1,034 book utterances is not a result for a conversational agent, and a latency result is not a result for a book. Each of the four wants its own units and its own demonstration. The studies this lesson is built on supply them.

Since those four products ask different questions, no single number closes a release. The number most releases lean on says as much in its own defining document. Mean opinion score is specified by ITU-T Recommendation P.800, approved in 1996 and still in force: “The quantity evaluated from the scores (mean listening-quality opinion score, or simply mean opinion score) is represented by the symbol MOS.” Its Absolute Category Rating listening-quality scale runs Excellent 5, Good 4, Fair 3, Poor 2, Bad 1. And the material it was written for is named exactly: “Category ratings are applied to short groups of unrelated sentences, each of which has been passed through a number of standard processes as well as the processes under test.” Short groups of unrelated sentences. Long-form content integrity is not something the instrument fails to detect by accident. It is outside what the instrument was ever built to measure.

The two dimensions come apart most clearly in a paper counting its own errors. On a separate 100-sentence custom test set, the Tacotron 2 authors went through the output by hand: “In a manual analysis of the error modes of our system, counting errors in each category independently, 0 sentences contained repeated words, 6 contained mispronunciations, 1 contained skipped words, and 23 were subjectively decided to contain unnatural prosody, such as emphasis on the wrong syllables or words, or unnatural pitch.” One skipped word. Twenty-three unnatural ones. The same system scored a MOS of 4.354 on that set. That single figure covers both columns without distinguishing them. The same section records where the remaining failure sat: “End-point prediction failed in a single case, on the input sentence containing the most characters.”

So text coverage and repetition error is the number that catches a dropped phrase. Duration, pitch and energy control accuracy is the number that catches a wooden reading. That is the failure FastSpeech traded for stable alignment, and the one FastSpeech 2 measured as a pitch kurtosis of −0.041 against 0.977. Two dimensions, not two views of one. Around them sit the difficult slices and long-form stability across languages and speakers. That is what turns a 4.31 shared by three mechanisms into a comparison that actually separates them.

Diagnosis here means producing an account another team can check against the audio, phrase by phrase, for text that was skipped or said twice. The first step is not a matter of taste. It has a published protocol, with a size, a composition and results for three systems: “To evaluate the robustness of FastSpeech, we select 50 sentences which are particularly hard for TTS system.” All 50 are listed in an appendix, and a footnote says what they are: “These cases include single letters, spellings, repeated numbers, and long sentences.” Scored per sentence, with each kind of word error counted at most once, that set returned 12 error sentences for Tacotron 2 (24%), 17 for Transformer TTS (34%) and 0 for FastSpeech.

So build the stress set to that recipe: single letters, spellings, repeated numbers, long clauses, rare names, dense punctuation, code-switching. Then add the length axis from the Harry Potter test, out to 1,648 characters. Inspect alignment maps or predicted durations for stalls, jumps, excessive compression and repeated regions. Compare model families with voice data and vocoder held fixed where possible — an in-domain MOS of about 4.3 was identical across all three mechanisms and told you nothing. Then add content checks. Back-transcribe with an ASR system, as Google did with Google Cloud Speech-To-Text, and score coverage against the target text. Verify the critical tokens by hand. That is how Tacotron 2's authors arrived at 0 repeats, 1 skip, 6 mispronunciations and 23 prosody failures rather than at a single number.