Neural Vocoders and End-to-End Speech Synthesis

A neural vocoder turns acoustic or latent conditioning into a waveform. Mel spectrograms are one such conditioning. The architectures on offer trade against each other in familiar ways: autoregressive fidelity, parallel speed, adversarial sharpness, invertibility, diffusion quality, hardware efficiency. Choosing between them is a real decision. It is not the decision that determines whether the audio is any good.

No vocoder repairs missing linguistic content or an incorrect alignment. Both arrive broken and leave broken. What a vocoder does add, on its own account, is periodic noise, buzzy harmonics, phase artifacts, speaker changes, and outright failure on conditioning outside its training range. So the constraint is not how sophisticated the vocoder is. It is whether the conditioning interface preserves what the waveform stage will be judged on. A listening test can only report what that interface allowed through.

That constraint is not a metaphor. It has a price, and the paper that defined the modern mel-conditioned pipeline paid it in public. In 2018 Tacotron 2 took a WaveNet vocoder trained on ground-truth mel spectrograms and made it synthesize from predicted ones. It scored 4.362 MOS. The same vocoder, trained on the features it would actually receive, scored 4.526. The number is in the paper. Where the number came from is what a naturalness test would never have told you.

A single naturalness score is a poor instrument for diagnosis. It reports what got through and nothing about what did not. Fidelity, speed and hardware efficiency are not traded once and for all. They are traded inside four choices: measurement, modeling, decision, verification. One number folds all four into a verdict, with no way of saying which of them went wrong. The path below separates them. Define the conditioning interface: feature scale, frame rate, normalization, speaker and style signals. Select a generator family: autoregressive, GAN, flow, diffusion, integrated latent synthesis. Train with matched data. Then evaluate the waveform for fidelity, naturalness, artifacts, latency, robustness and downstream intelligibility.

The choice that decides the rest is the first one, and the first one is what the vocoder is trained on. Three rows of one table make the whole argument. A WaveNet vocoder trained on ground-truth mel spectrograms and made to synthesize from predicted ones scored MOS 4.362 +/- 0.066. Trained on the predicted features it would actually receive, it scored 4.526 +/- 0.066. Fully matched on ground truth, 4.522 +/- 0.055. The two matched conditions sit together. The mismatched one is the outlier. That is what step three means in practice. Align acoustic-model outputs and natural conditioning, so that the other three choices are at least being made about the system you intend to ship.

Failure on conditioning outside the training range sounds like a footnote until you meet one in print. Tacotron 2 was evaluated on 37 news headlines drawn from outside the training domain. It scored MOS 4.148 +/- 0.124. A WaveNet conditioned on linguistic features scored 4.137 +/- 0.128. A virtual tie. In the main evaluation of the same paper that comparison had not been close at all: 4.526 +/- 0.066 against 4.341 +/- 0.051. The margin did not shrink because the vocoder got worse. It shrank because the material moved outside what the system had been trained to cover. Rater comments said where. Although the neural system sounded more natural, it ran into pronunciation difficulties on names.

The authors drew the conclusion themselves rather than leaving it to a reader: “This result points to a challenge for end-to-end approaches – they require training on data that cover intended usage.”

Nothing in a headline MOS of 4.148 says so on its own. A listening test run on material inside the training range cannot show a reviewer that the range was ever the question. Every sentence played was inside it. The rest of this lesson is about that gap: between a system that scores well on everything anyone thought to play it, and a system that works.

Setting 167.9 times real time beside a virtual tie on 37 news headlines, as the last two sections just did, is ordinary practice and slightly dishonest. Those numbers come from different conversations. Vocoder, conditioning mismatch, adversarial loss and real-time factor belong to modelling, training and operations. They still end up in one sentence. Each carries its own evidence, its own unit, and its own owner. A claim settled in one of these conversations is not settled in the other two.

Owners change with deployment too. Cloud TTS, on-device synthesis, voice conversion, and singing constrain the waveform generator in four incompatible ways. A generator that satisfies one of them can be the wrong choice for the next. Each is scored in units the out-of-domain headline test never touched, and what would prove any of them is not what would have caught that failure.

Whichever of those four settings you are in, one test applies. It exists so that another team can check whether the vocoder was only ever trained on features it will not receive in production. Prepare two conditioning sets, natural features and acoustic-model predictions, and run them side by side. That pairing is the mismatch table from the second section, turned into a procedure. Then test the difficult regimes. BigVGAN's out-of-distribution list is a ready-made one: unseen speakers, unseen languages, unseen recording environments, singing voices, music and instrumental audio. Names inside ordinary sentences belong on it too. Names are what the 37 headlines broke on. Then profile hardware, batch and streaming, on the target device, because 167.9 and 1.43 are the same model. Then catalogue artifacts: named examples of buzzing, dropouts, metallic tones, and instability.

Record three things as you go. What defining the conditioning interface assumes — which features, from which model, in which regimes. One counterexample. And what evaluating the waveform triggers once that counterexample has been played. The three notes matter more than the score attached to them. The score is the thing that failed to notice last time.

That is the honest reading of a naturalness score. Naturalness and artifact-specific listening tests will not, on their own, reveal that a vocoder was trained on one kind of feature and deployed on another. On 37 out-of-domain headlines the score stayed at 4.148 +/- 0.124 while the pronunciation of names fell apart. They belong in the report. They cannot be the whole of it.

So report the unit of each score and who listened. The BVCC listening-test dataset, reported in 2021, shows what that looks like when it is done properly: 187 systems including natural speech, 38 samples per system, a 5-grade MOS scale, and a stated listener design. “Each set was rated by eight different listeners, and overall, 304 different listeners participated in our test.” The agreement among those listeners was only moderate. Krippendorff's alpha and intra-class correlation both came to 0.50. That is how much weight one naturalness number can carry.

The operational side has the same discipline. HiFi-GAN's own abstract demonstrates it by quoting two speeds at once: “A subjective human evaluation (mean opinion score, MOS) of a single speaker dataset indicates that our proposed method demonstrates similarity to human quality while generating 22.05 kHz high-fidelity audio 167.9 times faster than real-time on a single V100 GPU. We further show the generality of HiFi-GAN to the mel-spectrogram inversion of unseen speakers and end-to-end speech synthesis. Finally, a small footprint version of HiFi-GAN generates samples 13.4 times faster than real-time on CPU with comparable quality to an autoregressive counterpart.” The paper's own table separates what the abstract merges. HiFi-GAN V1 runs at x167.86 on a V100 GPU. The same model runs at x1.43 on CPU, a MacBook Pro with an Intel i7 at 2.6 GHz. The small V3 runs at x13.44 on that CPU and x1,186.80 on the same GPU. For one fixed model, the machine alone moves the real-time factor by roughly a factor of 117.