Skip to content
AI.info

Research

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

Almost every strong TTS system today reaches the final audio through a chain of separately trained parts: a language model over codec tokens, or a Mel-spectrogram model plus a neural vocoder, or a…

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
arXiv
2611.12527
Published
2026-11-09
Authors
Sang-Hoon Lee, Ha-Yeong Choi

AI summary

Overview

Research area: Text-to-speech synthesis (cs.SD), specifically diffusion / conditional flow matching (CFM) generative models for audio, the shift away from multi-stage TTS pipelines, and zero-shot voice cloning.

Technical level: Advanced. The paper assumes familiarity with flow matching, diffusion transformers (DiT), Mel-spectrograms, neural vocoders, and TTS evaluation metrics (WER, SIM, DNSMOS, UTMOS).

Scope: WaveGen is a single-model, single-stage text-to-raw-waveform TTS system that eliminates every external module used by modern pipelines (codecs, vocoders, VAE latents, duration predictors, BERT/Wav2Vec representations, GAN training, REPA) by introducing "Spectral Forcing" — jointly learning an internal 64-bin Mel spectral trajectory alongside the target 24,000 Hz waveform trajectory.

What This Paper Is About

Almost every strong TTS system today reaches the final audio through a chain of separately trained parts: a language model over codec tokens, or a Mel-spectrogram model plus a neural vocoder, or a latent diffusion model plus a decoder. Each extra stage adds training complexity and caps the achievable quality. WaveGen asks whether a single network can go straight from text to a 24 kHz waveform and still match or beat those pipelines, including in zero-shot speaker cloning, without adversarial training or any pre-trained external representation.

Key Contributions

  1. WaveGen, a fully end-to-end, single-stage, zero-shot text-to-waveform diffusion framework. Table 1 positions it as the only compared system that is simultaneously end-to-end (T→W), single-stage, and zero-shot, while depending on no codec, vocoder, VAE/latent model, GAN training, duration predictor, or pre-trained LLM.
  2. Spectral Forcing, a method that jointly learns an internal low-resolution spectral trajectory and the target waveform trajectory inside the same model, so the spectral signal acts only as internal frequency-rich guidance rather than an externally decoded intermediate.
  3. A decoupled prediction/loss design — x-prediction (clean-target prediction) paired with v-loss (velocity-space objective) — plus a multi-scale energy-normalized spectral v-loss and an architecture with shared DiT blocks and disentangled DiT heads, 14 shared blocks and 2 disentangled heads in the baseline.
  4. Scalability evidence for end-to-end TTS: models scaled from 0.1B to 0.6B parameters, showing gains from model and data scale under one architecture, open-source data, and one-stage training.

Main Findings

  • Zero-shot results on Seed-en: WaveGen (0.6B, 100k hours) reaches a WER of 1.47 and a SIM of 0.75, versus a ground-truth reference of 2.14 WER and 0.73 SIM in the same table. WaveGen-S (0.1B, 120k) scores 4.42 WER / 0.53 SIM and WaveGen-B (0.4B, 120k) 1.77 / 0.63.
  • Competitive against much larger or multi-stage baselines: in Table 2, Qwen3-TTS at 1.7B reports 1.50 WER / 0.71 SIM and at 0.6B 1.68 / 0.70; F5-TTS (0.3B) 2.00 / 0.67; ZipVoice (0.1B) 1.70 / 0.69; CosyVoice3-RL (0.5B) 1.68 / 0.69.
  • Human evaluation holds up: on MTurk, WaveGen-B gets nMOS 4.000 ± 0.126 (GT 4.117 ± 0.112) and sMOS 3.988 ± 0.037 (GT 3.932 ± 0.038), comparable to F5-TTS, CosyVoice3, and Qwen3-TTS at 0.6B and 1.7B.
  • Spectral Forcing is essential: without it, training fails to align (WER 123.67, SIM 0.19). Linear spectrogram targets at 481, 128, and 64 bins all produce NaN. Mel bins trade off alignment against reconstruction: 256 bins gives 126.50 WER, 128 gives 18.67, 80 gives 6.80, 64 gives 4.96 (chosen), and 32 gives 4.26.
  • End-to-end beats the two-stage equivalent: at matched total NFE of 32, the 407M end-to-end model gets 2.91 WER / 0.53 SIM / 3.67 DNSMOS versus 4.39 / 0.46 / 3.53 for a 226M two-stage (text-to-Mel plus vocoder) system built from the same DiT architecture. Notably, a vocoder trained without GANs on ground-truth Mel reaches 1.47 WER / 0.70 SIM, showing the cascade is limited by the intermediate representation and error propagation.
  • x-prediction matters for perceptual quality: with Spectral Forcing, v-prediction gives slightly better alignment (3.93 WER / 0.48 SIM) but collapses in perceptual quality (DNSMOS 2.94), while x-prediction gives 5.57 / 0.46 / 3.45. The paper frames this as evidence that good waveform quality is achievable without adversarial training; it also warns that an x-loss space introduces significant metallic noise.
  • Multi-scale spectral loss helps, but only combined: the Flow2GAN-style spectral v-loss alone degrades to 68.82 WER; the multi-scale term alone gives 16.98 WER / 0.51 SIM / 3.57 DNSMOS; the final combination of waveform v-loss plus the multi-scale spectral term gives 4.96 / 0.47 / 3.72.
  • Both model and data scale help: WaveGen-S goes from 4.96 to 4.27 WER when data grows from 0.5k to 120k hours, and WaveGen-B from 2.91 to 2.37 (2.37 / 0.59 / 3.60 with basic CFG; 1.77 / 0.63 / 3.60 with the modified CFG).
  • Sampling steps: at NFE 4 the model collapses (52.0 WER); at 8 it reaches 4.85, at 16 2.16, at 32 1.77, and beyond that gains flatten (1.87 at 50, 1.84 at 100). The paper attributes early steps to style/alignment adaptation and later steps to waveform refinement.
  • Multilingual (Emilia en/zh, 680M params, 100k hours): with prompt anchoring, seed-en gives 1.47 WER / 0.75 SIM / 3.68 UTMOS, and seed-zh gives 1.75 CER / 0.80 SIM / 2.87 UTMOS. The authors state this is the first CFM-based end-to-end waveform TTS model to outperform conventional multi-stage systems on Emilia, and the best among waveform diffusion TTS models evaluated, including WavTTS (1.50 / 0.65 / 3.92 en) and BareWave (1.75 / 0.60 / 3.72 en).
  • Prompt anchoring matters: removing it raises seed-en WER from 1.47 to 1.55 and seed-zh CER from 1.66 to 1.75.
  • Data quality limits high frequencies: because Emilia is MP3-compressed, models trained on it struggle above 8 kHz, whereas LibriTTS-trained models generate richer high-frequency content — an argument for curating data specifically for end-to-end TTS.
  • REPA is optional and helps: a small-scale LibriTTS study with Whisper-small features on the 10th layer (λ_repa = 0.1) improves WER 4.96 → 3.54 and SIM 0.47 → 0.54 with DNSMOS roughly unchanged (3.72 → 3.70); the released models do not use it.

Methodology in Plain English

A raw 24 kHz waveform is enormously long and detailed, so standard diffusion struggles to align it with text. WaveGen sidesteps this by reshaping the high-resolution waveform into a low-resolution (100 Hz) representation for efficient processing, then training one transformer (DiT with AdaLN-SOLA) on two things at once: the noisy waveform and a noisy 64-bin Mel-spectrogram of the same speech. Text arrives as average-pooled embeddings expanded along time (following ZipVoice), so no separate duration predictor is needed — alignment is learned inside the model with masked infilling, where prompt speech and text are randomly dropped during training (0.3 and 0.2 dropout).

The model is a stack of 14 shared DiT blocks that serve as a common alignment backbone, split into two shallow, representation-specific heads: one predicts the clean waveform, the other the clean spectrum. Training uses x-prediction, but the error is measured in velocity space — the predicted clean signal is converted into a velocity and compared against the true one, only over masked regions. On top of that, an extra loss compares the difference between predicted and true waveform (scaled by 1−t) across six spectral resolutions (512 down to 16 filterbank bins, with matching FFT/window/hop sizes), normalizing by ground-truth spectral energy so loud regions do not dominate. The two auxiliary weights are both 0.1.

At sampling time the Euler ODE solver runs with NFE 32 and CFG 2; the text condition is dropped only for t < 0.02, because a looser threshold of t < 0.5 badly hurt speaker similarity. Prompt regions are anchored to a ground-truth-derived intermediate state each step to fix the train-inference mismatch that masked infilling creates. A small information-bottleneck pre-net (two MLP layers, bottleneck dimension 32, 8n Mel bins, dropout 0.2) prevents the model from lazily reconstructing audio from the spectral guidance instead of learning text alignment.

Why This Matters

Impact on research. The paper challenges the field's convergence on ever-more-complex cascades. If one model, one training stage, and open-source data can hit 1.47 WER / 0.75 SIM on Seed-en without codecs, vocoders, GANs, duration predictors, or BERT/Wav2Vec features, then several assumed-necessary components become optional. The specific findings — that x-prediction with v-loss is the right pairing for raw waveforms, that an information bottleneck in the auxiliary spectral target is necessary, and that 64 Mel bins is the sweet spot — are directly reusable design rules for anyone doing high-resolution diffusion.

Real-world applications:

  • Zero-shot voice cloning and personalized TTS for accessibility and assistive reading tools, where a short prompt sample is enough.
  • Dubbing, localization, and audiobook production, where a simpler single-stage pipeline lowers engineering and inference cost.
  • Low-resource and multilingual speech synthesis, since the method needs only open Emilia data and no pre-trained self-supervised encoders.
  • Streaming or on-device synthesis research, given the reported RTF of 0.24 at NFE 32 and 0.12–0.08 for smaller models on an A100.

Industry relevance. The removal of the vocoder, codec, and duration predictor collapses the number of artifacts a production team must train, version, and deploy. The paper's honest reporting of weaknesses — high-frequency loss from compressed training data, sensitivity to the CFG schedule, and only four epochs of training due to compute limits — gives practitioners realistic expectations about where the approach still needs work.

Future Directions

  • Training longer and with representation alignment. The authors explicitly note that limited compute forced only four epochs (11 for the 0.6B model) and that both extended training and the optional REPA results (WER 4.96 → 3.54) suggest further gains are available.
  • Audio data curation for end-to-end TTS. The finding that MP3-compressed Emilia limits generation above 8 kHz points to the need for high-fidelity, large-scale corpora, or methods that recover high-frequency detail from compressed training audio.
  • Better x-prediction objectives. Since v-prediction aligns better but sounds worse, and x-loss introduces metallic noise, a loss space that gets both alignment and clean energy modeling remains open.
  • Tuning the Spectral Forcing strength. The information-bottleneck dropout ablations (SF drop of 0, 0.2, and 0.5 trading WER against UTMOS and SIM) show a live trade-off between alignment speed and audio quality that is not yet resolved.

Target Audience

Researchers and advanced engineers working on speech synthesis, diffusion/flow-matching generative models, or audio generation who already understand Mel-spectrograms, vocoders, and zero-shot TTS evaluation. It is most valuable to those deciding whether to build a multi-stage TTS stack or attempt direct waveform generation, and to anyone studying prediction/loss space choices in high-dimensional diffusion.

Authors’ abstract

This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based…

The full abstract on arXiv
Read the original paper