# WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

AI.info

> Almost every strong TTS system today reaches the final audio through a chain of separately trained parts: a language model over codec tokens, or a Mel-spectrogram model plus a neural vocoder, or a…

**arXiv**: 2611.12527

**Published**: 2026-11-09

**Authors**: Sang-Hoon Lee, Ha-Yeong Choi

## AI summary

## Overview

**Research area:** Text-to-speech synthesis (cs.SD), specifically diffusion / conditional flow matching (CFM) generative models for audio, the shift away from multi-stage TTS pipelines, and zero-shot voice cloning.

**Technical level:** Advanced. The paper assumes familiarity with flow matching, diffusion transformers (DiT), Mel-spectrograms, neural vocoders, and TTS evaluation metrics (WER, SIM, DNSMOS, UTMOS).

**Scope:** WaveGen is a single-model, single-stage text-to-raw-waveform TTS system that eliminates every external module used by modern pipelines (codecs, vocoders, VAE latents, duration predictors, BERT/Wav2Vec representations, GAN training, REPA) by introducing "Spectral Forcing" — jointly learning an internal 64-bin Mel spectral trajectory alongside the target 24,000 Hz waveform trajectory.

## What This Paper Is About

Almost every strong TTS system today reaches the final audio through a chain of separately trained parts: a language model over codec tokens, or a Mel-spectrogram model plus a neural vocoder, or a latent diffusion model plus a decoder. Each extra stage adds training complexity and caps the achievable quality. WaveGen asks whether a *single* network can go straight from text to a 24 kHz waveform and still match or beat those pipelines, including in zero-shot speaker cloning, without adversarial training or any pre-trained external representation.

## Key Contributions

1. **WaveGen**, a fully end-to-end, single-stage, zero-shot text-to-waveform diffusion framework. Table 1 positions it as the only compared system that is simultaneously end-to-end (T→W), single-stage, and zero-shot, while depending on no codec, vocoder, VAE/latent model, GAN training, duration predictor, or pre-trained LLM.
2. **Spectral Forcing**, a method that jointly learns an internal low-resolution spectral trajectory and the target waveform trajectory inside the same model, so the spectral signal acts only as internal frequency-rich guidance rather than an externally decoded intermediate.
3. **A decoupled prediction/loss design** — `x`-prediction (clean-target prediction) paired with `v`-loss (velocity-space objective) — plus a **multi-scale energy-normalized spectral `v`-loss** and an architecture with shared DiT blocks and **disentangled DiT heads**, 14 shared blocks and 2 disentangled heads in the baseline.
4. **Scalability evidence** for end-to-end TTS: models scaled from 0.1B to 0.6B parameters, showing gains from model and data scale under one architecture, open-source data, and one-stage training.

## Main Findings

- **Zero-shot results on Seed-en:** WaveGen (0.6B, 100k hours) reaches a WER of **1.47** and a SIM of **0.75**, versus a ground-truth reference of 2.14 WER and 0.73 SIM in the same table. WaveGen-S (0.1B, 120k) scores 4.42 WER / 0.53 SIM and WaveGen-B (0.4B, 120k) 1.77 / 0.63.
- **Competitive against much larger or multi-stage baselines:** in Table 2, Qwen3-TTS at 1.7B reports 1.50 WER / 0.71 SIM and at 0.6B 1.68 / 0.70; F5-TTS (0.3B) 2.00 / 0.67; ZipVoice (0.1B) 1.70 / 0.69; CosyVoice3-RL (0.5B) 1.68 / 0.69.
- **Human evaluation holds up:** on MTurk, WaveGen-B gets nMOS 4.000 ± 0.126 (GT 4.117 ± 0.112) and sMOS 3.988 ± 0.037 (GT 3.932 ± 0.038), comparable to F5-TTS, CosyVoice3, and Qwen3-TTS at 0.6B and 1.7B.
- **Spectral Forcing is essential:** without it, training fails to align (WER 123.67, SIM 0.19). Linear spectrogram targets at 481, 128, and 64 bins all produce NaN. Mel bins trade off alignment against reconstruction: 256 bins gives 126.50 WER, 128 gives 18.67, 80 gives 6.80, 64 gives 4.96 (chosen), and 32 gives 4.26.
- **End-to-end beats the two-stage equivalent:** at matched total NFE of 32, the 407M end-to-end model gets 2.91 WER / 0.53 SIM / 3.67 DNSMOS versus 4.39 / 0.46 / 3.53 for a 226M two-stage (text-to-Mel plus vocoder) system built from the same DiT architecture. Notably, a vocoder trained without GANs on ground-truth Mel reaches 1.47 WER / 0.70 SIM, showing the cascade is limited by the intermediate representation and error propagation.
- **`x`-prediction matters for perceptual quality:** with Spectral Forcing, `v`-prediction gives slightly better alignment (3.93 WER / 0.48 SIM) but collapses in perceptual quality (DNSMOS 2.94), while `x`-prediction gives 5.57 / 0.46 / 3.45. The paper frames this as evidence that good waveform quality is achievable without adversarial training; it also warns that an `x`-loss space introduces significant metallic noise.
- **Multi-scale spectral loss helps, but only combined:** the Flow2GAN-style spectral `v`-loss alone degrades to 68.82 WER; the multi-scale term alone gives 16.98 WER / 0.51 SIM / 3.57 DNSMOS; the final combination of waveform `v`-loss plus the multi-scale spectral term gives 4.96 / 0.47 / 3.72.
- **Both model and data scale help:** WaveGen-S goes from 4.96 to 4.27 WER when data grows from 0.5k to 120k hours, and WaveGen-B from 2.91 to 2.37 (2.37 / 0.59 / 3.60 with basic CFG; 1.77 / 0.63 / 3.60 with the modified CFG).
- **Sampling steps:** at NFE 4 the model collapses (52.0 WER); at 8 it reaches 4.85, at 16 2.16, at 32 1.77, and beyond that gains flatten (1.87 at 50, 1.84 at 100). The paper attributes early steps to style/alignment adaptation and later steps to waveform refinement.
- **Multilingual (Emilia en/zh, 680M params, 100k hours):** with prompt anchoring, seed-en gives 1.47 WER / 0.75 SIM / 3.68 UTMOS, and seed-zh gives 1.75 CER / 0.80 SIM / 2.87 UTMOS. The authors state this is the first CFM-based end-to-end waveform TTS model to outperform conventional multi-stage systems on Emilia, and the best among waveform diffusion TTS models evaluated, including WavTTS (1.50 / 0.65 / 3.92 en) and BareWave (1.75 / 0.60 / 3.72 en).
- **Prompt anchoring matters:** removing it raises seed-en WER from 1.47 to 1.55 and seed-zh CER from 1.66 to 1.75.
- **Data quality limits high frequencies:** because Emilia is MP3-compressed, models trained on it struggle above 8 kHz, whereas LibriTTS-trained models generate richer high-frequency content — an argument for curating data specifically for end-to-end TTS.
- **REPA is optional and helps:** a small-scale LibriTTS study with Whisper-small features on the 10th layer (λ_repa = 0.1) improves WER 4.96 → 3.54 and SIM 0.47 → 0.54 with DNSMOS roughly unchanged (3.72 → 3.70); the released models do not use it.

## Methodology in Plain English

A raw 24 kHz waveform is enormously long and detailed, so standard diffusion struggles to align it with text. WaveGen sidesteps this by reshaping the high-resolution waveform into a low-resolution (100 Hz) representation for efficient processing, then training one transformer (DiT with AdaLN-SOLA) on two things at once: the noisy waveform and a noisy 64-bin Mel-spectrogram of the same speech. Text arrives as average-pooled embeddings expanded along time (following ZipVoice), so no separate duration predictor is needed — alignment is learned inside the model with masked infilling, where prompt speech and text are randomly dropped during training (0.3 and 0.2 dropout).

The model is a stack of 14 shared DiT blocks that serve as a common alignment backbone, split into two shallow, representation-specific heads: one predicts the clean waveform, the other the clean spectrum. Training uses `x`-prediction, but the error is measured in velocity space — the predicted clean signal is converted into a velocity and compared against the true one, only over masked regions. On top of that, an extra loss compares the *difference* between predicted and true waveform (scaled by 1−t) across six spectral resolutions (512 down to 16 filterbank bins, with matching FFT/window/hop sizes), normalizing by ground-truth spectral energy so loud regions do not dominate. The two auxiliary weights are both 0.1.

At sampling time the Euler ODE solver runs with NFE 32 and CFG 2; the text condition is dropped only for t < 0.02, because a looser threshold of t < 0.5 badly hurt speaker similarity. Prompt regions are anchored to a ground-truth-derived intermediate state each step to fix the train-inference mismatch that masked infilling creates. A small information-bottleneck pre-net (two MLP layers, bottleneck dimension 32, 8n Mel bins, dropout 0.2) prevents the model from lazily reconstructing audio from the spectral guidance instead of learning text alignment.

## Why This Matters

**Impact on research.** The paper challenges the field's convergence on ever-more-complex cascades. If one model, one training stage, and open-source data can hit 1.47 WER / 0.75 SIM on Seed-en without codecs, vocoders, GANs, duration predictors, or BERT/Wav2Vec features, then several assumed-necessary components become optional. The specific findings — that `x`-prediction with `v`-loss is the right pairing for raw waveforms, that an information bottleneck in the auxiliary spectral target is necessary, and that 64 Mel bins is the sweet spot — are directly reusable design rules for anyone doing high-resolution diffusion.

**Real-world applications:**

- Zero-shot voice cloning and personalized TTS for accessibility and assistive reading tools, where a short prompt sample is enough.
- Dubbing, localization, and audiobook production, where a simpler single-stage pipeline lowers engineering and inference cost.
- Low-resource and multilingual speech synthesis, since the method needs only open Emilia data and no pre-trained self-supervised encoders.
- Streaming or on-device synthesis research, given the reported RTF of 0.24 at NFE 32 and 0.12–0.08 for smaller models on an A100.

**Industry relevance.** The removal of the vocoder, codec, and duration predictor collapses the number of artifacts a production team must train, version, and deploy. The paper's honest reporting of weaknesses — high-frequency loss from compressed training data, sensitivity to the CFG schedule, and only four epochs of training due to compute limits — gives practitioners realistic expectations about where the approach still needs work.

## Future Directions

- **Training longer and with representation alignment.** The authors explicitly note that limited compute forced only four epochs (11 for the 0.6B model) and that both extended training and the optional REPA results (WER 4.96 → 3.54) suggest further gains are available.
- **Audio data curation for end-to-end TTS.** The finding that MP3-compressed Emilia limits generation above 8 kHz points to the need for high-fidelity, large-scale corpora, or methods that recover high-frequency detail from compressed training audio.
- **Better `x`-prediction objectives.** Since `v`-prediction aligns better but sounds worse, and `x`-loss introduces metallic noise, a loss space that gets both alignment and clean energy modeling remains open.
- **Tuning the Spectral Forcing strength.** The information-bottleneck dropout ablations (SF drop of 0, 0.2, and 0.5 trading WER against UTMOS and SIM) show a live trade-off between alignment speed and audio quality that is not yet resolved.

## Target Audience

Researchers and advanced engineers working on speech synthesis, diffusion/flow-matching generative models, or audio generation who already understand Mel-spectrograms, vocoders, and zero-shot TTS evaluation. It is most valuable to those deciding whether to build a multi-stage TTS stack or attempt direct waveform generation, and to anyone studying prediction/loss space choices in high-dimensional diffusion.

## Authors’ abstract

This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based…

[The full abstract on arXiv](https://arxiv.org/abs/2611.12527)

[Read the original paper](https://arxiv.org/abs/2611.12527)

---

Canonical: https://ai.info/paper/wavegen-truly-end-to-end-waveform-generation-via-internal-spectral-2611.12527
