From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation

AI summary

Overview

• Research area: Speech-driven 3D facial animation, discrete speech representation learning, and audio-visual text-to-speech.
• Technical level: Advanced. The paper assumes familiarity with self-supervised speech models (HuBERT), neural audio codecs, vector quantization, and TTS pipelines.
• Scope: A systematic comparison of four speech representation families — semantic, semantic+acoustic, acoustic, and label-based — across two facial decoders for 3D facial animation, supplemented by phonetic/articulatory probing and a proof-of-concept unified Audio Visual Text-to-Speech (AVTTS) pipeline.

What This Paper Is About

Most speech-driven 3D facial animation systems feed some kind of speech representation into a temporal decoder, but it is unclear which properties of that representation actually matter for producing accurate, natural facial motion. This paper asks whether representations that emphasize phonetic class information, acoustic reconstruction, or supervised labels are better suited to driving 3D faces, and whether a single tokenized representation can generate speech and facial motion together from text.

Key Contributions

1. A systematic comparison of four speech encoders and two facial decoders. The authors evaluate HuBERT (semantic), SpeechTokenizer (semantic+acoustic), WavTokenizer (acoustic), and CosyVoice2 (label-based) representations, each paired with either a Gated Recurrent Unit (GRU) or a Transformer decoder, yielding eight model variants trained on BEAT2.
2. Objective and perceptual evaluation. Performance is measured with Lips Vertex Error (LVE, adapted for blendshapes), a Jitter Score, and a proposed Bilabial Closure Score (BCS), alongside a MUSHRA-like perceptual study with 30 participants.
3. Probing analyses linking tokens to phonetic and articulatory units. Tokenized representations are related to phonetic classes and to 32 k-means viseme clusters derived from ground-truth blendshapes, using normalized entropy and a Ridge regression probe reporting median R² across blendshapes.
4. A proof-of-concept AVTTS pipeline. Because CosyVoice2 already produces speech tokens as an intermediate step, the authors reuse their facial Transformer decoder on those same tokens to generate synchronized speech audio and 3D facial motion directly from text, removing the conventional two-stage (TTS then animation) paradigm.

Main Findings

• Semantic and label-based representations both work well. The semantic baseline (HuBERT + GRU, reproducing FaceDiffuser) and the label-based variant (CosyVoice2 + Transformer) achieve comparable facial animation quality, with no statistically significant difference between them perceptually. The paper's abstract states that encoding phonetic classes is beneficial for accurate facial animation prediction on both semantic and label-based representations.
• Reconstruction accuracy favors HuBERT, slightly. On LVE (lower is better), [HB+GRU] and [HB+T.] both score 0.26, the best values in Table 1, while [CV2+T.] is close behind at 0.28. The acoustic representation is worst on LVE with a GRU (0.84) and mid-range with a Transformer (0.33).
• Discrete representations benefit more from the Transformer decoder. Models with discrete speech representations perform significantly better with the Transformer than with the GRU, whereas the continuous HuBERT representation shows equivalent results between the two decoders. For Jitter, all representations decoded with a Transformer outperform their GRU counterparts (45.5 vs. 80.3, 35.2 vs. 74.4, 43.6 vs. 93.3, 50.3 vs. 76.5).
• The baseline wins on lip closure. On BCS (higher is better), [HB+GRU] scores 57.5, followed by [CV2+T.] at 47.0, then [HB+T.] at 27.6. All other variants show very low values — 3.4, 2.3, 0.4, and 6.2 — indicating an absence of expected lip closure during bilabial phonemes.
• Perceptual results track BCS, not LVE. The three perceptually evaluated conditions form three groups: the reference rated significantly higher, then [HB+GRU] and [CV2+T.] with no significant difference between them, then [HB+T.]. This ordering matches BCS performance but not LVE, suggesting that for this work lip-closing metrics correlate more closely with human perception than traditional reconstruction metrics.
• Semantic+acoustic distillation encodes phonetics best but animates worst. SpeechTokenizer achieves the lowest normalized entropy for phonetic classes (H(P|t) = 39.6%), ahead of semantic HuBERT (44.4%), label-based CosyVoice2 (57.7%), and acoustic WavTokenizer (73.2%). Yet its facial animation results are comparable only with the acoustic representation, suggesting that low-structured acoustic information may be detrimental to predicting facial animation.
• Viseme information is weakly encoded everywhere. Normalized viseme entropy is high across all four representations (91.5%, 90.4%, 91.4%, 84.3%), with the label-based representation standing out as the lowest, possibly favored by its fully supervised discrete output training task.
• Only HuBERT carries a strong continuous facial signal. The median R² from the Ridge regression probe is 0.25 for HuBERT versus 0.04 (SpeechTokenizer), 0.08 (WavTokenizer), and 0.10 (CosyVoice2), consistent with the continuous-versus-discrete nature of the representations.
• Phonetic information appears necessary but not sufficient. The authors conclude that phonetic class information may be a necessary condition for accurate facial animation prediction, provided low-structured acoustic information is absent.
• The unified AVTTS pipeline shows promise but less lip contrast. The demo page displays encouraging results, though the rendered sequences in Figure 2 show less contrast of lip movement across phonemes than the audio-driven [CV2+T.] version.

Methodology in Plain English

The researchers took one standard facial animation architecture (FaceDiffuser, based on HuBERT features and a GRU) as a fixed baseline and varied one component at a time, so that every comparison is a minimal pair — either the speech encoder changes or the facial decoder changes.

Four speech encoders were chosen to span a spectrum of what is encoded: HuBERT represents highly contextualized "semantic" content from self-supervised masking; SpeechTokenizer combines semantic distillation with acoustic reconstruction; WavTokenizer is a single-codebook codec optimizing extreme acoustic compression; and CosyVoice2 is trained with an automatic speech recognition pretext task, producing "label-based" tokens. Each encoder was frozen with pre-trained weights and paired with a decoder trained from scratch.

Two decoders were used: a GRU that runs frame-by-frame as the denoising network of a diffusion process trained with L1 reconstruction loss to predict data directly, and a non-causal Transformer with cross-attention trained with L1 loss plus first- and second-order motion smoothness losses on velocity and acceleration.

Training used BEAT2: about 27 hours of English speech from 25 speakers in scripted monologues across 8 basic emotions, aligned with 3D facial motion in FLAME parameter space. The authors converted FLAME parameters to 51-dimensional ARKit blendshapes using a transformation matrix provided by the dataset authors, and the decoders map representations to those blendshapes.

For evaluation, they measured blendshape reconstruction error on lips, jaw, and mouth (LVE), motion smoothness via second-degree derivatives (Jitter), and a new Bilabial Closure Score. BCS was computed by setting a threshold on the ground truth that bounds blendshape values on 70% of bilabial frames (the lip closure position), then counting the fraction of predicted bilabial frames that pass it. Statistical effects of encoder and decoder on LVE and Jitter were tested with linear regression models (R function lme) and post-hoc pairwise comparisons (emmeans), with significance at p < 0.05.

For perception, they selected three representatives — [HB+GRU], [HB+T.], and [CV2+T.] — and ran a MUSHRA-like study on Prolific. Participants watched a reference video, then rated four videos (the three systems plus a hidden copy of the reference) from 0 to 100 for closeness to the reference, across 15 randomly selected stimuli each. Participants whose average rating of the hidden reference fell below 80% were excluded, leaving 30 participants; scores were analyzed with a beta regression model (glmmTMB) and post-hoc comparisons (emmeans) at p < 0.05.

For probing, they aligned phonetic and facial information with tokens for every frame, using the dataset's phone alignment for phonetic classes. For facial features, they clustered ground-truth blendshape vectors from the test set with k-means into 32 visemes, aligned predicted tokens with facial motion captured at 30 FPS using nearest-neighbour temporal matching, and associated the nearest viseme to each token. They then computed normalized co-occurrence entropy between token IDs and phoneme or viseme labels, where 0 means a one-to-one mapping and 1 means chance level. Finally, a Ridge regression probe predicted continuous blendshape values from one-hot token features, reporting the median R² across all blendshapes.

Why This Matters

Impact on research. The paper reframes a practical design question for speech-driven animation: rather than asking which model is best, it asks which information a bottleneck should expose. Its finding that phonetic encoding helps only in the absence of low-structured acoustic detail, plus the observation that human ratings track lip-closure metrics rather than reconstruction error, gives concrete guidance for both architecture choice and evaluation methodology.

Real-world applications.

• Virtual avatars and telepresence: driving a 3D face from a compact token stream lowers the bandwidth and compute needed for real-time animated communication.
• Automated dubbing and localization: a single token space that yields synchronized speech audio and lip motion from text simplifies producing matched audio-visual content across languages.
• Games and interactive characters: text-driven AVTTS removes the separate audio-then-animation pipeline, reducing system complexity and latency.
• Accessibility: synchronized speech and facial motion can support more natural-looking synthesized speech for users of communication devices.

Industry relevance. The unified AVTTS result is directly relevant to companies building TTS-backed avatars and digital assistants, because it suggests facial animation can be bolted onto an existing token-producing TTS backbone by training only a lightweight facial decoder. The authors also frame this as a step toward multimodal speech language models where decoders can be adapted to frozen LLM backbones, which fits the broader trend of reusing large frozen models with small task-specific heads.

Future Directions

• Dedicated AVTTS work. The AVTTS pipeline is explicitly a proof-of-concept, with less lip-movement contrast across phonemes than the audio-driven version; the authors call for a focused study on AVTTS using discrete representations.
• Understanding why hybrid representations fail. The semantic+acoustic representation encodes phonetic classes best yet animates poorly; isolating which properties of the added acoustic layers hurt facial prediction remains open.
• Testing whether facial representation encoding is necessary. The best models happen to encode continuous blendshapes ([HB+GRU], R² = 0.25) or discrete visemes ([CV2+T.], 84.3%), but the authors note these values remain low and that further investigation is needed to establish whether such encoding is required for accurate prediction.
• Scaling to broader multimodal models. The conclusion points toward more versatile multimodal speech language models in which facial and speech decoders attach to frozen LLM backbones, which would raise questions about generalization beyond the single English BEAT2 dataset used here.

Target Audience

This paper is most useful to speech and graphics researchers working on speech-driven facial animation, discrete speech tokenization, or multimodal generation, and to engineers designing TTS-integrated avatar systems. It is also relevant to practitioners who need to choose a speech encoder for an animation pipeline and want evidence about which representation families are worth the effort. Readers without a background in self-supervised speech models, neural codecs, and quantization will find the methodology sections dense; the results and discussion sections are more accessible.

Authors’ abstract

The choice of speech representation is critical in speech-driven 3D facial animation. Representations differ in what they encode: SSL features emphasize segmental and semantic cues, neural codecs yield latents optimized for acoustic reconstruction, and ASR-style objectives produce label-based spaces. We evaluate four speech representation families for 3D facial synthesis, comparing their facial reconstruction quality across two facial decoders using objective metrics and a perceptual evaluation. We additionally conduct probing analyses that relate tokenized representations to phonetic units and to articulatory deformations. We found that encoding phonetic classes is beneficial for accurate facial animation prediction on both semantic and label-based representations with comparable facial animation quality. From the latter, we introduce an Audio Visual Text-to-Speech (AVTTS) pipeline that leverages, as a shared space, discrete representations to decode speech and 3D facial motion.