Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

AI summary

Overview

• Research area: Visual speech recognition (lipreading) and audio-visual self-supervised representation learning, combining computer vision, speech processing, and large language model decoding.
• Technical level: Advanced. The paper assumes familiarity with self-supervised pretraining objectives, Conformer architectures, quantization-based targets, and LLM adaptation techniques such as LoRA.
• Scope: The paper proposes and evaluates FLARE, a framework that learns visual speech representations by distilling knowledge from a frozen Whisper audio teacher into a dual-stream face-plus-lip video encoder, and tests it on LRS2, LRS3, and WildVSR lipreading benchmarks.

What This Paper Is About

Lipreading is hard because many different phonemes look identical on the lips, and because visual cues degrade under large head pose, low video resolution, or imprecise lip cropping. The paper asks how to structure supervision and visual input so that a pretrained visual encoder learns robust, phoneme-discriminative speech representations. FLARE answers this by (1) distilling semantically rich deep Whisper audio representations as discrete targets via a frozen random-projection quantizer, and (2) jointly modeling cropped lip region-of-interest (ROI) video and full-face video through parallel frontends, rather than relying on lip ROI alone.

Key Contributions

1. A cross-modal distillation recipe using a single external teacher. FLARE uses audio-only Whisper to produce deep representations, converts them into discrete targets with a random-projection quantizer, and pretrains the visual model with a masked prediction objective. This differs from prior work that jointly optimizes audio, visual, and audio-visual representations, or that ensembles multiple speech foundation models.
2. Joint face-lip modeling with early fusion. Two parallel ConvNeXt-inspired visual frontends encode full-face and lip ROI streams; the resulting features are channel-wise concatenated with dropout regularization. The paper notes that prior multi-scale efforts mainly combined separately trained models at the decision level, whereas this work performs early-stage joint modeling.
3. Robustness and representation analysis. The authors quantify how face, lip, and joint inputs behave under downsampled video resolution and varying head pose, and visualize phoneme discriminability and cross-modal alignment across encoder depth.
4. A progressive design study and two decoding settings. Each component (teacher choice, distillation objective, modality dropout, context encoder, masking strategy, visual frontend, visual inputs) is ablated in sequence. Downstream, FLARE is paired either with a lightweight 6-layer Transformer decoder or with an LLM decoder (Qwen-2.5-7B with adaptor and rank-16 LoRA), the latter denoted FLARE-LLM.

Main Findings

• Best headline numbers: FLARE-LLM with a 342M encoder pretrained on 4966 hours achieves word error rates of 11.7% on LRS2, 13.2% on LRS3, and 34.3% on the out-of-domain WildVSR benchmark (with Rank_wer of 37.6 on WildVSR).
• Low-resource comparison (Table 1): Pretrained on LRS3 + VoxCeleb2-En (1759 hours), the 117M FLARE Base model reaches 29.3% WER when finetuned on the LRS3 30-hour subset and 24.0% on the 433-hour subset, outperforming larger baselines such as AV-HuBERT (103M: 46.1/34.8), AV-data2vec (103M: 37.8/32.7), RAVEn (97M: 40.2/33.1), VATLM (103M: 42.6/34.2), and DistillAV (103M: 36.9/31.4). In the Large-size group, AV-HuBERT (325M) scores 32.5/28.6 and DistillAV (325M) 30.2/26.2, both above FLARE's 117M result.
• Efficiency claim under high-resource comparison: FLARE's 13.2% LRS3 WER approaches the previous best of 12.8% reported by LP-Conformer, which the paper states used 100k hours of training data versus FLARE's 4966 hours. FLARE is also reported as achieving this without an external language model, CTC supervision, or joint CTC-attention decoding for the lightweight decoder setting.
• Whisper targets beat raw acoustic features: In the design study, replacing deep Whisper representations with raw Mel filterbanks under random-projection quantization caused a large degradation (47.8%/42.5% versus 43.3%/36.4% on the 30h/433h settings), while different teacher models (WavLM+Conformer ASR ensemble, Whisper, AV-HuBERT) and objectives (regression + KL divergence, regression, k-means, random projection) gave broadly comparable results.
• Visual-dominant modality dropout is best: Among dropout settings, p_a/p_v/p_av = 0/0.75/0.25 performed best (41.0%/34.5%), ahead of the balanced 0.25/0.25/0.5 setting (43.3%/36.4%), the strictly visual 0/1.0/0 setting (42.3%/36.2%), and a curriculum from 0/0/1.0 to 0/0.75/0.25 (41.3%/35.2%).
• Enhanced Conformer helps most: Conformer-based context encoders consistently beat standard Transformers (39.2%/32.3% versus 43.3%/36.4%), with the enhanced Conformer (RoPE, grouped-query attention, SwiGLU, RMSNorm) reaching 38.1%/31.8%, and 37.0%/32.6% when combined with transferred modality dropout.
• Span replacement masking wins: Input-level masking by intra-utterance span replacement (36.3%/30.7%) outperforms input-level Gaussian noise masking (38.3%/31.9%) and feature-level masking with a learned mask embedding (37.0%/32.6%).
• Joint face-lip input gives the largest single gain: With a 3D-ConvNeXt frontend, moving from face-only to face+lip inputs improved results from 36.3%/30.0% to 30.9%/25.4% under the 200k-update design study. After extending pretraining to 400k updates, 3D-ConvNeXt reached 32.2%/27.7% versus 33.9%/29.5% for 3D-ResNet.
• Robustness to resolution: Downsampling test video to 0.75x, 0.5x, and 0.25x of original spatial resolution, joint face-lip input consistently achieved the lowest WER. Face input consistently outperformed lip input on LRS3, whereas lip input generally gave lower WER on LRS2, indicating the relative advantage depends on dataset characteristics.
• Robustness to head pose: With lip-only input, larger head poses were associated with significantly higher WER on both LRS2 and LRS3 in a linear regression of WER against mean absolute yaw. Face-only and joint face-lip inputs showed no statistically significant association between WER and head pose, and joint face-lip input performed best across the full range of poses.
• Pose distribution differs between datasets: The average mean absolute yaw angle is significantly higher in LRS3 than LRS2 (28.7 degrees versus 22.6 degrees, p = 3 × 10⁻³³), which the authors suggest may partly explain why face input helps more on LRS3.
• Representation quality: Joint face-lip input produces the most phoneme-discriminative representations in hierarchical clustering of pairwise phoneme distances, and t-SNE visualization shows visual and audio-visual representations become progressively better aligned as context encoder depth increases.
• Input flexibility (Table 4): With modality dropout, models pretrained and finetuned on joint face-lip input can be evaluated with face-only or lip-only inputs while retaining performance comparable to matched-input models; joint face-lip finetuning and testing gives the best results at 30.9% (30h) and 25.4% (433h).
• Compute cost (Table 5): On a 6-second utterance on a single NVIDIA Tesla P40 GPU, FLARE Base with face input costs 66.7 encoder GFLOPs and 69.0 total GFLOPs at RTF 0.19, versus DistillAV Base at 60.6/63.2 and RTF 0.18. FLARE Base with face+lip costs 119.3/121.6 at RTF 0.22. FLARE-LLM configurations jump to roughly 2611.7 to 2697.3 total GFLOPs with RTF 1.31 to 1.42.

Methodology in Plain English

The authors start with a large corpus of paired audio-visual speech and a frozen audio-only Whisper model. They run clean speech through Whisper to get deep audio representations, normalize them, project them through a random projection matrix into a small code space, and snap each one to its nearest entry in a randomly initialized, frozen codebook. That produces a sequence of discrete labels at the teacher's frame rate. The visual model is then trained to predict these labels at both masked and unmasked positions, which transfers phonetic and linguistic structure from audio to video without needing an external language model or text at pretraining time.

On the input side, the model sees two video streams: a cropped lip ROI and the full face. Each passes through its own frontend — a 3D convolution followed by four 2D-ConvNeXt blocks, with layer normalization, large-kernel depthwise separable convolutions, and GELU activations. Auxiliary audio features are also fed in, and the waveform is randomly corrupted with additive noise so the model does not lean on acoustics. Span-based corruption replaces a run of frames with another span from the same utterance for video, and replaces selected audio frames with a learned embedding.

Before fusion, modality dropout randomly selects which modalities and which visual streams are used, governed by probabilities p_a/p_v/p_av and p_f/p_l/p_fl. Surviving features are concatenated channel-wise, linearly projected, and normalized. A context encoder of enhanced Conformer blocks — using rotary positional encoding instead of relative positions, grouped-query attention, SwiGLU, and RMSNorm — then models long-range temporal structure, which the paper argues is critical because sequential context reduces viseme-to-phoneme ambiguity.

For downstream lipreading, the pretrained encoder is paired either with a randomly initialized 6-layer Transformer decoder (encoder frozen for the first half of training, then jointly finetuned, with no CTC objective) or with Qwen-2.5-7B via a two-layer FFN adaptor and rank-16 LoRA, trained in stages: adaptor only, then adaptor plus LoRA.

Pretraining uses LRS2 (~223 h), LRS3 (~436 h), VoxCeleb2-English (~1322 h), AVSpeech-English (~1549 h), and AVYT-talking (~1437 h, lip-only), totaling approximately 4966 hours after preprocessing. Datasets providing only lip ROIs (AVYT-talking, WildVSR) have the face representation set to zero, which the framework accommodates by design. Videos are resized to 96 × 96 pixels and randomly cropped to 88 × 88.

Why This Matters

The work argues that how supervision is structured — not just how much of it exists — determines how strong the learned visual representation is, and shows that a 117M-parameter model can outperform larger baselines in the low-resource LRS3 setting. It also provides a systematic account of face versus lip input behavior under degraded resolution and large head pose, which the authors note was previously insufficiently understood.

Real-world applications implied by the paper and its framing:

• Assistive communication tools for people who cannot produce or rely on audible speech.
• Human-computer interaction and intelligent device control using silent speech commands.
• Robust speech recognition in settings where audio is degraded or unavailable, where lipreading complements audio-based ASR.
• Deployment scenarios where only lip ROI video is available due to privacy constraints, which the framework supports by zeroing face features.

Industry relevance: the paper includes authors from iFLYTEK Research alongside Shaanxi Normal University, and the WildVSR out-of-domain evaluation plus the computational cost table (RTF 0.19–0.23 for the visual-only models versus 1.31–1.42 for FLARE-LLM) speak directly to deployment trade-offs between accuracy and inference speed. Code and pretrained models are released at the project's GitHub repository.

Future Directions

• Extending the approach beyond lipreading to audio-visual speech recognition and broader multimodal speech tasks, as the conclusion explicitly proposes.
• Understanding and closing the gap between FLARE and systems trained on far larger corpora such as the 100k-hour LP-Conformer, given that FLARE's 13.2% LRS3 WER approaches 12.8% with much less data.
• Investigating the dataset-dependent behavior of face versus lip inputs, since face input helps on LRS3 but lip input is often better on LRS2, and the paper links this partly to differing pose distributions.
• Reconsidering the modality-scheduling and transferred dropout strategies, which improved results under limited finetuning data but showed comparable or slightly degraded performance with more extensive finetuning (34.0%/30.7% versus 35.2%/30.6% after 400k pretraining updates).

Target Audience

Researchers and graduate students working on visual speech recognition, audio-visual self-supervised learning, and cross-modal distillation will get the most from this paper, particularly those interested in how teacher choice, quantization targets, masking strategy, and visual input selection interact. Engineers building production lipreading or multimodal speech systems will also benefit from the ablation tables and the GFLOPs/RTF comparison, since the paper reports accuracy alongside computational cost for each configuration.

Authors’ abstract

Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically rich Whisper representations as the acoustic teacher signal, converting them into discrete distillation targets through a random-projection quantizer and pretraining the model with a masked prediction objective. To capture complementary visual information, FLARE employs a dual-stream architecture that jointly processes lip region-of-interest (ROI) and full-face inputs, where the lip stream captures finegrained articulatory motion and the face stream encodes broader facial dynamics. During pretraining, auxiliary audio input is incorporated to facilitate cross-modal alignment, and modality dropout is applied to encourage robust visual-only representations. For downstream lipreading, FLARE serves as a visual encoder coupled with either a lightweight Transformer decoder or an LLM-based decoder. Evaluated on LRS2, LRS3, and the out-of-domain WildVSR benchmark, FLARE achieves strong lipreading performance, with word error rates of 11.7%, 13.2%, and 34.3%, respectively. A progressive design study confirms the contribution of each major component. Further analyses demonstrate that joint face-lip modeling improves robustness under degraded video resolution and large head pose variation, while yielding more phoneme-discriminative visual representations. Code and pretrained models are available at https://github.com/jxzhanggg/FLARE.