Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios with Synthetic Visual Data

AI summary

Overview

Research area: Audiovisual speech recognition (AVSR) — specifically multimodal speech recognition that fuses acoustic signals with visual articulatory cues (lip and facial movement), combined with synthetic data generation via lip-syncing.

Technical level: Advanced. The paper assumes familiarity with self-supervised speech representation learning, transformer encoder-decoder architectures, Word Error Rate (WER), and signal-to-noise ratio (SNR) evaluation, though its central idea is stated simply.

Scope: The paper proposes and evaluates a framework that replaces real audiovisual training data with synthetic talking-head video in order to build AVSR systems for languages that have audio recordings but no annotated video corpora.

What This Paper Is About

Audiovisual speech recognition improves transcription robustness because it can use lip and facial movements alongside audio, and it still works when the audio is degraded or partly missing. Building these systems normally requires large labeled video corpora, which exist for only a handful of high-resource languages, even though many languages have plenty of audio-only data.

The authors ask whether synthetic video — created by animating static face photographs with lip movements synchronized to real speech — can stand in for real video. If it can, then any language with audio recordings could get a multimodal speech recognition system without ever collecting a single labeled video.

Key Contributions

1. Empirical evidence that synthetic lip-synchronized video can serve as effective visual supervision for AVSR training, demonstrated both as an augmentation strategy on Spanish and as the sole visual signal for Catalan.
2. The first AVSR system for Catalan (to the best of the authors' knowledge), trained in a zero-AV-resource setting using roughly 723 hours of Catalan audio paired with synthetic visual streams.
3. A large synthetic Catalan audiovisual corpus of over 700 hours of talking-head video, generated from audio-only corpora (TV3Parla, approximately 291 hours, and ParlamentParla, approximately 432 hours).
4. A manually annotated Catalan AV benchmark and a semi-automatic audiovisual annotation pipeline — the test set contains 51 minutes and 38 seconds of broadcast material with frame-aligned transcriptions; the pipeline combines segment extraction, morphological filtering, pseudo-labeling, and manual verification via a custom graphical interface.

Main Findings

• Synthetic video helps as augmentation on Spanish: Adding synthetic talking-head videos to real Spanish AV training data reduced WER from 9.3% to 8.1% on LIP-RTVE (a 12.9% relative reduction) and from 15.4% to 12.9% on CMU-MOSEAS Spanish (a 16.2% relative reduction).

• The gain is not just extra audio: Because the synthetic clips are paired with real audio that was not in the original AV training set, the authors controlled for added acoustic coverage. Training an audiovisual model and an audio-only model on identical data, the AV model reached 8.1% versus 8.6% on LIP-RTVE (5.8% relative reduction) and 12.9% versus 13.5% on CMU-MOSEAS Spanish (4.4% relative reduction). This isolates a genuine contribution from the synthetic visual stream.

• Catalan AVSR works without any real training video: On the manually annotated Catalan benchmark, the audiovisual model achieved 19.6% WER, versus 23.1% for the audio-only variant (visual inputs masked) — the paper reports this as a 15.1% relative improvement.

• Video-only performance is poor: Masking audio to leave only synthetic video produced 105% WER. The authors attribute this to the visual modality being inherently weaker than audio and to synthetic video being less intelligible than real video, consistent with prior lipreading work on synthetic input.

• Competitive with far larger ASR systems: Whisper-large scored 31.4% WER with 1,550M parameters and 1,883 hours of Catalan data; Whisper-large-v3 scored 18.3% with 1,550M parameters and more than 1,883 hours of Catalan data. The authors' model scored 19.6% with 325M parameters and 723 hours of Catalan data, so it clearly beats Whisper-large and approaches Whisper-large-v3.

• Better robustness under noise: Across additive white Gaussian noise at SNR levels from −5 dB to 20 dB, Whisper-large-v3 was slightly better than the authors' model in clean conditions but degraded rapidly as noise increased. The AV model degraded more gradually, outperforming both Whisper variants at low SNRs with a flatter WER curve.

• Model configuration: The backbone is AV-HuBERT Large, initialized from a checkpoint pre-trained on English LRS3 and VoxCeleb2, with a randomly initialized 6-layer Transformer decoder producing SentencePiece (unigram) subwords. Fine-tuning used Adam with a base learning rate of 1×10⁻³, a tri-stage learning rate schedule with warmup and decay, and a frozen encoder for the first 22,500 updates.

Methodology in Plain English

Step 1 — Pick faces and audio. Still portraits come from the FFHQ dataset. Audio comes from Mozilla CommonVoice for Spanish (approximately 512 hours) and from TV3Parla and ParlamentParla for Catalan. For the Spanish real-video comparisons, LIP-RTVE (approximately 13 hours) and CMU-MOSEAS (approximately 13 hours) are used.

Step 2 — Filter for usable faces. A morphological face detector keeps only images with a clearly visible mouth region, since those animate well.

Step 3 — Generate the fake video. Each audio clip is paired with a randomly chosen still image to build a still-frame video matching the clip's duration. A pre-trained Wav2Lip+GAN model then animates the mouth, producing lip movements synchronized to that speech. The result is a synthetic talking-head corpus aligned with the original transcriptions and mirroring the original audio dataset's duration and characteristics.

Step 4 — Fine-tune a pretrained model. Starting from AV-HuBERT Large, the authors attach a decoder and train the system sequence-to-sequence on these synthetic pairs. The same recipe is applied to Spanish (with and without synthetic augmentation on top of real video) and to Catalan (synthetic video only, no real training video).

Step 5 — Build an evaluation set, since none exists. For Catalan, the authors developed a semi-automatic annotation pipeline: segments are extracted, morphologically filtered for mouth visibility, automatically pseudo-labeled, and then manually verified and refined in a custom graphical interface. This yielded the 51-minute-and-38-second manually verified test set used throughout the Catalan experiments.

Step 6 — Test whether the visual signal is actually being used. The same architecture is fine-tuned three ways on the Catalan data — audiovisual, audio-only (visual inputs masked), and video-only (audio inputs masked). If the audiovisual model beats audio-only using identical data, the synthetic visual stream is doing real work.

Why This Matters

Impact on research. The paper opens a route for multimodal speech recognition in the many languages that have audio but no annotated video. It also extends synthetic video beyond lipreading — prior work used it for visual-only recognition or visual-only data augmentation, whereas this work uses it as visual supervision inside an audiovisual system, which the authors state had not previously been investigated. The released pipelines for synthetic data generation and semi-automatic annotation (https://github.com/Pol-Buitrago/SynthAVSR) make the method reusable.

Real-world applications:

• Under-resourced and minority languages: Any language with audio recordings and a supply of face images could obtain an AVSR system without funding a video annotation campaign.
• Broadcast and archival transcription: Catalan broadcast material is exactly the source used for the benchmark here, suggesting direct applicability to media archives.
• Noisy or degraded audio environments: The robustness results at low SNR matter for real recordings affected by noise, reverberation, channel distortion, or low-bitrate codecs.
• Degraded or partially missing audio: The paper notes AVSR can reconstruct partially missing audio segments, which matters for damaged recordings and for conditions where the audio channel is partly or fully unavailable.

Industry relevance. The result that a 325M-parameter model trained on 723 hours approaches a 1,550M-parameter model trained on more than 1,883 hours of Catalan data is a cost argument: synthetic video generation is cheap and automated, while collecting and annotating native video is not. For product teams targeting languages outside the high-resource set, this is a concrete path to multimodal robustness at far lower data and compute cost.

Future Directions

1. Does the method scale to more languages? The authors describe the pipeline as language-agnostic and applicable to any language given audio corpora, but it is validated on Spanish and Catalan only. How it behaves for languages with far less audio than Catalan's roughly 723 hours is not reported.

2. Can synthetic video approach real video quality? The video-only model scored 105% WER, which the authors partly attribute to synthetic video being less intelligible than real video. Whether better lip-sync generation narrows the gap between synthetic and real visual supervision remains open.

3. Can the annotation pipeline be pushed further toward low supervision? The authors note the pipeline could be extended to annotate real training data, complementing the synthetic corpus and providing a low-supervision alternative for zero-AV-resource scenarios — which would test synthetic and real data in combination rather than as substitutes.

4. Where does the robustness advantage saturate? The AV model overtakes both Whisper variants only at low SNRs. The exact crossover point, and whether other noise types beyond additive white Gaussian noise behave the same way, is not reported.

Target Audience

Researchers and engineers working on multimodal and audiovisual speech recognition, particularly those targeting under-resourced languages; practitioners who need robust transcription under noise but lack labeled video; and teams interested in synthetic data generation, generative lip-sync models, or self-supervised speech representation transfer. Readers with a background in speech processing or self-supervised learning will get the most from the fine-tuning details and the controlled audio-only versus audiovisual comparisons.

Authors’ abstract

Audiovisual speech recognition (AVSR) combines acoustic and visual cues to improve transcription robustness under challenging conditions but remains out of reach for most under-resourced languages due to the lack of labeled video corpora for training. We propose a zero-AV-resource AVSR framework that relies on synthetic visual streams generated by lip-syncing static facial images with real audio. We first evaluate synthetic visual augmentation on Spanish benchmarks, then apply it to Catalan, a language with no annotated audiovisual corpora. We synthesize over 700 hours of talking-head video and fine-tune a pre-trained AV-HuBERT model. On a manually annotated Catalan benchmark, our model achieves near state-of-the-art performance with much fewer parameters and training data, outperforms an identically trained audio-only baseline, and preserves multimodal advantages in noise. Scalable synthetic video thus offers a viable substitute for real recordings in zero-AV-resource AVSR.