Synthetic-to-Real Code-Switched ASR: A Spontaneous Nepali-English Podcast Benchmark

AI summary

Overview

Research area: Automatic speech recognition (ASR) for code-switched speech, specifically Nepali–English conversational audio, with a focus on synthetic-to-real domain transfer and sequence-level reinforcement learning.

Technical level: Intermediate (assumes familiarity with ASR metrics like WER/CER and basic fine-tuning concepts; the reinforcement-learning sections are more advanced).

Scope: The paper introduces a verified 15.45-hour spontaneous Nepali–English podcast benchmark and measures how far supervised fine-tuning on under one hour of synthetic code-switched speech, followed by Group Relative Policy Optimization (GRPO), can transfer to real human conversational speech.

What This Paper Is About

Nepali–English code-switched conversation is hard for ASR systems for two reasons: "script confusion" (writing English words in Devanagari phonetics, or Nepali words in Latin letters) and elevated errors right at the points where speakers switch languages. Collecting and annotating real code-switched audio is expensive, so the author asks whether cheap synthetic speech can do the heavy lifting. The paper builds a carefully audited real podcast test set and then measures a two-stage recipe — synthetic supervised fine-tuning, then GRPO with transcription and script rewards — against an unadapted 1.22B-parameter baseline.

Key Contributions

1. Synthetic-to-real transfer evidence: A FastConformer encoder–decoder model (indic-transcribe-flex) fine-tuned exclusively on 0.95 h (517 utterances, 211 unique sentences) of synthetic code-switched audio transfers zero-shot to unconstrained human conversational speech, reducing MER from 23.40% to 21.52% and SPER from 35.89% to 28.77%.

2. Empirical analysis of GRPO and script rewards: Sequence-level policy optimization produces small but statistically significant adjustments on 173 of 10,214 test utterances (−0.07 pp SPER, −0.09 pp English script confusion), while an auxiliary script reward term adds no measurable gain over MER-only optimization in an SFT-converged encoder–decoder model.

3. An open, audited benchmark: 15.45 h / 13,044 verified utterances (10,214 frozen test, 2,830 dev) from 28 public Nepali podcast episodes and 72 speakers, with transcripts, timestamps, pseudonymous speaker labels, YouTube identifiers, and alignment scripts released on Hugging Face under CC BY-NC-SA 4.0.

4. A negative architectural result: A token-level script gating head attached to decoder hidden states degrades performance relative to plain SFT on dev (MER 18.92% → 19.14%, SCR_en 5.35% → 5.75%).

Main Findings

• Synthetic SFT does most of the work: On the 10,214-utterance frozen test split, SFT drops MER by 1.88 pp (23.40% → 21.52%) and switch-point error rate by 7.12 pp (35.89% → 28.77%). WER falls from 35.65% to 32.16% and CER from 20.65% to 19.11%.

• GRPO gains are tiny but statistically real: S4 vs. S1 gives ΔSPER = −0.07 pp (95% CI [−0.12, −0.03], p < 0.05) and ΔSCR_en = −0.09 pp (95% CI [−0.13, −0.06], p ≤ 0.001), with a mean MER difference of −0.03 pp (95% CI [−0.05, −0.01]). An episode-clustered block bootstrap over 25 episodes confirms this (SPER p = 0.022).

• Only 1.69% of test utterances change: Exactly 173 of 10,214 utterances differ between S1 and S4; the remaining 10,041 are identical. The 173 diffs span 20 episodes with at most 25 per episode, ruling out single-speaker artifacts.

• Total pipeline vs. baseline: S4 vs. S0 yields −1.90 pp MER (95% CI [−2.37, −1.42]), −7.19 pp SPER (95% CI [−7.97, −6.46]), and −0.35 pp SCR_en (95% CI [−0.65, −0.04]), all intervals excluding zero (p ≤ 0.001 for MER and SPER, p = 0.026 for SCR_en).

• The script reward is redundant here: S3 (λ_scr = 0.0) and S4 (λ_scr = 0.5) produce identical dev numbers (18.89% MER, 25.53% SPER, 5.24% SCR_en) and select the identical checkpoint at step 50. The explanation: SFT already reaches 0.0200 MER on synthetic prompts, so rollout candidate groups have near-zero script variance — over 75% of mini-batches had zero within-group script variance.

• Token-level gating backfires: Adding a script gate (S2) raises dev MER from 18.92% to 19.14% and SCR_en from 5.35% to 5.75%, because greedy logit biasing forces premature script assignment at loanword prefixes.

• Nepali monolingual performance improves, English is preserved: On BHASAANUVAD (1,000 utterances, 1.95 h), SFT lowers Nepali MER from 20.80% to 18.09% (−2.71 pp), settling at 18.42% after GRPO (−2.38 pp vs. baseline), with SCR dropping from 8.71% to 6.95%. On LibriSpeech test-clean (1,000 utterances, 1.50 h), WER moves from 3.33% to 3.42% (S1) and 3.38% (S4), a difference that is not significant (95% CI [−0.17, +0.30], p > 0.05), though English MER rises from 4.21% to 5.27% and SCR from 0.59% to 1.03%.

• Synthetic checkpoints saturate fast: Synthetic dev MER is 2.00% at steps 50, 100, and 150 (tied with SFT), then drifts to 2.20%–2.32% across steps 200–500 as policy updates exhaust the synthetic signal.

• Benchmark is heavily audited: From a raw pool of 13,277 clips (15.74 h), 233 were quarantined for audio loops/music, 161 machine-translation leakages were removed, 819 token-doubling glitches from diarization timestamp overlaps were corrected, and 86 digit instances were verbalized. Inter-annotator character agreement was 98.4% on a 500-clip dual-audit sample.

Methodology in Plain English

Building fake training data cheaply. The author prompted Gemini 3.8 Flash to generate sentences with natural Nepali–English mixing, filtered them against strict lexical rules and a 5-gram check to make sure no sentence overlapped the real test references, then spoke them aloud with two different TTS systems: OmniVoice (fine-tuned on 100+ hours of native Nepali speech, two male voices) and Gemini TTS (four voices with varied English accents). Audio was downsampled to 16 kHz mono, band-passed to telephone range (300–3400 Hz), given 10–30 dB SNR noise, and round-trip-checked with whisper-large-v3-turbo (CER ≤ 0.20). Result: 517 utterances, 0.95 h.

Building the real test set. Clips from 28 public Nepali podcasts were auto-transcribed with gemini-3.5-transcribe, normalized with gemini-3.8-flash, and then every single clip was audited by two bilingual annotators against the raw audio. Speakers were linked across episodes using acoustic speaker embeddings and manually partitioned so that dev (20 speakers) and test (52 speakers) share zero speakers; in the 13 episodes used by both splits, host turns went to dev and guest turns to test.

Adapting the model. The base is indic-transcribe-flex (1.22B parameters). The acoustic encoder is frozen entirely; only the 24-layer Transformer decoder and output projection are trained — 410,455,024 parameters, or 33.60% of the base model. Stage 1 is standard cross-entropy SFT (1,000 steps, 1.0e-5 learning rate). Stage 2 is GRPO trained on the same 517 synthetic utterances: for each input the model samples 8 candidate transcripts, scores each against the reference, normalizes the scores within the group into advantages, and applies a clipped policy-gradient update.

Measuring what matters. Four metrics are used: MER (word errors on English segments combined with character errors on Devanagari segments), WER, CER, SPER (MER restricted to tokens within ±1 of a language switch point), and SCR (the share of word substitutions that flip script, with mixed-script tokens and insertions/deletions excluded). Significance is tested with paired bootstrap resampling (B = 1,000, seed 42) and an episode-clustered block bootstrap.

Why This Matters

For research: The paper provides a clean, audited code-switched benchmark and a negative result that matters: script-aware RL rewards, which prior work found effective for decoder-only audio LLMs, do not help a converged encoder–decoder model. The near-zero within-group script variance diagnosis gives a concrete mechanism for why, and the token-gating ablation gives a second failure mode for auxiliary script heads.

Real-world applications:

• Transcription and subtitling of Nepali podcasts, interviews, and YouTube content where English technical terms appear mid-sentence.
• Call-center and customer-service analytics for South Asian markets where agents mix Nepali and English.
• Clinical, legal, and government dictation in Nepal, where errors on English loanwords like drug names or brand names are costly.
• Voice interfaces and meeting-notes tools for urban Kathmandu speakers, where the paper notes English embedding is routine.

Industry relevance: The entire adaptation runs on a single NVIDIA RTX 4090 (24 GB VRAM) in about 4 minutes for SFT (3m 52s) and about 17 minutes for GRPO (16m 51s) under FP16. That makes synthetic-only adaptation to a new language pair or domain economically attractive compared to collecting and annotating human code-switched audio.

Future Directions

• Separating GRPO gains from extra gradient steps: The paper explicitly notes that without an extended SFT compute control, the marginal gain cannot be separated with certainty from simply training longer.
• Larger and more varied synthetic corpora: Training used only 517 utterances (0.95 h) across two male timbres; scaling voice diversity and volume is an obvious untested lever.
• External baselines and multi-seed variance: The author flags benchmarking external foundation models (for example Whisper) and measuring multi-seed training variance as remaining work.
• Fixing the dev/test role asymmetry: Because hosts went to dev (2.65 switch points per utterance) and guests to test (2.11), dev has higher code-switching density than test; a rebalanced partition would make cross-split comparisons cleaner.

Target Audience

Researchers and engineers working on multilingual and code-switched ASR, especially those targeting South Asian languages; practitioners who want a low-cost recipe for adapting a foundation ASR model to a new language pair without human-recorded training audio; and anyone needing a verified Nepali–English evaluation set with published transcripts, timestamps, and speaker-disjoint splits. Readers interested in applying RL to speech will find the ablation and the null result on script rewards particularly useful.

Authors’ abstract

Nepali-English code-switched conversational speech presents severe challenges for automatic speech recognition (ASR), chiefly through script confusion (inappropriate mixing of Devanagari and Latin orthography) and elevated error rates at language switch points. Supervised fine-tuning (SFT) can adapt multilingual foundation models, but human code-switched audio is expensive to collect and annotate. We curate a new benchmark of 13,044 spontaneous podcast utterances (15.45 h total; 10,214 test, 2,830 dev), audited and verified to eliminate machine translation leakages and acoustic loops. We investigate a two-stage synthetic-to-real adaptation framework: Stage 1 applies SFT on 0.95 h of speech synthesized via a multi-speaker, dual-engine TTS pipeline; Stage 2 explores Group Relative Policy Optimization (GRPO) with transcription and script rewards. We find that synthetic SFT provides the primary cross-domain transfer to spontaneous human speech, reducing Mixed Error Rate (MER) from 23.40% to 21.52% (-1.88 pp) and switch-point error rate (SPER) from 35.89% to 28.77% (-7.12 pp). Subsequent GRPO yields small, statistically significant adjustments on 173 test utterances (1.69%), driving English script confusion (SCR_en) from 6.45% to 6.36% (p &lt;= 0.001) and SPER to 28.70% (p = 0.022 clustered). Crucially, ablation reveals that in this SFT-converged encoder-decoder setting, an auxiliary script reward term yields performance identical to MER-only optimization because rollout candidate groups exhibit near-zero script variance on synthetic data. On monolingual benchmarks, SFT improves Nepali MER by 2.71 pp (stabilizing at 18.42% after GRPO, a 2.38 pp gain over baseline) while preserving English lexical WER (3.38% vs. 3.33%). Verified transcripts, timestamps, episode metadata, and reconstruction scripts are publicly released.