Constrained CTC Decoding for Efficient Diacritic Restoration

AI summary

Overview

Research area: Natural Language Processing — Arabic speech recognition and diacritic restoration, specifically constrained CTC decoding.

Technical level: Intermediate. Readers need some familiarity with Connectionist Temporal Classification (CTC), weighted finite-state transducers (WFSTs), and word/character error rate metrics.

Scope: The paper proposes a non-autoregressive, CTC-based decoding constraint for restoring Arabic diacritics from speech plus an undiacritized transcript, and compares it against text-only and multi-modal (text+ASR) baselines on Classical Arabic and Modern Standard Arabic test sets.

What This Paper Is About

Arabic is written without most of its diacritical marks, which are normally inferred from context — leaving words ambiguous and causing pronunciation errors in downstream applications. When the corresponding speech is available, acoustic cues such as vowel quality, length, and gemination can help recover the missing diacritics. The paper's goal is to restore diacritics for speech transcripts more efficiently than existing multi-modal systems, by forcing a CTC-based ASR model to reproduce the given undiacritized characters exactly while predicting only the diacritics at permitted positions.

Key Contributions

1. A constrained CTC decoding method for speech-to-text diacritization. The method constructs a character-level diacritization lattice G_char(u) from the undiacritized transcript u and restricts decoding hypotheses to paths that correspond to valid diacritized realizations, so base alphabetic characters are emitted in their reference locations and the model predicts only diacritics.

2. A compact, single-model architecture. Unlike prior multi-modal approaches that fuse an acoustic model with a separate text-based decoder, the proposed method integrates as a drop-in decoding constraint on top of a standard CTC ASR model, requiring no additional text encoder and no multi-modal training.

3. Demonstrated gains in both performance and efficiency. Against a more computationally complex multi-modal diacritic restoration baseline, the method achieves statistically significant reductions in diacritic error rates on Classical Arabic and Modern Standard Arabic test sets (ArVoice and ClArTTS), with larger gains under cross-dataset mismatch and combined training.

4. An extension to partially diacritized training data (Appendix A). Two special wildcard tokens, <no_diac> (true label is no diacritic, used for fully diacritized data) and <unk_diac> (true diacritic unknown, used for partially diacritized or undiacritized data), are introduced; <unk_diac> is excluded from the wildcard set at inference to preserve diacritic coverage.

Main Findings

• Speech modality helps: Consistent with prior work, the Text+ASR baseline substantially outperforms the text-only baseline on diacritic restoration, confirming that acoustic information complements text for inferring missing diacritics.

• Matched-setting parity: When trained and tested on ClArTTS, the Text+ASR baseline reports WER 12.33 and DER 3.54, while the proposed method reports WER 11.21 and DER 3.53 — nearly identical DER.

• Stronger cross-dataset robustness: Trained on ClArTTS and tested on ArVoice, the Text+ASR baseline degrades to WER 56.20 / DER 19.21 versus WER 39.89 / DER 12.04 for the proposed method. Trained on ArVoice and tested on ClArTTS, the Text+ASR baseline reaches WER 99.94 / DER 76.61 versus WER 34.94 / DER 11.86 for the proposed method.

• Best combined-training results: Trained on ClArTTS + ArVoice, the proposed method reports WER 13.05 / DER 3.80 on ClArTTS and WER 30.36 / DER 8.69 on ArVoice, compared with WER 29.63 / DER 9.05 and WER 34.47 / DER 9.93 for the Text+ASR baseline.

• Statistically significant improvement: Using 2,000 bootstrap samples with replacement, the 95% confidence intervals for the DER difference between the proposed method and the Text+ASR baseline (combined-training setting) lie entirely below zero for both datasets, indicating the DER improvement is significant.

• ASR fine-tuning matters: The Wav2vec2-XLSR model improves from DCov 50.55 / Diac WER 88.58 on ArVoice zero-shot to DCov 75.25 / Diac WER 40.82 after fine-tuning; on ClArTTS it goes from DCov 70.19 / Diac WER 58.91 to DCov 79.12 / Diac WER 17.17. Reference diacritic coverage is ArVoice = 73.19 and ClarTTS = 79.13.

• Partially diacritized training degrades coverage naively: With naive mixed-coverage training (Setup-2), seen-speaker DCov falls to 59.75% and unseen-speaker DCov to 53.13%, versus 75.92% and 75.52% in the fully diacritized upper bound (Setup-1).

• Special tokens recover coverage without hurting quality: Setup-3 raises DCov to 71.50% (seen) and 73.17% (unseen), and its DER is not higher than Setup-2's (12.18 vs. 22.57 including "no diacritic" for seen speakers; 10.99 vs. 30.44 for unseen speakers), indicating the additional diacritics introduced by excluding <unk_diac> match the default prediction quality.

• Speaker-specific effects: In Setup-2, diacritic coverage for unseen speaker female_ab drops to 35% versus 75% in Setup-1, while unseen speaker male_ac matches the baseline; the authors suggest the higher representation of fully diacritized male voices in training (particularly the larger male_asc subset) as a possible confound. Speakers male_ae and female_af are excluded from evaluation because their transcripts are undiacritized in both train and test splits.

Methodology in Plain English

The researchers take an off-the-shelf CTC-based ASR model (Wav2vec2-XLSR, fine-tuned on Arabic speech) and change how it decodes. Instead of allowing the decoder to output any character at any time step, they build a small graph from the undiacritized transcript the user already has. The graph walks through that transcript character by character, and after every Arabic letter it opens a slot in which only diacritics (or the CTC blank, meaning no diacritic) may be emitted. This graph is drawn as a linear-chain weighted finite-state transducer — the pattern for an undiacritized input c1c2c3 becomes c1.c2.c3., where each dot is a wildcard slot.

Decoding is then restricted to paths through this graph: base letters must appear in the given order and in their given positions, and only the diacritic slots are free to vary. The authors describe this as a partial forced-alignment formulation. In practice it can be implemented by composing the CTC decoding graph with the lattice or by restricting beam expansion to symbols permitted by the current lattice state.

Training uses a CTC loss on the negative log-likelihood of the reference diacritized transcript. Each base character has at most one diacritic, and multi-diacritic combinations are merged into a single-character token to keep a 1-to-1 character-level representation; the CTC blank token denotes the absence of a diacritic. Text is normalized into Unicode Normalization Form Canonical Composition (NFC) and multiple whitespace characters are collapsed. All models are fine-tuned with AdamW, a linear learning-rate schedule (warmup then decay), a peak learning rate of 3e-4, 1,500 warmup steps, 100 epochs, and a batch size of 64 on an NVIDIA A100 GPU with 80GB VRAM. Evaluation reports WER, CER, Diacritic Coverage Rate (DCov), and Diacritic Error Rate (DER), where DER is computed strictly under the assumption that reference and hypothesis base-character sequences match, and includes both "no diacritic" and word-ending diacritics.

Why This Matters

Impact on research: The work shows that a decoding-time constraint on a standard CTC model can replace a more computationally complex multi-modal architecture for speech-conditioned diacritization, and that it generalizes better across Arabic varieties (Classical Arabic vs. Modern Standard Arabic) than a baseline that relies only on the final ASR hypothesis. It also connects CTC decoding constraints to the older WFST tradition of encoding linguistic constraints in decoding graphs, and offers an approach for absorbing partially diacritized and undiacritized training data without losing diacritic coverage.

Real-world applications:

• Text-to-speech systems for Arabic, where missing diacritics cause systematic pronunciation errors.
• Assistive reading interfaces for Arabic, where under-specification of the script causes ambiguity for homographs and morphologically complex forms.
• Speech data curation for Arabic, since most existing Arabic speech datasets have undiacritized transcripts and can be processed by this model.
• Producing fully diacritized ASR transcripts from unlabeled audio, using the ASR model directly as a diacritized transcription system.

Industry relevance: The method is described as a drop-in decoding constraint that preserves the simplicity and speed advantages of CTC decoding and needs no extra text-based module or multi-modal training. That translates to lower inference and training cost for production Arabic ASR and TTS pipelines, with particular value for the large volume of conversational and news-domain speech that is not diacritized.

Future Directions

• Generalization beyond Classical and Modern Standard Arabic. The paper notes that multi-modal approaches generalize poorly to Modern Standard and Dialectal Arabic, and that more research is needed to generalize performance; the dialectal case remains open.

• Better handling of the 1,600-hour style data imbalance and mixed-coverage training. The appendix shows that naively mixing undiacritized, partially diacritized, and fully diacritized data reduces coverage for unseen target speakers in Setup-2, and that coverage varied by speaker gender representation; further work is needed to remove these confounds.

• Closing the gap with text augmentation. The authors note that the baseline can leverage the Tashkeela text corpus while their model cannot; finding a way to use large text-only resources within this decoding-constrained framework is a natural next step.

• Extending the lattice and wildcard design. The <no_diac> and <unk_diac> tokens were only explored in the appendix; systematically tuning how unknown diacritics are represented, and how inference filters wildcards, could further improve the coverage-versus-accuracy trade-off.

Target Audience

Researchers and engineers working on Arabic NLP, speech recognition, and speech synthesis, particularly those interested in CTC decoding, constrained or lattice-based decoding, and WFST-based inference. It is also relevant to practitioners who need diacritized Arabic transcripts for TTS, assistive reading, or speech corpus curation, and to anyone studying how to exploit partially diacritized or undiacritized speech data in ASR training.

Authors’ abstract

In this work, we address diacritic restoration for Arabic speech transcripts. Most speech data are undiacritized, limiting the ability of modeling fine-grained phonological distinctions. The speech modality has recently been explored as a way to complement text-based diacritic restoration efforts. We propose an efficient non-autoregressive approach for speech-to-text diacritization based on Connectionist Temporal Classification (CTC). Our method incorporates hard constraints during decoding by constructing a character-level diacritization lattice from an undiacritized transcript and restricting hypotheses to valid diacritized realizations. We evaluate on Classical Arabic and Modern Standard Arabic test sets (namely, ArVoice and ClArTTS) against a more computationally-complex multi-modal diacritic restoration baseline, and show statistically significant reductions in diacritic error rates in both, demonstrating that the proposed approach offers both performance and efficiency gains.