NADIR: Differential Attention Flow for Non-Autoregressive Transliteration in Indic Languages

AI summary

Overview

Research area: Natural Language Processing — non-autoregressive sequence-to-sequence modeling, specifically multilingual transliteration between Roman script and 21 Indic languages.

Technical level: Intermediate. The paper assumes familiarity with transformer attention, autoregressive versus non-autoregressive decoding, and mixture-of-experts routing, though the core ideas are explained with worked linguistic examples.

One-sentence scope: The paper introduces NADIR, a non-autoregressive transliteration architecture combining a Differential Transformer with Mixture-of-Experts routing, and evaluates it against the autoregressive state-of-the-art IndicXLIT on the Aksharantar benchmark for speed, character accuracy, and hallucination error types.

What This Paper Is About

Autoregressive models generate text one token at a time, which is accurate but slow; non-autoregressive (NAR) models generate everything in parallel, which is fast but prone to producing wrong characters, dropped characters, and repeated spans. The authors argue that transliteration, unlike open-domain translation, depends mostly on local character context, so the heavy sequential machinery of autoregressive models may be unnecessary. Their goal is to build an NAR transliteration system that keeps the speed advantage while closing the accuracy gap that normally comes with abandoning autoregression.

Key Contributions

1. A named failure taxonomy for NAR models. The authors characterize "NAR Hallucinations" — insertions, substitutions, omissions, and repetitions — as the recurring errors introduced when switching from autoregressive to non-autoregressive decoding, and they quantify each type across three model variants.

2. NADIR, a non-autoregressive transliteration architecture. It combines multi-head differential attention with a Mixture-of-Experts (MoE) module with learned top-2 routing, and is described as the first work to address the hallucination problem in non-autoregressive NLP models for transliteration.

3. Implicit sequence termination without a length predictor. Instead of a separate length-prediction network, NADIR appends an [EOS] token to every target sequence during training and computes loss only up to and including the first predicted [EOS], letting the model learn where to stop.

4. A component-wise ablation. The paper isolates the contribution of differential attention and of MoE, reporting per-error-type counts for a Standard NAR, a Differential NAR, and a Differential MoE NAR variant.

Main Findings

• Speed: NADIR transliterates 180.1k words in 2 minutes and 55 seconds (approximately 1005 words/second), while IndicXLIT requires 38 minutes and 50 seconds (approximately 77 words/second). The abstract reports over a 13× speed-up over the state-of-the-art autoregressive baseline.

• Roman → Indic accuracy: Across the 20 languages in the results table, NADIR reaches a mean CER of 15.78 (std. dev. 5.67) versus IndicXLIT's 14.44 (std. dev. 4.35). Mean WAcc is 50.13% for NADIR versus 51.23% for IndicXLIT. Mean inference time drops from 116.48 seconds to 8.95 seconds (std. dev. 4.14).

• Indic → Roman accuracy: NADIR achieves a mean CER of 17.56 versus IndicXLIT's 16.59, and mean WAcc of 34.5% versus 36.29%. Mean inference time falls from 124.18 seconds to 9.07 seconds (std. dev. 3.95).

• Head-to-head wins: Despite similar averages, NADIR outperforms IndicXLIT in both CER and WAcc for 5 out of the 20 languages.

• Hallucination reduction over the standard NAR baseline: Repetition errors drop by 49.53%, substitution errors by 24.45%, omission errors by 32.92%, and insertion errors by 16.87%.

• Component ablation (mean CER / mean WAcc): Standard Attention NAR — 21.88 / 38.98; Differential Attention NAR — 16.12 / 46.89; Differential Attention + MoE — 15.78 / 50.13. The authors conclude differential attention is the primary contributor to improvement, with MoE cleaning up remaining edge cases.

• MoE's marginal effect: Adding MoE on top of differential attention reduces insertion errors by 14.55%, substitution errors by 4.98%, and repetition errors by 22.78%, but increases omission errors by approximately 8% (-8.02% gain).

• Differential attention's marginal effect: Moving from standard attention to differential attention yields gains of 2.71% on insertions, 20.49% on substitutions, 37.90% on omissions, and 34.64% on repetitions — limited impact on insertions specifically.

• Latency-accuracy deltas: For Indic → Roman, mean ΔCER is -0.96 (std. dev. -0.2), mean ΔWAcc is 1.80 (std. dev. 0.25), and mean ΔInfT is 115.11 seconds (std. dev. 67.08). For Roman → Indic, mean ΔCER is -1.34 (std. dev. -1.32), mean ΔWAcc is 1.1 (std. dev. -1.34), and mean ΔInfT is 107.53 seconds (std. dev. 58.57).

• Batch-size behavior: Both models improve with larger batches up to a point, then latency rises. IndicXLIT shows a sharp U-shaped curve with a narrow optimal batch window, whereas NADIR is near-optimal over a much wider range.

Methodology in Plain English

The pipeline starts with preprocessing: a tokenizer, learnable token embeddings, and rotary positional embeddings (RoPE). The core is a stack of four encoder layers, each built from a Differential Transformer layer plus an MoE routing block, followed by a lightweight MLP-based decoder that emits all output characters in parallel.

Differential attention splits the query and key projections into two halves each, computes two separate attention score maps via softmax, and subtracts the second from the first before multiplying by the values. A learnable scalar λ, parameterized from learnable vectors and an initial bias, controls how much of the subtracted map is applied. The intuition given in the paper: a subtractive head can carve away an incorrect competing interpretation (for instance, removing a "ka" feature from a "ksha" representation) so the final vector points more sharply at the correct output. The authors use RMSNorm in the differential attention block because it empirically outperformed GroupNorm.

Mixture-of-Experts replaces the single feed-forward network in each encoder layer with five smaller expert FFNs. A learned router assigns routing probabilities over experts via a softmax over logit scores, and only the top-2 experts are used per token, with the output being the probability-weighted combination of those two experts. The authors report that an earlier hardcoded routing scheme based on linguistic or regional knowledge helped low-resource languages but was limited by small expert capacity and did not scale, motivating learned routing.

Termination and training. Every target sequence gets an appended [EOS] token, and the loss is computed only over positions up to and including the first predicted [EOS]. The training objective is a weighted sum of token-level cross-entropy (for local accuracy) and a load-balancing loss (to keep expert usage uniform), with the weights α = 0.8 and β = 0.2. The paper deliberately avoids CTC loss, citing its rigid monotonic alignment and conditional independence assumptions, and avoids a separate length predictor, which it identifies as an instability source in prior NAR systems.

Setup. Two models were trained, one per direction, for 100 epochs with 8 attention heads and 5 experts per encoder layer, expert_dim 512 and embed_dim 768, approximately 27 million total parameters. Optimization used AdamW with learning rate 1×10⁻³ and weight decay 1×10⁻³, a linear scheduler with 15% warmup, dropout 0.1, and capacity factor 1.25. Training ran on two NVIDIA RTX 3090 GPUs; inference used a single GPU with batch size 8192.

Data. Evaluation used Aksharantar, described as the largest open-source parallel transliteration dataset for Indian languages, covering 21 Indic languages. Training sets range from Malayalam (4.1M), Tamil (3.2M), Kannada (2.9M), and Telugu (2.4M) at the high end, through Gujarati, Hindi, and Bengali (each over 1M), Urdu (699k), Konkani (612k), Panjabi (514k), and Oriya (346k), down to low-resource languages including Manipuri (106k), Sindhi (59k), Kashmiri (46k), and Bodo (35k). The full dataset comprises 24.8 million training, 129.6k validation, and 180.1k test samples. The results table reports 20 languages.

Error measurement. Insertion, substitution, and omission errors were quantified with the editdistance algorithm. Repetition errors were split into Insert Repeat (repeated spans absent from the ground truth), Substitute Repeat (repeated spans replacing expected content but not in the ground truth), and Valid Repeat (ground-truth spans repeated more than necessary), counting character spans from bigrams to four-grams and counting each distinct repeated span once regardless of frequency.

Why This Matters

Impact on research. The paper challenges the default assumption that strong sequential inductive biases are required for all sequence-to-sequence tasks. It shows that two architectural interventions — noise-suppressing differential attention and dynamic expert routing — can make a fully parallel model competitive on a task where local phonetic consistency matters, and it provides a reusable taxonomy and measurement protocol for NAR hallucinations that other NAR research can adopt.

Real-world applications:

• Real-time Indic transliteration keyboards and input methods, where per-keystroke latency matters and the reported ~1005 words/second throughput at large batch sizes is well beyond interactive requirements.
• Large-scale corpus and entity-name transliteration pipelines, such as building parallel name indexes across scripts from 180.1k-word test sets up to full web-scale corpora.
• Search and content discovery across script boundaries, where query terms typed in Roman script must be matched against Indic-script documents.
• Accessibility and language-preservation tools for lower-resource languages in the Aksharantar set, where inference budgets are tight and the reported robustness to batch-size variation reduces tuning burden.

Industry relevance. The paper frames its contribution explicitly around "real-time, large-scale deployment" and "resource-constrained or large-scale multilingual settings." The wide near-optimal batch window is a practical operational advantage over IndicXLIT's narrow optimal window, since production services cannot always tune to a single sweet spot. The 27 million parameter count and single-GPU inference setup indicate a system that can be served without the hardware footprint of a large autoregressive model.

Future Directions

• Extending beyond transliteration. The authors state an intent to extend NADIR to tasks beyond transliteration and to the other task families named in the abstract — code refactoring, grammatical correction, and text normalization — which they argue also rely on local dependencies.

• Absorbing newer MoE advances. The conclusion notes a focus on incorporating recent advancements in Mixture-of-Experts into the architecture, suggesting the current 5-expert, top-2 design is not the endpoint.

• Addressing the omission trade-off. Adding MoE reduced insertions, substitutions, and repetitions but produced an approximately 8% increase in omission errors. Closing that gap is an open problem the ablation surfaces but does not solve.

• Generalizing implicit termination. The paper acknowledges that its implicit [EOS]-based length control works best for tasks with clearly defined termination points and will not generalize to tasks with highly ambiguous or variable-length outputs — leaving the question of how to handle such outputs in a fully parallel model.

Target Audience

This paper suits NLP engineers and researchers working on efficient inference, non-autoregressive generation, or multilingual Indic language technology. It is also relevant to practitioners deploying transliteration or other local-dependency sequence tasks in latency- or throughput-constrained production systems, and to researchers who need a concrete, quantified starting point for studying NAR hallucination and mitigation strategies.

Authors’ abstract

In this work, we argue that not all sequence-to-sequence tasks require the strong inductive biases of autoregressive (AR) models. Tasks like multilingual transliteration, code refactoring, grammatical correction or text normalization often rely on local dependencies where the full modeling capacity of AR models can be overkill, creating a trade-off between their high accuracy and high inference latency. While non-autoregressive (NAR) models offer speed, they typically suffer from hallucinations and poor length control. To explore this trade-off, we focus on the multilingual transliteration task in Indic languages and introduce NADIR, a novel NAR architecture designed to strike a balance between speed and accuracy. NADIR integrates a Differential Transformer and a Mixture-of-Experts mechanism, enabling it to robustly model complex character mappings without sequential dependencies. NADIR achieves over a 13x speed-up compared to the state-of-the-art AR baseline. It maintains a competitive mean Character Error Rate of 15.78%, compared to 14.44% for the AR model and 21.88% for a standard NAR equivalent. Importantly, NADIR reduces Repetition errors by 49.53%, Substitution errors by 24.45%, Omission errors by 32.92%, and Insertion errors by 16.87%. This work provides a practical blueprint for building fast and reliable NAR systems, effectively bridging the gap between AR accuracy and the demands of real-time, large-scale deployment.