Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors

AI summary

Overview

Research area: Natural language processing; robustness evaluation of large language models; multilingual NLP; typographical error (noise) modeling.

Technical level: Intermediate. The paper is readable without deep technical background, but the evaluation design (multiple model families, task types, typo rates, and prompting regimes) and the MulTypo algorithm's sampling details require familiarity with standard LLM benchmarking practice.

Scope: The paper introduces MulTypo, a language-specific keyboard-aware typo generation algorithm, and uses it to measure how 18 open-source LLMs from three model families degrade across five multilingual downstream tasks under simulated human-like typographical errors.

What This Paper Is About

Most LLM benchmarks feed models clean, error-free text, even though real users typing on real keyboards constantly make typos. The authors argue this means reported performance likely overstates real-world reliability, and that almost all prior robustness work has been English-centric and based on edit-distance heuristics rather than actual typing behavior. Their goal is to build a realistic multilingual typo generator and then systematically measure how sensitive current open-source LLMs are to that noise across languages, tasks, model sizes, and training strategies.

Key Contributions

1. MulTypo, a multilingual typo generation algorithm that simulates realistic human-like errors using language-specific keyboard layouts and typing behavior (10-finger typing convention, horizontal key neighbors, position-dependent error likelihood), and validates its realism through human evaluation against a naive baseline that ignores keyboard layouts.
2. A comprehensive robustness evaluation suite spanning 18 open-source LLMs from three families (Gemma, Qwen, OLMo), each in base and instruction-tuned form, across five downstream tasks covering language inference, multi-choice question answering, mathematical reasoning, and machine translation.
3. Fine-grained robustness analysis under both zero-shot and few-shot prompting (0, 1, 3, and 5 examples) and under varying typo rates (0, 0.1, 0.4, 0.7), including cross-lingual comparisons and translation direction (from English vs. into English).
4. A released Python package and source code for MulTypo at https://github.com/cisnlp/multypo, to support further multilingual robustness research.

Main Findings

• Typos consistently degrade performance across all models. Across three model families and all tasks, even minor typographical noise impairs performance. For example, Qwen achieves over 50 on Belebele in the clean setting but drops to around 45 with just a 10% typo rate, and performance declines continuously as noise increases.

• Robustness varies substantially by task type. XNLI (natural language inference) is comparatively robust: Qwen's performance remains nearly unchanged under 10% noise, and even the primarily monolingual OLMo models sustain less than a 10-point absolute drop at the highest noise level. Generative reasoning tasks are far more fragile: Qwen's accuracy on MGSM plummets from around 40 (clean) to around 27 at 70% noise.

• Model size helps but does not immunize. At a 10% typo rate, Gemma's relative drop decreases from 9.9% (Small) to 3.7% (Large); OLMo goes from 9.5% (Small) to 4.3% (Large); Qwen shows 5.7% (Small), 8.2% (Medium), 5.7% (Large). Larger models score higher overall and generally degrade less, but all sizes suffer under noise.

• Instruction tuning improves clean-input performance but not robustness. Instruction-tuned models outperform base models on clean input, yet in many cases their absolute degradation under 10% or 40% noise is as severe as or worse than their base counterparts. On MGSM, Gemma's instruction-tuned models drop from around 48 to 33 under 40% corruption.

• More few-shot examples do not buy robustness. Increasing demonstrations from 0 to 1 to 3 to 5 yields consistent performance gains up to 3 shots for Gemma and Qwen, but the performance gap between clean and noisy inputs remains nearly unchanged regardless of shot count. For OLMo, adding examples does not help and occasionally harms performance.

• Robustness is language-dependent. Aggregating across tasks for languages supported by at least three datasets, English shows among the lowest relative drops (Gemma -2.0%, Qwen 5.6%, OLMo 3.0%), while Arabic, Hindi, and Bengali tend to degrade more (e.g., Bengali: 10.6% for Gemma, 17.5% for Qwen). Russian, despite being high-resource, shows sharp degradation (11.9% for Gemma, 15.8% for OLMo).

• Translation direction matters. On Flores200 with Gemma models, translation from English is more robust than translation into English.

• MulTypo's degradation pattern sits between a naive baseline and WikiTypos. In a comparison using gemma-3-4b-it on four overlapping languages (English, German, French, Hindi) across five tasks, models appear relatively more robust to MulTypo and WikiTypos perturbations than to the naive baseline, which the authors attribute to those typo types better reflecting realistic human errors possibly already seen in pretraining data.

• Findings generalize to harder reasoning benchmarks. On multilingual AIME (AIME 2025 + AIME 2026, 60 math problems per language) with the Qwen3 family across five languages, accuracy generally decreases as the typo rate increases; larger models remain more robust, and Bengali and Russian show the largest drops.

• Human evaluation supports MulTypo's realism. In six of seven languages, MulTypo typos were judged significantly more natural than the naive baseline (at least p < 0.05); Arabic was the only exception, with slightly higher ratings for the baseline but no statistical significance.

Methodology in Plain English

The authors built a typo generator that mimics how people actually mistype on their own language's keyboard. Instead of randomly swapping characters, MulTypo consults a keyboard layout database (kbdlayout.info, using Windows version 10.0.27729.1000) and only considers keys immediately adjacent on the same row. It uses the standard 10-finger typing convention (characters such as "5TGB" belong to the left hand, "6YHN" to the right) to determine which hand types each key, and restricts transposition errors to characters typed by different hands. Four typo types are simulated: replacement, insertion, deletion, and transposition.

Sampling is deliberately non-uniform. Words are selected with probability proportional to the square root of their length, reflecting that longer words attract more typos. Within a word, the first character is never selected (weight 0), the second character gets weight 0.1, the final character gets 0.2, and intermediate positions are linearly interpolated. Insertion is sampled with probability 15.25%, while replacement, deletion, and transposition each get 28.25%. After each typo, the word's sampling weight is halved to spread errors out. A language-specific "ignoring string set" excludes numerical expressions, in both digit and word form, so that benchmark results reflect typo robustness rather than altered numeric values. A validity check prevents self-cancelling edits (for example, replacing a character and then replacing it back).

To validate realism, the researchers sampled 30 sentences per language from Flores200, with 15 corrupted by MulTypo and 15 by a naive baseline, balanced across corruption levels 0.1, 0.4, and 0.7 (five sentences per level). At least 15 participants per language judged each sentence as natural or unnatural across seven languages: Arabic, German, Greek, English, French, Hindi, and Russian.

The evaluation covered 12 languages spanning 7 language families and 7 scripts, and 18 decoder-only models: gemma-3-1b-pt, gemma-3-4b-pt, gemma-3-12b-pt; Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base; and OLMo-2-0425-1B, OLMo-2-1124-7B, OLMo-2-1124-13B, plus each model's instruction-tuned version. Six datasets spanning four task types were used: XNLI for natural language inference; Belebele and MMMLU for multiple-choice question answering; MGSM plus Arabic and Hindi adaptations of GSM8K for mathematical reasoning; and FLORES200 for machine translation. Typos were injected only into dataset instances, never into task instructions. Three-shot prompting was the default.

Why This Matters

Impact on research. The paper moves robustness evaluation away from English-only, heuristic character noise toward empirically grounded, language-specific typing behavior, and provides a validated generator plus a reproducible evaluation suite. It documents a gap between clean-input benchmark numbers and noisy-input behavior that the authors argue leads to overestimated real-world performance, and it supplies a controlled alternative to real-world typo datasets like WikiTypos, which also contain semantic substitutions beyond typographical errors.

Real-world applications:

• Chatbots and conversational assistants that receive typed user messages in many languages, where a single keystroke slip can change or break an answer.
• Machine translation tools, particularly because the paper finds translation from English is more robust than translation into English.
• Search engines and text-entry interfaces that must parse imperfect user queries across different keyboard layouts.
• Mathematical and reasoning assistants, where the paper shows multi-step reasoning is disrupted more severely than classification, and where number-related strings are deliberately protected to isolate typo effects.

Industry relevance. Deployment teams choosing between base and instruction-tuned models, or between model sizes, can use these results to set expectations: scaling and instruction tuning improve clean-input quality but do not reliably improve noise tolerance, and adding more few-shot demonstrations does not close the robustness gap. The findings also flag that low-resource and non-Latin-script user populations face greater degradation, which is directly relevant to fairness and reliability of globally deployed services.

Future Directions

• Extending language and script coverage. MulTypo currently requires manually specified keyboard layouts and typing conventions for each new language, and does not support logographic or syllabic writing systems such as Chinese, which are typed through phonetic systems like Pinyin rather than direct keypresses. Modeling Pinyin mistyping or candidate misselection would require a different corruption strategy.
• Improving fidelity for specific languages. The Arabic human evaluation was the one case where MulTypo was not judged significantly more natural than the naive baseline, suggesting the algorithm may not capture all language-specific properties equally well.
• Covering more input modalities. The work focuses exclusively on physical keyboards (e.g., QWERTY) and ignores touchscreen keyboards on mobile devices, where typing behavior, error distributions, and auto-correct interference differ substantially.
• Noise-aware training and evaluation. The authors call for noise-aware multilingual pretraining, evaluation, and human-centric error modeling, since current tuning methods prioritize clean prompts and may underprepare models for noisy real-world input.

Target Audience

Researchers and practitioners in multilingual NLP and LLM evaluation who need to understand model reliability under realistic user noise; engineers deploying multilingual chatbots, translation systems, or reasoning assistants who must weigh model size and instruction tuning against robustness; and benchmark designers interested in how typographical error simulation should be grounded in keyboard layouts, typing conventions, and human validation rather than edit-distance heuristics.

Authors’ abstract

Large language models (LLMs) are increasingly deployed in multilingual, real-world applications with user inputs -- naturally introducing \emph{typographical errors} (typos). Yet most benchmarks assume clean input, leaving the robustness of LLMs to typos across languages largely underexplored. To address this gap, we introduce MulTypo, a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. We evaluate 18 open-source LLMs across three model families and five downstream tasks spanning language inference, multi-choice question answering, mathematical reasoning, and machine translation tasks. Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring reasoning -- while the natural language inference task is comparatively more robust. Instruction tuning improves clean-input performance but may increase brittleness under noise. We also observe language-dependent robustness: high-resource languages are generally more robust than low-resource ones, and translation from English is more robust than translation into English. Our findings underscore the need for noise-aware training and multilingual robustness evaluation. We release a Python package for MulTypo and make the source code publicly available at https://github.com/cisnlp/multypo.