Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation

AI summary

Overview

• Research area: Multilingual natural language processing, specifically LLM-generated counterfactual explanations and counterfactual data augmentation (CDA).
• Technical level: Intermediate — the paper is an empirical study built on standard, well-documented metrics (label flip rate, perplexity, embedding similarity) and open-source models, so no specialized theory is required, though familiarity with text classification and prompting helps.
• Scope: A systematic evaluation of how well three open-source LLMs (Qwen2.5-7B, Gemma3-27B, Llama3.3-70B) generate counterfactual examples across six languages (English, Arabic, German, Spanish, Hindi, Swahili) on two multilingual datasets (XNLI and SIB200), covering generation quality, cross-lingual edit patterns, error types, and downstream effects of counterfactual data augmentation.

What This Paper Is About

Counterfactual examples are minimally edited inputs that change a model's prediction, and they are a promising way to explain black-box model behavior. Almost all existing work on generating and evaluating them has been done in English, even though LLMs claim broad multilingual competence, so it is unclear whether LLMs can produce good counterfactuals in other languages. This paper runs a large-scale empirical study comparing counterfactuals generated directly in six target languages against counterfactuals produced by translating English counterfactuals, then probes what edits look like across languages, what errors occur, and whether the resulting data actually helps train better models.

Key Contributions

1. A six-language, two-dataset, three-model evaluation of multilingual counterfactuals. The authors compare directly generated counterfactuals (DG-CFs) in the target language with translation-based counterfactuals (TB-CFs) derived from English, using label flip rate (LFR), textual similarity (TS), and perplexity (PPL; computed with mGPT-1.3B rather than the English-only GPT-2 used in prior work).

2. A cross-lingual edit similarity analysis. Using a multilingual SBERT encoder, the paper measures pairwise cosine similarity between counterfactuals across languages, both directly and after back-translation into English, to test whether perturbations follow shared strategies across languages.

3. A taxonomy of four recurring error types. The authors identify and illustrate copy-paste, negation, inconsistency, and language confusion errors in generated multilingual counterfactuals, with quantified rates for the two measurable categories.

4. A counterfactual data augmentation study. Both cross-lingual CDA (English-only base data) and multilingual CDA (all six languages) are tested with a fine-tuned multilingual BERT, including analysis of how excluding error cases affects the gains. Code and evaluation results are released at https://github.com/qiaw99/multicfe.

Main Findings

• English counterfactuals usually flip labels most often. On both XNLI and SIB200, English counterfactuals tend to achieve the highest LFR. On XNLI, the gap between high-resource and low-resource languages widens as model scale grows, reaching up to 16.46 percentage points. Llama3.3-70B reached 50.88% LFR on English XNLI, and 92.16% was the highest SIB200 LFR recorded (Qwen2.5-7B, English).

• SIB200 is much easier to flip than XNLI. LFR is dramatically higher for all three models on SIB200 than on XNLI, which the authors attribute to the greater inherent difficulty of natural language inference. On SIB200 the high-resource/low-resource LFR gap narrows; Swahili counterfactuals from Llama3.3-70B attained the highest LFR in that setting.

• No language or model is best on every metric. Hindi counterfactuals consistently achieved the best (lowest) perplexity across all three models, while Arabic counterfactuals were generally less fluent and involved larger edits (lower textual similarity). Swahili and German counterfactuals were less edited, but Swahili's higher similarity reflected fewer edits and went along with lower LFR. The authors conclude that "optimal" language quality is contextual and metric-dependent.

• Translation-based counterfactuals flip labels more often than directly generated ones, but cost more. TB-CFs generally achieved higher LFR than DG-CFs in the target language, with the improvement most pronounced for German and least significant for Hindi. However, TB-CFs still fell short of the original English counterfactuals they were translated from, were 15.44% less similar to the input on average because of machine-translation artifacts, and were 38% less fluent on average.

• Translation quality only weakly predicts counterfactual quality. The LFR improvement was weakly positively correlated with machine translation quality, with Spearman's rho = 0.27 for automatic evaluation and rho = 0.07 for human evaluation. The authors interpret this as evidence that gains come mainly from the quality of the English counterfactuals rather than from translation quality.

• European-language edits look alike; Swahili and Arabic edits diverge. Counterfactuals in English, German, and Spanish showed a high degree of similarity in which words were edited, suggesting shared cross-lingual perturbation strategies, whereas Swahili and Arabic counterfactuals were edited substantially differently, with lower cosine similarity scores. Edit patterns also differed between SIB200 and XNLI, which the authors link to topic classification offering more distinct ways to build valid counterfactuals once a target label is specified.

• Copy-paste is the most common error. Four error types were catalogued: copy-paste, negation, inconsistency, and language confusion. The copy-paste rate averaged 6.7% on SIB200 versus 2.1% on XNLI, with high-resource languages such as English and Spanish more affected on SIB200 and lower-resource languages such as Hindi and Swahili more affected on XNLI. Counterfactuals in high-resource German, English, and Spanish were usually produced in the intended language, whereas targeting Arabic or Swahili frequently led models to misinterpret prompts or default to English.

• Multilingual CDA beats cross-lingual CDA, but gains are uneven. Multilingual CDA generally improved model performance more than cross-lingual CDA on both datasets, with the largest reported effect for Arabic (the paper reports average accuracy gains of 64.45%) and the smallest for English, which already performed well in the cross-lingual setting. On XNLI, cross-lingual CDA improved only English and degraded the other languages, while multilingual CDA improved all languages except Swahili. On SIB200, cross-lingual CDA was generally harmful, and multilingual CDA yielded reliable gains only for English and Spanish while consistently hurting Swahili.

• Imperfect counterfactuals limit augmentation payoffs. Even when counterfactuals for SIB200 flipped model predictions, they often failed to flip the ground-truth labels because of insufficient revision, producing noisy labels that can degrade performance. After excluding error cases (copy-paste and language confusion), overall performance improved, but by varying amounts across languages.

Methodology in Plain English

The authors pick an existing, well-established counterfactual generation recipe rather than designing a new one, so that they can characterize multilingual behavior rather than chase a state-of-the-art score. That recipe uses one-shot chain-of-thought prompting and asks the LLM to do three things: identify the words in the input that most influence the model's prediction, propose replacements likely to push the prediction toward a target label, and substitute those words in. Prompts are written in English only, which the authors justify by noting that LLMs remain English-centric and that English prompts avoid confounding factors. No human intervention or extra LLM training is needed.

For the direct condition, the same three-step process is run in the target language. For the translation condition, the LLM first generates a counterfactual in English and then translates it into the target language with the same model.

Two datasets are used: XNLI (cross-lingual natural language inference, built by extending English MultiNLI into 14 additional languages, with entailment/contradiction/neutral labels) and SIB200 (topic classification over 205 languages with seven topics). Six languages overlap between them and are selected for typological and script diversity.

Generation uses three open-source instruction-tuned models of increasing size — Qwen2.5-7B, Gemma3-27B, and Llama3.3-70B — while the model being explained is a fine-tuned multilingual BERT. Quality is measured with label flip rate (how often the prediction changes), textual similarity via a pretrained multilingual SBERT embedding, and perplexity via mGPT-1.3B (used instead of GPT-2 because GPT-2 is English-only). Cross-lingual edit similarity is measured by comparing counterfactual embeddings across languages, both directly and after back-translating into English. For the augmentation experiments, the base model is fine-tuned on English data (cross-lingual CDA) or on all six languages (multilingual CDA), then retrained with the corresponding counterfactuals added. Errors are examined both manually and automatically, with counterfactuals translated into English where needed for comparison.

Why This Matters

Impact on research. Counterfactual explanation research has been overwhelmingly English-only, and this paper shows that the standard metrics do not transfer cleanly: validity, fluency, and minimality trade off differently across languages, and no single language or model dominates all three. It also shows that a translation-based pipeline — the obvious shortcut for extending English methods to other languages — does not match the quality of the English counterfactuals it starts from, and that machine translation quality is only weakly correlated with counterfactual validity. The released error taxonomy gives future work a concrete checklist of failure modes.

Real-world applications:

• Building explanation interfaces for multilingual content moderation and classification systems, where stakeholders may need to see why a post or document was labeled a certain way in their own language.
• Auditing multilingual models for behavioral differences between languages, since the paper documents systematic variation in how easily predictions can be flipped.
• Generating training data for low-resource languages, where the paper finds the largest CDA gains but also the largest risk from noisy or erroneous counterfactuals.
• Evaluating translation pipelines: the finding that translation artifacts inflate edits and reduce fluency is directly relevant to anyone using translation as a data-production shortcut.

Industry relevance. Companies deploying multilingual classifiers need explanations that are valid in each supported language, not just in English. This study shows that directly prompting an LLM in the target language and translating from English produce measurably different results, and that the cheaper path (translation) costs extra edits and fluency. The documented language-confusion rates for Arabic and Swahili also flag a concrete reliability risk for products marketed as multilingual.

Future Directions

• Better multilingual similarity metrics. The authors note that the embedding-based similarity they use may not fully capture similarity for non-Latin scripts, and that token-level Levenshtein distance — a common alternative — has the same weakness. They call for new token-level textual similarity metrics designed for multilingual settings.
• Post-training instead of translation. Since machine translation quality is only weakly correlated with counterfactual validity improvements, the authors suggest that translation-based generation may not be optimal and that post-training methods such as MAPO could substantially improve multilingual counterfactual quality.
• Subjective evaluation through user studies. The study restricts itself to automatic measures of validity, fluency, and minimality. The authors plan a user study to assess aspects such as usefulness, helpfulness, and coherence, which they say can only be judged subjectively.
• Broader coverage. The authors explicitly limit themselves to six languages and three models, and list expanding to more languages and to models of different architectures and sizes as future work.

Target Audience

Researchers and practitioners in multilingual NLP, explainable AI, and LLM evaluation who work on counterfactual explanations, model interpretability, or data augmentation. It is also useful for engineers building or auditing multilingual classification systems who need to know whether LLM-generated explanations hold up outside English, and for anyone considering translation as a shortcut for producing multilingual training or explanation data. Readers with a basic grasp of text classification, prompting, and standard evaluation metrics will follow it comfortably; the paper assumes no specialized background beyond that.

Authors’ abstract

Counterfactuals refer to minimally edited inputs that cause a model's prediction to change, serving as a promising approach to explaining the model's behavior. Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency. However, their effectiveness in generating multilingual counterfactuals remains unclear. To this end, we conduct a comprehensive study on multilingual counterfactuals. We first conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages. Although translation-based counterfactuals offer higher validity than their directly generated counterparts, they demand substantially more modifications and still fall short of matching the quality of the original English counterfactuals. Second, we find the patterns of edits applied to high-resource European-language counterfactuals to be remarkably similar, suggesting that cross-lingual perturbations follow common strategic principles. Third, we identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages. Finally, we reveal that multilingual counterfactual data augmentation (CDA) yields larger model performance improvements than cross-lingual CDA, especially for lower-resource languages. Yet, the imperfections of the generated counterfactuals limit gains in model performance and robustness.