Research
The Narrative Gap: Can LLMs Help Us Navigate Diverse Narratives Across Languages?
People increasingly ask LLMs about contested topics, but the accounts available about those topics differ by language and region — what the authors call "narratives."
- arXiv
- 2611.12750
- Published
- 2026-11-09
- Authors
- Nikhil Sharma, Kelly Marchisio, Kenton Murray, Ziang Xiao
AI summary
Overview
Research area: Natural Language Processing — multilingual LLMs, information retrieval and access, bias and pluralism in AI-mediated information seeking.
Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with benchmark evaluation, retrieval-augmented generation, and precision/recall-style metrics.
Scope: The paper defines six capabilities LLMs need to help users navigate competing narratives across languages, introduces the NarrativeBench benchmark (1,333 news articles, 23 publication languages, 23 contested topics, 460 queries), and evaluates 14 LLMs across 24 query languages.
What This Paper Is About
People increasingly ask LLMs about contested topics, but the accounts available about those topics differ by language and region — what the authors call "narratives." The paper asks whether LLMs can actually help users reach beyond their own linguistic and regional information bubble: whether a model can include a perspective you ask for, exclude one you don't want, tell you which perspectives its answer left out, and present diverse views even when your question is built to confirm what you already believe. The authors build a benchmark to measure these behaviors and find that current models largely fail at them.
Key Contributions
-
A capability framework for "diverse multilingual information seeking." The authors define three foundational capabilities — narrative inclusion (FC-1), narrative exclusion (FC-2), and missing narrative identification (FC-3) — and three higher-level ones: presenting diverse narratives (HC-1), preserving narrative coverage under confirmatory queries (HC-2), and maintaining information parity across languages (HC-3).
-
NarrativeBench: an ecologically grounded dataset and benchmark. 1,333 news articles from 43 countries, regions, and regional organizations, spanning 23 contested topics and 23 publication languages, grouped into 180 narrative groups, with 460 queries (283 gathered from Reddit plus 177 context-specific adaptations), split into 230 neutral and 230 confirmatory queries, and translated into 24 languages for 11,038 query–language versions.
-
An empirical audit of 14 models under 15 configurations. The evaluation covers 4,800 cases and 72,000 scored responses, comparing model size, base versus instruction-tuned models, and reasoning settings.
-
A diagnosis of specific failure modes: weak selective narrative control, incomplete coverage even under explicit diversity requests, framing-dependent coverage gaps, and uneven performance across languages.
Main Findings
-
Selective narrative control is weak. Narrative inclusion (asking a model to present one specific perspective and exclude others) reaches a mean F1 of only 19.9% (SE = 0.6%) across the 15 configurations, versus 49.3% (SE = 0.6%) for narrative exclusion. Inclusion F1 is lower than exclusion F1 in every configuration. The best inclusion score is 52.5% (SE = 1.3%) from Gemini 3.8 Flash; the best exclusion score is 74.2% (SE = 0.7%) from Gemini 2.5 Pro.
-
Recovering omitted perspectives is the strongest foundational capability but still imperfect. Missing narrative identification reaches a mean recall of 56.8% (SE = 0.6%); Gemini 3.8 Flash is highest at 82.8% (SE = 0.7%).
-
Diverse prompting helps only modestly. Narrative recall averages 44.3% (SE = 0.7%) under the baseline prompt and 49.2% (SE = 0.7%) under the diverse prompt. The best configuration, Gemini 2.5 Pro, reaches 70.6% (SE = 0.9%) under baseline and 74.9% (SE = 0.8%) under the diverse prompt — meaning even the strongest model leaves roughly one quarter of available narratives unrepresented after an explicit request for diversity.
-
Coverage shifts with how the question is framed. The mean absolute difference in recall between neutral and confirmatory queries is 3.23 percentage points, ranging from 0.32 points (Qwen3.5 9B IT) to 10.09 points (Gemini 3.8 Flash). Task-level gaps are largest for narrative inclusion F1 (3.41 points), then baseline-prompt recall (3.32), diverse-prompt recall (3.30), exclusion F1 (2.72), and missing-narrative recall (2.43). The paper's Greenland example shows a response can cite more narratives (3/6 → 5/6) while still endorsing the query's premise.
-
Access is uneven across languages. Mean absolute gaps from English are 8.74 percentage points for query languages and 7.51 percentage points for target narrative languages. Query-language gaps span 3.59 points (Kimi K2.6) to 13.84 points (GPT-OSS 20B, low reasoning); target narrative-language gaps span 2.76 points (Qwen3.5 0.8B IT) to 10.72 points (Gemini 3.8 Flash).
-
Some source languages are systematically dropped. Under diverse prompting, omission rates for Tigrinya, Nepali, and Armenian sources exceed English by 59.8, 57.9, and 54.6 percentage points respectively. Models omitted available sources less often when the query language matched the source language.
-
Service-level refusals also shape access. The Kimi K2.6 service rejected 552 of 4,800 requests as high risk, concentrated in topics including India–China tensions, Taiwan, and student protests in India.
-
The corpus does capture distinguishable narratives. Same-group articles were more similar in multilingual embedding space than different-group articles (mean cosine similarity difference 0.047, 95% bootstrap CI [0.044, 0.051], permutation p < 0.001); restricting to same-language comparisons preserved the effect (difference 0.012, 95% CI [0.009, 0.015]).
Methodology in Plain English
The authors start by defining a "narrative" as a culturally situated account of events, operationalized as a topic, a publication language, and a source perspective anchored in a region. They then collect news coverage of 23 contested topics (including Israel–Palestine, India–Pakistan, Russia–Ukraine, Taiwan, AI regulation, and UK immigration) using localized searches with region-specific terms in local languages across search engines such as Google Advanced Search, Yandex, and Baidu. Articles are screened for relevance, accessible text, and identifiable sources, then grouped into 180 narrative groups.
To check the groups are meaningfully different, they embed all 1,333 articles and test whether same-group articles are more similar than cross-group ones within a topic.
For queries, they harvest real questions people asked on Reddit about these topics and adapt the format to under-covered topics, then split queries evenly into neutral and confirmatory versions. Queries are machine-translated into 24 languages using Google Translate and validated with COMETKiwi scores plus human ratings.
In each evaluation case, the model receives a document context (three articles per selected narrative), a user query, and a task instruction, and must answer in the query language while citing document IDs. Narratives count as "present" only if the response cites at least one article from that narrative — a design that makes the measurement auditable, since article IDs like [D001] leak no region or language information. Precision, recall, and F1 are computed against the set of narratives each task requires, and an answer in the wrong language scores zero.
Why This Matters
The paper argues that an LLM that appears to follow instructions while silently omitting perspectives creates an illusion of control — users cannot ask for what they do not know exists. Because these gaps are uneven across languages, the result risks reinforcing linguistic filter bubbles, echo chambers, and information disparities that erode shared ground for democratic discourse.
Real-world applications:
- AI search and chat assistants answering questions about geopolitical conflicts, where users in different regions currently receive different accounts of the same events.
- News aggregation and recommendation interfaces that need to surface missing perspectives rather than amplify what a user already believes.
- Scientific literature synthesis, where disciplines describe the same phenomenon with different terms and evidentiary standards, and an assistant drawing only on easily accessible literature can make a field look more settled than it is.
- Policy and organizational decision-making, where technical teams, regulators, and affected communities explain the same issue differently and regional teams hold evidence absent from headquarters' records.
Industry relevance: The findings give model developers concrete training and evaluation targets (include, exclude, recover, diversify, hold under confirmatory framing, and equalize across languages), and give procurement and policy teams a reporting standard for narrative-access disparities. The paper also notes that safety refusal policies differ across providers, so comparable access across services matters.
Future Directions
- Finer-grained narrative representation. The current grouping collapses disagreement within a region or language; extending it would capture how accounts within the same group differ in the events, causes, and consequences they emphasize.
- Broader and multimodal sampling. The corpus reflects what localized web searches surface in text; sampling across media formats, regions, and harder-to-reach information streams would test how robust the gaps are.
- Source quality and misinformation. Contested reporting can contain propaganda and selective framing, so future work should examine whether models can preserve competing perspectives while distinguishing reliable evidence from unsupported claims.
- Human-subject and multi-turn studies. Measuring whether improved narrative coverage actually broadens what users consider requires user studies, including how people selectively attend to belief-confirming perspectives, and how models behave when intents evolve across a conversation.
Target Audience
Researchers and practitioners in multilingual NLP, LLM evaluation, and information retrieval; trust-and-safety and responsible-AI teams auditing model behavior on contested topics; designers of AI search and conversational information systems; and policy or procurement staff who need measurable standards for cross-linguistic information access. Readers focused on the societal effects of LLMs will find the "illusion of control" framing and the language-disparity results the most directly useful parts.
Authors’ abstract
Large language models (LLMs) increasingly mediate how people seek and consume information, including accounts that differ across languages and regions, especially on contested topics. Supporting diverse information seeking requires models to help users navigate competing narratives across languages and information streams. We define six capabilities necessary for diverse multilingual information seeking: three foundational capabilities for narrative inclusion, narrative…
The full abstract on arXiv