AtlasNLP: A Country-Aware Atlas of Dataset Representation in NLP

AI summary

Overview

• Research area: Natural Language Processing (NLP), specifically dataset documentation, data provenance, and geographic/population representation in NLP resources.
• Technical level: Intermediate. The conceptual framing (represented vs. producer geography) is accessible to beginners, while the extraction pipeline and validation metrics assume familiarity with dataset curation and LLM-based annotation.
• Scope: The paper introduces AtlasNLP, a country-aware atlas of NLP dataset records that tracks which countries and populations datasets represent and where those datasets are produced, built from human curation plus an LLM-assisted pipeline over ACL Anthology.

What This Paper Is About

NLP datasets are usually organized and described by language, task, or benchmark, and geographic metadata is rarely recorded, so it is hard to tell which countries and populations are actually represented. Because languages span many countries and countries contain many language communities, language-level claims can make geographic coverage look broader than the evidence supports. AtlasNLP addresses this by building a structured, country-level resource that separates the countries a dataset represents from the institutional countries where it was produced, then using it to measure coverage gaps across countries and tasks.

Key Contributions

1. Releases two country-aware resources: AtlasNLP-Core, with 13,462 ACL-derived paper-level dataset records, and AtlasNLP-Gold, with 1,480 human-curated entries corresponding to 989 normalized dataset-name groups, both using a shared schema covering represented countries, producer countries, task category, language, modality, licensing, and attribution method.
2. Documents strong unevenness in coverage: 79.2% of country-task pairs contain no dataset records with documented explicit country representation (75.0% under explicit+inferred attribution), and coverage is long-tailed across 158 of 197 countries represented explicitly.
3. Separates representation from production: shows that many countries are represented largely through datasets produced outside them; among countries with at least 10 represented records with producer metadata and 10 producer-representation associations, 39 of 62 (62.9%) have content self-representation below 0.5.
4. Shows dataset availability tracks research infrastructure: coverage aligns with World Bank income groups and national university counts (Pearson r = .65, Spearman ρ = .61 under explicit attribution), and demonstrates that language coverage does not imply geographic coverage.

Main Findings

• Coverage is highly uneven and long-tailed: The 2,447 explicitly attributed Core dataset records yield 4,421 country-record associations across 158 of 197 countries, with the United States, China, India, the United Kingdom, and Germany accounting for 36.4% of those associations. 39 countries have no explicitly attributed dataset records, and 121 of 197 countries have ten or fewer represented dataset records.
• Task-level coverage is sparse: 79.2% of country-task pairs have no explicitly attributed dataset records, dropping only to 75.0% when inferred attribution is included.
• Missing country attribution is systematic and tied to English: Among unattributed Core records, 70.7% include English, versus 49.9% of explicitly attributed records; the authors interpret low or zero coverage as gaps in documented representation in ACL dataset contributions rather than proof that no resources exist.
• Attribution gaps remain large even with inference: Under explicit+inferred attribution, a represented country could not be established for 9,956 of 13,462 Core dataset records (74.0%). Explicit attribution covers 2,447 Core records (18.2%); explicit+inferred covers 3,506 (26.0%).
• Production and representation are asymmetric: The United States accounts for 21.3% of expanded producer-representation associations, and the ten largest producer countries account for 61%. Country examples show the range: China has content self-representation 0.747 and producer self-representation 0.615, the United States 0.597 and 0.265, Denmark 0.786 and 0.278, Nigeria 0.216 and 0.276, Egypt 0.109 and 0.115, Russia 0.140 and 0.700, and Brazil 0.386 and 0.600.
• Dataset availability tracks research infrastructure: Median represented dataset records are substantially higher for high-income countries, and university counts correlate positively with coverage (r = .65, ρ = .61; r = .70, ρ = .67 under explicit+inferred). The authors note the relationship is not deterministic.
• Task portfolios vary widely: Among countries with at least 10 represented records, the median country spans 11 of 30 task categories, with its three most common tasks accounting for 55.6% of records (13 tasks and 54.3% under explicit+inferred). The United States, China, India, and France have broad portfolios (top-3 shares of 0.50, 0.48, 0.46, and 0.44), while Belgium, Bahrain, Israel, and Oman are concentrated (0.81, 0.73, 0.70, 0.70).
• Language does not map onto geography: Chinese-language records are 89.2% associated with China; Brazil and Portugal together account for about 69% of Portuguese records; France, Switzerland, and Canada about 63% of French records; English records are more distributed but the United States, United Kingdom, and India still account for about half.
• Validation results support precision: Against Gold, 225 high-confidence aligned dataset records were matched; 97.7% share at least one country with Gold and 94.7% of Core country assignments are supported by Gold, with 79.9% exact task agreement and 94.0% country-assignment precision when inferred evidence is included on both sides. Producer-country extraction on 100 human-labeled records achieved 96.9% precision and 97.7% recall, with the primary producer country recovered in all audited cases.

Methodology in Plain English

The authors first designed an annotation schema and a shared country list: 197 geopolitical entities made up of the 193 UN member states, two UN observer states, Taiwan, and Kosovo, plus task categories normalized from ACL and EMNLP thematic areas into 30 categories (31 overall, including a Raw Corpus category for Gold).

They then built a human-curated reference set. Contributors from several countries, including Nigeria, China, Peru, the United States, Romania, South Korea, and Ethiopia, gathered datasets for countries they knew well from ACL Anthology, Hugging Face, Papers With Code, GitHub, institutional archives, and regional collections, aiming for up to five datasets per country with attention to less-represented countries. Curation ran in two stages: independent compilation, then shuffled cross-validation where reviewers checked country attribution, task categorization, metadata consistency, duplicates, and provenance.

To scale up, they processed 119,963 ACL Anthology records spanning 1952–2025. Of these, 71,847 had non-empty abstracts, which were tested with ModernBERT-base-NLI against the hypothesis "This paper proposes a dataset." Papers scoring above 0.5 entailment (20,277 candidates) went into a pipeline combining rule-based dataset-role checks with GPT-4o-mini extraction from full text, producing 18,035 successful initial paper-level extractions. A 400-record human audit exposed recurring errors, especially reuse of existing datasets and country assignments based on affiliation, language, or incidental mentions. The authors tightened the criteria and re-evaluated everything using GPT-5.4-mini, retaining records only when a paper introduces a dataset, materially extends one, or compiles a new dataset as a primary contribution, yielding 13,462 records.

Country evidence is graded: explicit (direct evidence linking content, participants, sources, or collection to a country), inferred (plausible relevance from indirect signals), or unattributed. Language alone was never treated as sufficient for explicit attribution. Producer country is derived separately from author affiliations, and the authors validated it on 100 human-labeled records.

Why This Matters

The paper shows that common ways of describing NLP resources (language and task metadata) leave large parts of the geographic picture invisible, and that the absence of standardized geographic metadata makes coverage claims hard to audit. This gives researchers and policymakers a measurable way to talk about representation gaps rather than relying on language-level proxies, and it offers a schema others can adopt.

Real-world applications:

• AI policy and national planning: governments can see what documented dataset resources exist for their countries and where investment is most needed, since the paper notes local governments currently struggle to determine this.
• Dataset documentation and release practices: teams can record represented populations, geographic scope, and provenance alongside language and task metadata.
• Benchmark and evaluation design: builders can assemble country-aware evaluation suites rather than assuming language coverage implies population coverage.
• Research funding and capacity building: funders can identify countries whose institutional production is low relative to how much they are represented by outside producers.

Industry relevance: companies that train and deploy multilingual systems can use country-level representation signals to audit where evaluation data is missing, and dataset platforms can adopt structured geographic fields to make population coverage auditable at scale. The paper notes that large platforms such as Hugging Face have improved resource visibility but that population-level coverage remains hard to audit without standardized metadata.

Future Directions

• Standardize geographic metadata as first-class dataset infrastructure, including represented countries or populations, geographic scope, and data provenance, so that coverage claims can be verified rather than inferred.
• Extend beyond ACL Anthology to datasets released only through Hugging Face, GitHub, industry repositories, government portals, community archives, or non-ACL venues, which the authors identify as underrepresented in AtlasNLP-Core.
• Move from paper-level records to deduplicated dataset entities, since related records may describe extensions, compilations, or versions of the same resource, and entity-level resolution is not always straightforward.
• Improve detection recall and revisit temporal claims, because the ModernBERT NLI filter was not evaluated for population-wide recall across the full ACL Anthology, leaving open whether coverage is improving over time.

Target Audience

NLP dataset creators and maintainers, benchmark and evaluation researchers, AI policy analysts and government planners, and dataset platform teams who want to document or audit geographic representation. It is also useful for researchers studying multilingual evaluation, dataset documentation practices, or equity in AI, and for students entering data-centric NLP who want a concrete example of building and validating a large annotation resource.

Authors’ abstract

Understanding which countries are represented in NLP datasets is essential for identifying gaps, targeting data collection, measuring progress, and informing AI policy. However, geographic metadata is very rarely available, and country-level representation is often hidden behind broad language-level claims. We introduce AtlasNLP, a country-aware atlas of over 13,000 NLP dataset records across normalized NLP task categories, tracking both the populations represented and where datasets are produced. AtlasNLP includes AtlasNLP-Gold, a human-curated reference set, and AtlasNLP-Core, an ACL-derived large-scale collection. Using this resource, we show that (1) dataset coverage is highly uneven across countries and tasks; (2) dataset production and representation are geographically asymmetric; and (3) language coverage does not imply geographic representation. These findings reveal blind spots in current dataset documentation practices and motivate more explicit geographic metadata for country-aware NLP evaluation.