Ethics & Governance

The Language Left Behind: How AI Is Failing Seven Thousand Languages — and What It Would Take to Fix It

AI's failure across most of the world's languages: the MEGA benchmark gradient, Masakhane's community NLP, Te Hiku Media's Maori data sovereignty, India's BharatGen, and what Meta's 1,600-language Omnilingual ASR proves about whose choice t

Gabriele Masetti · 2026-08-24

A Map of Exclusion

When people speak about the AI revolution, they tend to speak in a register that assumes a certain kind of person: someone who uses English as their primary language, who has reliable access to high-speed internet, who interacts with technology through text interfaces, and who is, in some sense, already connected to the global information economy. For that person — roughly a billion people on a planet of eight billion — the AI revolution is real, immediate, and transformative.

For the other seven billion, the picture is more complicated. And for a very large subset of those seven billion — the speakers of the roughly 7,000 languages that exist on Earth today — the AI revolution is not arriving at all, or is arriving in forms that actively exclude them.

Of those 7,000 languages, fewer than 100 are meaningfully represented in the training data of modern text-based large language models. Speech is the partial exception, and a recent one: Meta's Omnilingual ASR, released in November 2025, transcribes more than 1,600 languages. Reading and writing remain the narrow gate. For the remaining 6,900-plus — including every Indigenous language of the Americas, every Aboriginal language of Australia, the majority of African languages, dozens of Oceanian languages, and hundreds of South and Southeast Asian minority languages — AI systems either fail entirely or perform so poorly as to be practically useless. Not marginally worse. Catastrophically worse.

The failure is not primarily a technical problem in the sense of being a hard unsolved challenge at the frontier of research. The methods for building high-quality language models for low-resource languages exist; researchers know broadly what would be needed. It is a problem of priorities, resources, and power: a question of which languages are considered worth the investment, and by whom.

The answer that the current AI industry has provided, by its resource allocation decisions, is that most languages are not worth it. The consequences of this answer, playing out over decades, will be profound.

What the Data Actually Shows

One of the most rigorous systematic assessments of language model performance across diverse languages comes from the MEGA benchmark (Massively Multilingual NLP Evaluation), published by researchers including Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Prachi Jain, Kalika Bali, and others — a collaboration spanning Microsoft Research India, Google Research, and the Allen Institute for AI.

MEGA evaluated the state-of-the-art language models of its era — including GPT-4, GPT-3.5, mT5, BLOOM, and others — on a suite of NLP tasks (question answering, natural language inference, commonsense reasoning, machine translation) across 70 languages. The performance patterns were stark.

For high-resource languages — English, French, German, Spanish, Chinese, Japanese — GPT-4 performed at or near human-level accuracy on most tasks. For mid-resource languages — Hindi, Swahili, Turkish, Persian — performance was substantially lower but still functional. For low-resource languages — including Haitian Creole, Burmese, Oriya, and dozens of African languages — performance dropped dramatically: in some cases to near-chance accuracy, in others to levels below smaller, specialised multilingual models trained specifically for those languages.

The specific numbers vary by task and model, but typical patterns in the MEGA evaluation showed GPT-4 dropping from around 90% accuracy on English commonsense reasoning to 70–75% for Hindi, 60–65% for Swahili, and below 50% for many low-resource languages — performance that would be considered unacceptably poor if achieved on English tasks.

Those figures are a GPT-4-era snapshot, and the frontier has turned over several times since. The gradient they describe, though, is a function of how much text exists in each language, and that has not changed: the ranking of languages by available training data in 2026 is very close to the ranking in 2023.

Two factors explain this performance gradient. The first is training data volume: the amount of text available to train on varies enormously across languages. English has hundreds of billions of words of high-quality web text; many minority languages have fewer than a million words of digital text in existence. The second is data quality: even where text data exists for a language, it may not be in a form suitable for training — it may lack consistent orthography, mix languages at the sentence level, or consist primarily of social media text with nonstandard usage.

Who Is Actually Affected

To understand the stakes of this failure, it is worth being specific about who the affected speakers are.

The approximately 1,500 languages of Sub-Saharan Africa are spoken by over a billion people. Most are either entirely absent from major AI systems or represented only in the lowest-quality training data. A Yoruba-speaking business owner in Lagos who wants to use AI tools to draft documents, answer customer queries, or search for information must either do so in English — at a disadvantage compared to native English speakers — or accept AI assistance that is genuinely unreliable.

The approximately 800 languages of Papua New Guinea — among the highest linguistic densities in the world — are almost entirely absent. The roughly 300 Aboriginal languages of Australia, most of which are critically endangered, receive essentially no representation in commercial AI systems. The Indigenous languages of the Americas — from Navajo and Cherokee to Quechua and Guaraní — are largely absent, despite having millions of speakers.

In South and Southeast Asia, the picture is mixed but troubling. Languages like Tamil (80 million speakers), Gujarati (55 million), and Punjabi (100 million) are substantially underrepresented relative to their speaker populations. The NLP4Dev report, examining language model performance for development applications, found that health information chatbots and agricultural advisory systems for rural populations — applications with potentially transformative real-world impact — consistently failed to deliver reliable information when operating in local languages.

The effect is not neutral. Populations whose languages are well-represented in AI training data gain access to powerful tools for productivity, education, health information, financial services, and government interaction. Populations whose languages are absent are excluded — not through any active discrimination, but through the passive inequity of resource allocation decisions made by a small number of predominantly English-speaking technology companies.

Masakhane: The African NLP Movement

The most significant grassroots response to this exclusion has emerged from the African AI research community. Masakhane — a name meaning "we build together" in Zulu — was founded in 2019 by Jade Abbott, Salomey Osei, Irene Relicanth, and a growing network of African researchers and technologists who were tired of waiting for global tech companies to include African languages in their systems.

Masakhane operates on a model that is unusual in machine learning: it is fundamentally a community-building organisation before it is a technical research group. Its premise is that high-quality NLP systems for African languages cannot be built without the active participation of native speakers and local researchers who understand the linguistic and cultural context. Foreign researchers, however well-intentioned, will make systematic errors in data collection, annotation, and evaluation if they do not deeply understand the communities whose languages they are working with.

The practical outputs have been substantial. Masakhane has produced benchmark datasets for dozens of African languages, including the MAFAND-MT machine translation benchmark (20 African languages) and the MasakhaNER named entity recognition benchmark (10 African languages). These datasets have enabled a wave of research on African language NLP that has demonstrated the feasibility of building high-quality models for languages that the global AI industry had previously ignored.

The organisation has also produced trained models — most notably AfriBERTa, a multilingual BERT-style model pre-trained on 11 African languages, and language-specific models for Yoruba, Swahili, Amharic, and others — that outperform multilingual models like mBERT and BLOOM on African language benchmarks, despite being trained with a fraction of the compute budget.

The Masakhane approach demonstrates something important: that community-led, culturally grounded data collection and model development can produce results that large-scale, centralised efforts with vast resources have failed to achieve. The bottleneck is not primarily technical or even financial — it is the absence of researchers with the linguistic and cultural knowledge to do the work correctly.

Te Hiku Media and the Maori Data Sovereignty Revolution

If Masakhane represents the collective community approach to language AI, the Te Hiku Media initiative in New Zealand represents something equally important: a framework for Indigenous data sovereignty that challenges the assumption that AI training data is a commons freely available to whoever wants to use it.

Te Hiku Media is a Maori-owned media organisation that has been building AI tools for the Maori language (Te Reo Maori) since 2018, most notably a speech recognition system called Papa Reo (meaning "language nest"). The system was built through years of community data collection — recorded conversations, interviews, and oral traditions donated by Maori speakers — and trained to recognise the diverse regional dialects and speaking styles of a language that had been suppressed for generations under colonial educational policy.

What makes Te Hiku Media's approach distinctive is not just the technical achievement but the governance framework surrounding it. The training data for Papa Reo is not publicly available. It is owned by and licensed to Te Hiku Media under frameworks negotiated with the Maori communities whose speech it represents. Large technology companies, including Google and Microsoft, have been denied access to the data on the grounds that the communities' linguistic heritage is not available for commercial exploitation without community consent and benefit-sharing.

That position — that communities have sovereign rights over the linguistic data that represents their cultural heritage — is articulated in the CARE Principles for Indigenous Data Governance (Collective Benefit, Authority to Control, Responsibility, Ethics) and the Indigenized AI Manifesto developed at the LT4All conference in 2025. These frameworks argue that Indigenous communities should have:

• Custodianship of data collected from or about their communities
• Veto rights over uses of that data they consider harmful or exploitative
• Benefit sharing when their data is used to create commercial value
• Capacity building support to develop AI capabilities within their communities, not just for their communities

The implications for the AI industry are significant. If the data sovereignty principle is adopted broadly — as is happening through emerging Indigenous data governance frameworks in Canada, Australia, New Zealand, and parts of the Americas — it would fundamentally change the economics and ethics of training multilingual AI systems. Data collection would need to be participatory, consensual, and governed by affected communities, not extracted by large companies and fed into models that return no benefit to the source communities.

India's BharatGen: National Sovereignty Through Open Models

While Masakhane and Te Hiku Media represent grassroots and community approaches, BharatGen represents a third model: national AI sovereignty through state investment in open-source multilingual models.

Launched in 2024 by the Government of India's National Language Technology Mission under the Ministry of Electronics and Information Technology, BharatGen is an effort to build foundation models for all 22 official languages of India — from Hindi (600 million speakers) to Manipuri (1.8 million) — that are freely available for Indian researchers, businesses, and government services.

It has since grown well past text. BharatGen says it has been funded with ₹988.6 crore under the IndiaAI Mission, and now lists multilingual text models across 22-plus Indian languages, speech recognition and a text-to-speech line called Sooktam, and vision models for image and document understanding. Its deployed applications include Krishi Sathi for agriculture, e-VikrAI for commerce and Patram for documents, built with IIT Bombay, IIT Kharagpur and Larsen & Toubro.

The motivation is explicitly sovereign: India does not want its citizens' interactions with AI-mediated government services, healthcare systems, and financial institutions to be mediated by foreign models trained primarily on English and Western cultural content. A farmer in Odisha seeking crop advisory information, a government employee in Tamil Nadu using AI-assisted document processing, a patient in a rural Gujarat clinic receiving health information — all should have access to AI systems that work reliably in their language and are trained on data that reflects Indian cultural and institutional context.

BharatGen builds on earlier efforts including the IndicBERT multilingual model from AI4Bharat, the research collective at IIT Madras that has produced the most comprehensive NLP resource suite for Indian languages to date. The combination of state funding, academic research infrastructure, and open-source release policy represents a model that several other multilingual nations — including Brazil (Portuguese and Indigenous languages), Indonesia (Bahasa Indonesia and regional languages), and Nigeria (Hausa, Yoruba, Igbo, and others) — are watching closely.

Technical Approaches: What Actually Works

Beyond specific institutional initiatives, the research community has developed a clearer picture of which technical approaches are most effective for building high-quality language AI in low-resource settings.

Cross-lingual transfer learning — training a model on one language and adapting it to a related language using small amounts of target-language data — works well for language families with sufficient structural similarity. Models trained on Spanish and Portuguese transfer well to other Romance languages; Swahili-trained models transfer to related Bantu languages. But the method degrades rapidly for languages in different families, and most of the world's most endangered languages have few close high-resource relatives.

Massively multilingual pre-training — the approach of models like mBERT, XLM-R, and BLOOM, which train on dozens or hundreds of languages simultaneously — provides a foundation for many languages but often exhibits the curse of multilinguality: adding more languages degrades performance on each individual language, as the model's capacity is spread more thinly. The optimal multilingual model for a specific low-resource language is often a much smaller model trained specifically on that language and its close relatives, not a large model trained on everything at once.

Community data collection — the Masakhane approach of involving native speakers in data creation and annotation — consistently outperforms automated data collection (scraping the web for any text in a target language) in data quality, appropriate domain coverage, and dialect representation. The cost is higher per unit data, but the downstream model quality is substantially better.

Speech-first AI deserves special attention for languages that have historically been oral rather than written. For many Indigenous languages — and for many speakers of even relatively well-resourced languages who have limited literacy — speech is the natural modality. Building speech recognition and synthesis systems, not text-based language models, may be the most practically important AI application for these communities.

It is also where the largest single move of recent years happened, and it came from exactly the kind of company this essay has been criticising. On 10 November 2025 Meta released Omnilingual ASR, a family of speech recognition models covering more than 1,600 languages — 500 of which, the company says, had never been transcribed by AI before. The models run from 300 million to 7 billion parameters under an Apache 2.0 licence, and Meta reports character error rates below 10 for 78% of the languages covered.

Two details matter more than the headline number. The release includes the Omnilingual ASR Corpus, transcribed speech in 350 underserved languages under a CC-BY licence, which Meta describes as the largest ultra-low-resource spontaneous speech dataset made available. And the system can be extended to a language it has never seen from a handful of paired audio-text examples, with no training run, no specialist expertise and no access to large compute — which puts adding a language within reach of the people who speak it rather than only of the company that shipped the model.

None of that settles the argument. A transcription model is not a language model: it does not answer a farmer's question or draft a contract, and coverage across 1,600 languages is not the same as quality in any one of them. But it does damage the claim that the technical barrier is what keeps most languages out. When a large company decided to spend on the problem, the coverage arrived in a single release.

The Structural Problem: Who Decides Which Languages Matter?

Behind all the specific technical and institutional challenges lies a structural problem that technical solutions alone cannot address: the decisions about which languages receive investment in AI are made overwhelmingly by a small number of large technology companies whose commercial incentives naturally direct attention to high-resource, economically valuable markets.

The economics are not mysterious. A language model that works for English, Spanish, French, Mandarin, Hindi, Arabic, and a handful of other major languages covers the markets where tech companies generate the vast majority of their revenue. The marginal commercial return from adding a language with a few million speakers is small, especially when those speakers are predominantly in lower-income markets. The investment required to do the language well — native speaker researchers, linguists, annotators, quality evaluators — is non-trivial. The business case simply does not close for most of the world's languages.

None of this is unique to AI. The economics of language technology — dictionaries, spell checkers, machine translation, speech recognition — have always favoured majority languages. AI is accelerating and amplifying this dynamic rather than creating it.

What is different about AI is the scale and scope of the impact. When language technology was limited to spell checkers and basic machine translation, the exclusion of minority language speakers was an inconvenience. When AI begins to mediate access to healthcare information, educational content, government services, financial tools, legal assistance, and economic opportunity, exclusion becomes a fundamental equity and human rights issue.

A Path Forward: What Would Actually Help

Several concrete changes could shift the trajectory substantially.

First, computational access: cloud compute costs are a significant barrier for researchers in low-resource language communities. Programmes that provide subsidised or free compute access to researchers working on endangered and minority languages — along the lines of Google's TPU Research Cloud but specifically targeted — would directly address a binding constraint.

Second, data infrastructure: comprehensive, well-documented, openly licensed datasets for a much broader range of languages would dramatically reduce the barrier to building new models. The CommonCrawl-based datasets that power English language models are not available for most minority languages; targeted data collection initiatives with proper community governance could change that. Meta's Omnilingual ASR Corpus, 350 underserved languages released under a CC-BY licence, is a partial answer to exactly this ask — for one modality, and one only.

Third, community capacity building: the most sustainable path to language AI for any community is to develop AI researchers and practitioners within that community. Programmes that fund graduate education, research visits, and community AI labs in regions with many underserved languages are investments in a decades-long pipeline, not quick fixes.

Fourth, data sovereignty frameworks: adopting the CARE Principles and similar frameworks for Indigenous data governance would create a foundation of trust that makes community data contribution sustainable. Communities that know their data will be used in ways they control are more likely to contribute; communities that have seen their data extracted without consent or benefit are — rationally — reluctant.

Fifth, evaluation and accountability: the AI industry lacks robust mechanisms for measuring and reporting on its performance across languages. Mandatory benchmarking and public reporting on language coverage, alongside the performance metrics already published for high-resource languages, would create accountability where none currently exists.

What Is at Stake

The languages left behind are not an unfortunate side effect of AI progress. They are evidence of a choice — not made by any individual actor consciously, but the aggregate result of thousands of small resource allocation decisions — about whose knowledge, culture, and way of communicating with the world is worth preserving and serving.

Of the roughly 7,000 languages alive today, linguists estimate that half will be extinct by the end of this century, most of them replaced by English, Mandarin, Spanish, or regional majority languages. The loss of a language is not merely the loss of a communication system. It is the loss of an entire cognitive framework — a unique way of categorising experience, encoding knowledge, expressing identity, and transmitting culture across generations.

AI that works only in a small number of majority languages accelerates this extinction. Communities whose economic and social participation increasingly requires navigating AI-mediated systems have a powerful incentive to shift toward languages those systems support — and away from the languages of their grandparents.

Conversely, AI that works well across diverse languages could be a tool for language preservation and revitalisation, as Papa Reo is already demonstrating for Te Reo Maori. The technology is neutral on this question. The choices made by those who fund and build AI systems are not.