The Pulse
NVIDIA Maps a 2.8 Trillion-Token Path for Indic AI
NVIDIA and AI4Bharat announced a 2.8 trillion-token corpus for 22 Indian languages, with native-script data in all 22 and romanized text in 12. The Hugging Face post details the translation and quality-check pipeline and reports benchmark g
AI.info Team · Written with AI ·
On October 16, NVIDIA and AI4Bharat announced a planned release of about 2.8 trillion tokens of training text across 22 Indian languages, alongside a GPU-based recipe for generating the data. The post, published on Hugging Face, describes a corpus built largely by translating English material—not a collection of 2.8 trillion tokens originally written in Indian languages. NVIDIA says the dataset is still “releasing soon”; the generation code and workflow are being shared separately.
A 2.8 trillion-token corpus in 34 configurations
Named Nemotron-pretraining-Indic-v1, the corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. It covers native-script text for all 22 languages and Latin-script versions for 12, creating 34 language-and-script configurations. The 22 languages are those listed in India’s Constitution, the post says.
The romanized material is not English: it is Indic-language text written in Latin letters. The corpus also includes translated records and script variants derived from shared English sources, so the token total does not represent unique documents or a training run’s token budget. NVIDIA says the dataset draws on 15 Nemotron-CC-v2 subsets covering web text, questions and answers, mathematics, code and specialized material.
Translation, scoring and repair
For 12 languages, the workflow uses Gemma-4-31B-it to translate full English documents, assess each translation against its source and help repair outputs that score below the acceptance threshold. The rubric checks meaning, fluency, terminology, style and formatting; accepted translations must score at least 75 out of 100, a rubric score rather than an accuracy percentage. Prompts tell the model to preserve elements such as equations, tables, code and URLs.
Accepted native-script translations also supply the starting point for romanized versions, which receive a separate check against the native-script text. The other 10 languages follow a different path: Bodhan AI’s Indic Translate performs translation, and Gemma-4-31B-it assesses the results. NVIDIA says the repair process recovered 438.4 billion tokens across native-script and romanized data, already included in the corpus total.
The reported gains came from a smaller training mixture
The announced corpus is much larger than the amount used in the reported Nemotron 3.5 Super VL continued-pretraining stage. That run had a 100-billion-token budget, with 10 billion tokens allocated to a mixture of synthetic and naturally authored Indic text; the remaining 90 billion retained the earlier training mixture’s proportions. The model improved by 4.98 to 9.82 percentage points across five reported Indic benchmarks, according to NVIDIA’s technical report.
The results do not isolate the synthetic corpus’s contribution: they measure the combined stage using synthetic and natural Indic material. The report also identifies the benchmarks as comparisons between checkpoints immediately before and after continued pretraining, not a direct test of the full 2.8-trillion-token corpus. NVIDIA’s blog says a study of the complete corpus, including repaired text and the 10-language workflow, remains to be done.
Data volume does not replace native writing
The authors frame the project as a response to a shortage of training material. “We do not have enough data for Indian languages,” Mitesh Khapra, head of AI4Bharat, writes in the post. The translated corpus can broaden the supply of material for technical subjects, but it cannot ensure coverage of local knowledge or culture, and the authors recommend combining it with appropriately sourced, naturally written Indic text.
Quality checks are primarily automated, with sampled human review whose coverage varies by language. In the 12-language workflow, the same model translates and judges the text; NVIDIA cautions that retained outputs can still contain errors. The dataset’s token count is large, but its practical value will depend on the forthcoming release, the language-specific data, and how well the synthetic text holds up when researchers evaluate it alongside native sources.