Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation

AI summary

Overview

• Research area: Natural Language Processing — low-resource language modeling, multilingual LLM training, synthetic instruction-data generation, and cultural alignment for Urdu.
• Technical level: Intermediate. The paper is readable without deep math, but assumes familiarity with instruction tuning, LoRA adapters, continued pre-training, quantization formats, and LLM-as-a-judge evaluation.
• Scope: The paper describes the Urdu-Instruct synthetic dataset, the Alif-1.0-8B-Instruct multilingual Urdu-English model trained on it, and benchmark, English-retention, and quantization evaluations of that model.

What This Paper Is About

Building a strong LLM for Urdu is hard because high-quality Urdu datasets are scarce, translations from English lose cultural nuance, and existing multilingual models generate weak Urdu text. The authors' goal is to produce a competitive Urdu-English instruction model cheaply by generating a culturally grounded synthetic Urdu dataset with a modified self-instruct method rather than by translating large volumes of English data.

Key Contributions

1. A multilingual Urdu-English model: Alif-1.0-8B-Instruct, built on the pretrained Llama-3.1-8B base, reported to outperform leading multilingual LLMs on Urdu-translated MGSM, Alpaca Eval, and Dolly General QA benchmarks.
2. A modified self-instruct technique: Each task gets unique prompts and seed values, all generated instructions are consolidated in a single global task pool, and duplicates are rejected via a ROUGE similarity threshold of 0.7. The authors state this method can be adapted to other low-resource languages.
3. The Urdu-Instruct dataset: 51,686 synthetic examples generated with GPT-4o (API version '2024-08-01-preview'), covering Generation (5,907), Ethics (9,002), QA (8,177), Reasoning (9,590), Translation (10,001), Classification (4,662), and Sentiment Analysis (4,347), plus a new Urdu evaluation set of roughly 150 examples per task.
4. Evaluation across Urdu-translated benchmarks and a new evaluation set: The model is assessed on Urdu-translated MGSM, AlpacaEval, and Dolly General QA, and on the newly created Urdu evaluation set.

Main Findings

• Large gains on the Urdu evaluation set: Alif-1.0-8B-Instruct versus Llama-3.1-8B-Instruct scored Generation 90.2 vs 42.8, Ethics 85.7 vs 27.3, QA 73.8 vs 30.5, Reasoning 83.5 vs 45.6, Translation 89.3 vs 58.9, Classification 93.9 vs 61.4, Sentiment 94.3 vs 54.3, and Weighted Avg. 87.1 vs 45.7.
• Best average on Urdu-translated benchmarks: Alif-1.0-8B-Instruct scored MGSM 72.2, Alpaca Eval 78.4, Dolly General QA 75.9, average 75.5 — higher than Aya-expanse-8b (average 68.9), Ministral-8B-Instruct-2410 (62.0), Qwen2.5-7B-Instruct (59.8), Gemma-7b-it (56.6), Granite-3.2-8b-instruct (55.3), Llama-3.1-8B-Instruct (49.6), Mistral-7B-Instruct-v0.3 (41.9), Phi-3-small-8k-instruct (39.1), and Falcon-7b-instruct (21.8).
• English abilities largely preserved, with one regression: On lm-evaluation-harness, Alif improved over Llama-3.1-8B-Instruct on arc_challenge (acc 0.5478 vs 0.5171; acc_norm 0.5623 vs 0.5512), arc_easy (acc 0.8258 vs 0.8190; acc_norm 0.8194 vs 0.7950), and hellaswag (acc 0.6135 vs 0.5914; acc_norm 0.8022 vs 0.7922). MMLU dropped (acc 0.6177 vs 0.6798), including humanities (0.5530 vs 0.6425), other (0.7007 vs 0.7438), social sciences (0.7260 vs 0.7702), and stem (0.5268 vs 0.5842).
• Quantization trade-off: Using llama.cpp across GGUF formats Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0, and F16, the authors report that 6-bit and 8-bit formats keep perplexity close to FP16 while reducing model size, lower-bit formats (2-bit, 3-bit, 4-bit) raise perplexity, and Q2_K is not viable due to a substantial perplexity increase. Q6_K and Q8_0 are identified as the optimal deployment choices. The Urdu text corpus consistently showed lower perplexity than the English corpus.
• Low training cost: Total training took 39 hours on an Nvidia A100 80GB via Runpod at $1.64 per hour (23 hours of continued pre-training for one epoch over 200K Urdu Wikipedia articles, plus 16 hours of fine-tuning for two epochs), keeping the overall cost under $100 as of February 12, 2025.
• Fine-tuning mixture: 105,339 total instruction examples: Urdu-Instruct (51,686), a translated dataset (28,910), ULS_WSD (4,343), English Alpaca (10,400) as replay, and OpenOrca (10,000) as replay.

Methodology in Plain English

The authors start from a pretrained Llama-3.1-8B model and first run continued pre-training on 200K Urdu Wikipedia articles using standard causal language modeling, so the model sees more Urdu text before instruction tuning.

For the instruction data, they modify the self-instruct recipe: rather than one generic prompt generator, each of the seven task categories receives its own prompt and seed values, and every generated instruction is checked against one shared global task pool. Each prompt is expanded with four human-annotated seeds and two machine-generated seeds, and GPT-4o produces 20 instructions with outputs per batch. Automated filters remove instructions shorter than three words or longer than 150 words, flag unsuitable keywords, reject instructions starting with punctuation or containing characters outside Urdu and English, and reject anything scoring above 0.7 ROUGE similarity against prior instructions. Human annotators then clean Urdu grammar, factual correctness, and ethical content.

Fine-tuning mixes the synthetic Urdu-Instruct data with translated Urdu instructions, ULS_WSD, and replay data in English (Alpaca and OpenOrca) to limit catastrophic forgetting, using the Stanford Alpaca prompt template with loss computed only on the output portion. Training uses LoRA (rank 128, alpha 32) on QKVO attention projections, MLP projections, and embedding/output layers — 14.72% of parameters trainable — with BF16 precision, cosine schedules (2e-5 for continued pre-training, 5e-5 for fine-tuning), batch size 64, and max length 2048, via the Unsloth framework.

Evaluation uses GPT-4o as an LLM judge on a 10-point scale, refined with human feedback, with each score expressed as the percentage ratio of the generated response score to the reference response score. The benchmarks were translated to Urdu by GPT-4o: MGSM (250 math questions), AlpacaEval (806 instructions), and a 220-example subset of Dolly General QA. English retention was measured with lm-evaluation-harness, and quantization effects with llama.cpp on wiki-test-raw and an Urdu-translated version of it.

Why This Matters

This work argues that a culturally grounded, low-cost synthetic data pipeline can close part of the gap between high-resource and low-resource languages without expensive large-scale translation, and it releases the dataset, model, and code for reuse. It also provides a concrete demonstration that English replay data can preserve general ability while Urdu performance improves sharply, alongside a candid record of where knowledge-intensive English performance degrades.

Real-world applications:

• Urdu-language question answering, text generation, classification, and sentiment analysis tools for consumer or enterprise use.
• Urdu-English and English-Urdu translation and bilingual content workflows.
• Locally deployable Urdu assistants, since the paper shows 6-bit and 8-bit GGUF quantization is practical on personal computers.
• Safer, culturally aware Urdu conversational systems, given the Ethics and Translation task categories and the safety alignment in the dataset.

Industry relevance: The reported training budget of under $100, the open releases, and the quantization analysis lower the barrier for organizations serving Urdu speakers or other low-resource language markets, and the modified self-instruct recipe is presented as reusable for other underrepresented languages.

Future Directions

• Incorporate objective, task-specific metrics such as Exact Match, F1, BLEU, COMET, and BERTScore to quantify alignment, factuality, and stylistic correctness in bilingual settings.
• Compare multiple judge models and prompts to test robustness across cultural and linguistic variations.
• Broaden high-quality datasets and improve reasoning through model merging and reinforcement learning.
• Benchmark Alif against evolving multilingual and reasoning standards; the authors also flag remaining limitations in Urdu domain-specific knowledge, harmful or misaligned outputs, and inconsistent behavior on adversarial or rare inputs, with MMLU showing the largest retention loss in this study.

Target Audience

Researchers and engineers working on low-resource and multilingual NLP, practitioners who need a deployable Urdu or Urdu-English instruction model, and teams interested in synthetic data generation, self-instruct variants, LoRA-based low-budget fine-tuning, or quantization for local deployment. It is also useful to anyone studying catastrophic forgetting and replay-based mitigation strategies.

Authors’ abstract

Developing a high-performing large language models (LLMs) for low-resource languages such as Urdu, present several challenges. These challenges include the scarcity of high-quality datasets, multilingual inconsistencies, and safety concerns. Existing multilingual LLMs often address these issues by translating large volumes of available data. However, such translations often lack quality and cultural nuance while also incurring significant costs for data curation and training. To address these issues, we propose Alif-1.0-8B-Instruct, a multilingual Urdu-English model, that tackles these challenges with a unique approach. We train the model on a high-quality, multilingual synthetic dataset (Urdu-Instruct), developed using a modified self-instruct technique. By using unique prompts and seed values for each task along with a global task pool, this dataset incorporates Urdu-native chain-of-thought based reasoning, bilingual translation, cultural relevance, and ethical safety alignments. This technique significantly enhances the comprehension of Alif-1.0-8B-Instruct model for Urdu-specific tasks. As a result, Alif-1.0-8B-Instruct, built upon the pretrained Llama-3.1-8B, demonstrates superior performance compared to Llama-3.1-8B-Instruct for Urdu specific-tasks. It also outperformed leading multilingual LLMs, including Mistral-7B-Instruct-v0.3, Qwen-2.5-7B-Instruct, and Cohere-Aya-Expanse-8B, all within a training budget of under $100. Our results demonstrate that high-performance and low-resource language LLMs can be developed efficiently and culturally aligned using our modified self-instruct approach. All datasets, models, and code are publicly available at: https://github.com/traversaal-ai/alif-urdu-llm.