BHRAM-IL: A Benchmark for Hallucination Recognition and Assessment in Multiple Indian Languages

AI summary

Overview

• Research area: Natural Language Processing — multilingual evaluation of large language models, specifically hallucination detection and benchmarking for under-resourced Indian languages.
• Technical level: Intermediate. The core idea (measuring how often models produce plausible but wrong answers) is easy to grasp, but the paper involves benchmark design, multiple evaluation metrics, and cross-lingual analysis.
• Scope (one sentence): The paper introduces BHRAM-IL, a multilingual benchmark for measuring hallucination in large language models across Hindi, Gujarati, Marathi, Odia, and English, and reports an initial evaluation of 14 state-of-the-art multilingual models.

What This Paper Is About

Large language models frequently produce fluent but incorrect or misleading text — hallucinations — and most existing work on detecting this problem focuses on English. Speakers of Indian languages therefore have little evidence about how reliably these models behave in their languages.

The paper addresses that gap by building BHRAM-IL, a curated benchmark of questions in several Indian languages plus English, and using it to compare how different multilingual models hallucinate across languages, task types, and domains.

Key Contributions

1. A multilingual hallucination benchmark. BHRAM-IL covers Hindi, Gujarati, Marathi, Odia, and English, comprising 36,047 curated questions.
2. A categorized question set. The benchmark spans nine categories covering factual, numerical, reasoning, and linguistic tasks.
3. A large multi-model evaluation. The authors evaluate 14 state-of-the-art multilingual LLMs on a benchmark subset of 10,265 questions, examining cross-lingual and factual hallucinations across languages, models, scales, categories, and domains.
4. Public release of resources. The dataset, along with code for question generation and evaluation, is released on GitHub and HuggingFace to support further work on multilingual hallucination detection and mitigation.

Main Findings

• Aggregate hallucination performance is low: Aggregating over all categories and models yields a primary score of 0.23 (on a metric normalized to the 0–1 range).
• A second, language-corrected measure is also low: The language-corrected fuzzy score is 0.385, indicating that performance remains limited even when language-related factors are accounted for.
• Evaluation spans multiple dimensions: The study analyzes hallucinations by language, model, model scale, category, and domain, using category-specific metrics normalized to the 0–1 range.
• The benchmark is shown to be usable: The authors state that these scores demonstrate the usefulness of BHRAM-IL for hallucination-focused evaluation.
• Per-language and per-model breakdowns are not given in the abstract: The abstract does not report which languages or which models performed best or worst, nor does it give category-level results. Those details would appear in the full paper.

Methodology in Plain English

The authors first assembled a large set of questions — 36,047 in total — across five languages (four Indian languages plus English) and nine categories that mix factual recall, numerical work, reasoning, and language-specific tasks. Questions were curated rather than simply translated, so the benchmark covers tasks relevant to each language.

They then selected 14 widely used multilingual language models and ran them on a subset of 10,265 questions. Model answers were scored with metrics designed for each question category, and all scores were rescaled to a 0-to-1 range so that results could be compared across categories. Two headline numbers are reported: a primary aggregate score across all categories and models (0.23), and a language-corrected fuzzy score (0.385) that adjusts for language-related factors. The team also broke results down by language, model, model size, category, and domain to see where hallucinations concentrate. The dataset and the code used to generate and evaluate it were released publicly.

Why This Matters

• Research impact: Hallucination research has been heavily English-centric. BHRAM-IL provides a shared, multi-language testbed so that claims about multilingual model reliability can be measured rather than assumed, and so that results across languages become directly comparable.
• Real-world applications:
  • Customer service and chatbots deployed in Hindi, Gujarati, Marathi, or Odia, where a confident wrong answer can mislead users.
  • Healthcare and public information in regional languages, where factual accuracy is safety-critical.
  • Education, where learners may rely on model-generated explanations and would not easily spot fabricated content.
  • Government and civic services that serve citizens in multiple Indian languages and need trustworthy automated responses.
• Industry relevance: Organizations building or deploying multilingual products in India need evidence about which models hallucinate less in which languages and task types. A categorized benchmark gives them a way to compare models before deployment and to justify model choices in languages where training data is scarce.

Future Directions

• Mitigation, not just measurement. The benchmark identifies where hallucination is high; the natural next step is developing and testing techniques that reduce it in under-resourced Indian languages.
• Extending coverage. The abstract covers four Indian languages. Applying the same methodology to other under-resourced languages in India and beyond is an open opportunity.
• Deeper error analysis. A finer breakdown of which categories (factual vs. numerical vs. reasoning vs. linguistic) drive failures, and how this varies by model scale, is raised by the abstract's multi-dimensional framing but not resolved in it.
• Improving metric design. The gap between the primary score (0.23) and the language-corrected fuzzy score (0.385) suggests that how hallucination is measured across languages matters — a question the field will need to settle.

Target Audience

This paper is most useful to NLP researchers working on hallucination detection, multilingual evaluation, and low-resource languages; to engineers and product teams deploying multilingual LLMs in Indian languages who need comparative reliability data; to evaluation and safety practitioners who design benchmarks; and to graduate students or newcomers looking for a structured entry point into cross-lingual hallucination work, since the released dataset and code lower the barrier to building on it.

Authors’ abstract

Large language models (LLMs) are increasingly deployed in multilingual applications but often generate plausible yet incorrect or misleading outputs, known as hallucinations. While hallucination detection has been studied extensively in English, under-resourced Indian languages remain largely unexplored. We present BHRAM-IL, a benchmark for hallucination recognition and assessment in multiple Indian languages, covering Hindi, Gujarati, Marathi, Odia, along with English. The benchmark comprises 36,047 curated questions across nine categories spanning factual, numerical, reasoning, and linguistic tasks. We evaluate 14 state-of-the-art multilingual LLMs on a benchmark subset of 10,265 questions, analyzing cross-lingual and factual hallucinations across languages, models, scales, categories, and domains using category-specific metrics normalized to (0,1) range. Aggregation over all categories and models yields a primary score of 0.23 and a language-corrected fuzzy score of 0.385, demonstrating the usefulness of BHRAM-IL for hallucination-focused evaluation. The dataset, and the code for generation and evaluation are available on GitHub (https://github.com/sambhashana/BHRAM-IL/) and HuggingFace (https://huggingface.co/datasets/sambhashana/BHRAM-IL/) to support future research in multilingual hallucination detection and mitigation.