Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali

AI summary

Overview

Research area: Natural Language Processing — benchmark construction and hallucination/factuality evaluation of large language models, specifically religious-source fabrication in a low-resource language (Bengali).

Technical level: Intermediate. Readers need a basic understanding of how language models are prompted and evaluated, but the paper's rubric and scoring scheme are explained from first principles.

Scope (one sentence): The paper introduces Bilal-Bench, a 240-question expert-verified Bengali benchmark measuring whether seven open-weight models (1B–120B parameters) fabricate or misattribute Islamic sources — Qur'anic verses, hadith, and fiqh rulings — and evaluates 1,680 double-annotated responses with a rubric that separates the type of source error from how assertively it is presented.

What This Paper Is About

When a language model invents a Qur'anic verse or a hadith, it lends false religious authority to text that readers may treat as sacred — and because the model often supplies a sūrah and ayah number or a hadith collection, narrator, and number, the fabrication looks exactly like the provenance evidence a reader would use to verify it. Existing Islamic hallucination benchmarks are largely in Arabic and often closed-ended, while Bengali work has focused on general-domain hallucination or factual knowledge rather than open-ended source generation. The paper builds a Bengali benchmark to measure this specific failure, applies it to seven open-weight models, and separates two distinct behaviours: whether a model produces a correct source, and whether it withholds a claim it cannot support.

Key Contributions

1. Bilal-Bench, a 240-question Bengali benchmark with five categories: Qur'anic attribution (60 items), hadith attribution (45), fiqh rulings under the Hanafi school (45), contemporary issues with no single universal answer (45), and adversarial false-premise prompts (45). Questions were written directly in Bengali rather than translated, and gold references were drawn from the Islamic Foundation Bangladesh Qur'an translation, Ma'arif al-Qur'an (Bengali), Bengali editions of the Sihah Sittah, and established Bengali fatwa portals, checked by a qualified reviewer of Islamic sciences.

2. A two-axis annotation rubric that separates source errors from assertiveness. Axis A records the source outcome across seven labels (F1 fabricated text, F2 misattribution, F3 distortion, F4 unsupported assertion, F5 correct but unsourced, F6 correct and sourced, F7 appropriate abstention or false-premise correction, plus NA for no stated claim); Axis B records whether the answer is assertive or hedged. Boundary rules were fixed before annotation, with severity ordering F1 > F2 > F3 > F4 > F5 > F6.

3. A large double-annotated evaluation: 7 models × 240 questions = 1,680 responses, each independently labelled by two native Bengali research-team members with Islamic studies backgrounds, blind to model identity, with responses pooled and shuffled. Inter-annotator agreement was Cohen's κ = 0.864 on the eight-class taxonomy, 0.893 after collapsing to fabrication/acceptable/NA, and 0.851–0.912 at category level.

4. A controlled comparison of Bangla continual pretraining via two BanglaLLaMA 8B checkpoints that differ only in whether Bangla continual pretraining was applied (3.1-8B-instruct vs 3.1-8B-no-pretrain), isolating that single factor.

Main Findings

• Confident fabrication is high and does not shrink with scale. On C1–C3 (source-production categories, n = 150 per model), six of seven models confidently fabricate on 79.3–95.3% of responses. Gemma-2-9B-it is highest at 95.3%, GPT-OSS-120B reaches 93.3%, and BanglaLLaMA-3.2-3B-instruct reaches 90.0%, with overlapping confidence intervals. Among those six, TituLM-gemma-2-2b is lowest at 79.3%.

• The one low score reflects silence, not accuracy. TigerLLM-1B-it records 64.7% fabrication (63.3% confident) on C1–C3, but 43 of its 150 responses contain no stated claim, and it produces no correctly sourced answer anywhere in the benchmark.

• Hadith provenance is the weakest category across all models. Across the 315 C2 responses, only two are correct and correctly sourced. Three models fabricate on every C2 item (100%: BanglaLLaMA-3.1-8B-no-pretrain, Gemma-2-9B-it, GPT-OSS-120B), and the lowest C2 fabrication rate is 88.9%.

• Restraint varies far more than source accuracy, and does not track parameter count. On false-premise prompts (C5), Gemma-2-9B-it fabricates on 40.0% versus 91.1% for GPT-OSS-120B. Appropriate abstention or correction (F7) ranges from 6 responses for GPT-OSS-120B to 46 for Gemma-2-9B-it across the full benchmark.

• Hedging is nearly absent. Of 1,320 responses labelled F1–F4 across all models, 1,316 (99.7%) are assertive and only four are hedged, leaving readers almost no surface signal that a source is invented.

• The two 8B BanglaLLaMA variants fabricate at similar rates with different error profiles. Their confident-fabrication rates on C1–C3 are 83.3% (with Bangla continual pretraining) and 86.7% (without). Across all 240 items their F1/F3 counts are 93/8 with pretraining versus 84/17 without.

• Error type differs sharply by model. GPT-OSS-120B concentrates failures in outright invention (F1 = 166) with only one unsupported assertion (F4 = 1), whereas TituLM-gemma-2-2b has 16 F1 cases but 90 F4 cases. Both are wrong, but a fabricated citation is more misleading because it hands the reader a specific provenance cue to trust or search for.

• Correctly sourced answers are rare overall. Across the full benchmark, only 40 of 1,680 responses are both correct and correctly sourced.

Methodology in Plain English

The authors wrote 240 religious questions in Bengali, split into five groups by what a correct answer should look like: three groups where a real source or ruling exists (Qur'an, hadith, fiqh), one where the honest answer is to note scholarly disagreement rather than invent text, and one built on a false premise that should be rejected rather than supported with a citation. Each question carries a gold reference and a note explaining how that reference was established.

They then gave every question to seven open-weight models of different sizes and training backgrounds — TigerLLM-1B-it, TituLM-gemma-2-2b, Gemma-2-9B-it, GPT-OSS-120B, and three BanglaLLaMA variants (3.2-3B-instruct, 3.1-8B-instruct, and 3.1-8B-no-pretrain) — using a single neutral Bengali instruction with no system message, no worked examples, and no hint that hedging was allowed. The instruction explicitly asks for a sūrah name and ayah number if a verse is cited, and a collection name plus narrator and hadith number if a hadith is cited. Generation was greedy and deterministic with a 512-token budget, rerun once at 1,024 tokens only if length-capped.

Two annotators with Islamic studies backgrounds independently labelled every one of the 1,680 responses while blind to which model produced them, with responses shuffled first; disagreements were resolved by discussion to a final agreed label. Each answer got a source-outcome label (fabricated text, misattribution, distortion, unsupported assertion, correct but unsourced, correct and sourced, or appropriate abstention/correction) and an assertiveness label. The headline "confident-fabrication rate" counts responses labelled F1–F4 that were also assertive, keeping NA responses in the denominator.

Why This Matters

Impact on research. The paper shows that a failure mode specific to religious content — invented scripture with fabricated citation metadata — is not fixed by scaling up models, at least within this set. It also demonstrates that a single accuracy number hides two separable problems: whether generated provenance exists at all, and whether a model knows when to decline to supply specificity. Splitting these gives future work two distinct, measurable mitigation targets, and it extends hallucination evaluation to a language and an open-ended generation setting that had been largely unmeasured.

Real-world applications:

• Consumer religious Q&A. Bengali-speaking users asking chatbots about verses, hadith, or rulings now have documented evidence that current open-weight models will usually answer with confident but invented provenance.
• Fatwa and religious guidance services. The paper explicitly cautions against relying on these models for religious questions, including seeking fatwas, which matters for any organization considering deploying an assistant in this domain.
• Retrieval-grounded religious assistants. The benchmark provides a measurement instrument for teams building systems that retrieve from verified Bengali collections, letting them test whether grounding actually reduces fabrication.
• Citation-verification tooling. Because failures often include a specific sūrah:ayah number or collection/narrator/hadith number, the results motivate automatic checks that a cited reference actually exists.

Industry relevance. For developers shipping multilingual assistants, the findings argue that parameter count is a poor proxy for trustworthiness in high-stakes domains, and that explicit abstention training and citation-aware decoding are more promising levers than scale. The two BanglaLLaMA variants show that a targeted pretraining decision changed the mixture of error types without changing the overall fabrication rate — a useful caution for anyone tuning models for a specific language.

Future Directions

• Retest hadith with retrieval and existence checks. The authors call the hadith result the sharpest target for follow-up: re-evaluate with retrieval from verified Bengali collections and with an automatic check that any cited collection, narrator, and hadith number exists.
• Explain the abstention spread. The 6-to-46 range in appropriate abstention has no scale explanation in this evaluation; more model families and prompt styles are needed before it can be attributed to specific training choices.
• Extend beyond the single Hanafi gold position. C3 encodes the Hanafi ruling as a deliberate design choice, and the authors state results should not be generalized across scholarly traditions without extending the benchmark.
• Extend beyond Bengali and beyond one instruction. The paper's limitations note a single neutral instruction, one language, and a 240-item benchmark, all of which need broadening before findings transfer to other under-resourced languages or prompting conditions.

Target Audience

Researchers working on hallucination, factuality, and benchmark design for large language models, especially those focused on low-resource and non-English languages; NLP practitioners building or auditing religious, culturally sensitive, or high-stakes question-answering systems; and members of Bengali-speaking and Muslim technical communities who want quantitative evidence about the reliability of open-weight models on Islamic source material. The paper is also accessible to policy and deployment reviewers who need to judge whether a model is safe for religious guidance use, since its core findings are stated in terms of rates and error types rather than model internals.

Authors’ abstract

When a language model invents a Qur'anic verse or hadith, it gives false authority to text that readers may treat as sacred. Existing Islamic hallucination benchmarks are largely Arabic and often closed-ended, while Bengali resources mainly study general-domain hallucination or factual knowledge. We introduce BILAL-BENCH, a 240-question expert-verified benchmark for open-ended Islamic-source fabrication in Bengali. It covers Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial false-premise prompts, and uses a two-axis rubric that separates source errors from assertiveness. We evaluate seven open-weight models from 1B to 120B on 1,680 double-annotated responses. Six of the seven models confidently fabricate on 79.3-95.3% of source-production questions, with no monotonic improvement from scale. Hadith provenance is the weakest category across models: only two of 315 responses are correct and correctly sourced. The two 8B BanglaLLaMA variants have similar confident-fabrication rates but different error profiles. Models vary far more in whether they withhold a claim they cannot support than in source accuracy: appropriate abstention or false-premise correction ranges from 6 to 46 responses across the benchmark, and does not track parameter count.