Visual Sanad: Auditing Vision-Language Models for False Acceptance of Qur'anic Citations

AI summary

Overview

Research area: Multimodal factuality and cultural-safety auditing for vision–language models (VLMs), specifically citation verification of Qur'anic quotations rendered as images. Workshop paper: MusIML@NeurIPS 2026 (arXiv:2610.22009v1 [cs.CV]).

Technical level: Intermediate — the paper is readable without deep mathematics, but interpreting it requires familiarity with benchmark design, false-accept/false-reject tradeoffs, discrimination metrics, and paired significance tests (McNemar).

Scope (one sentence): The paper introduces and evaluates a 2,400-image controlled benchmark testing whether 12 VLMs (4 closed/API, 8 open-weight) correctly verify English Qur'anic quote–reference pairs, and whether decorative or "verified citation" visual framing shifts their decisions.

What This Paper Is About

People increasingly encounter Qur'anic quotations as images — posters, lecture slides, messaging-app cards, social-media graphics — each pairing a translated verse with a surah:ayah reference that may be wrong. This paper asks a narrow, testable question: when shown such an image, can a VLM tell whether the displayed quote actually belongs to the displayed reference, and do visual markers of authority (decorative Islamic styling, "Verified Qur'anic Citation" headers) make models more likely to accept a false attribution? The goal is to build a controlled, human-audited benchmark that isolates citation-verification ability from ordinary image reading, and to measure both model capability gaps and presentation bias.

Key Contributions

1. A controlled 2,400-image benchmark. 100 English Qur'an verses × 4 factual states (1 valid, 3 invalid foil types) × 6 visual conditions, yielding 600 valid and 1,800 invalid images.
2. A human-audited construction pipeline. Source text drawn from the AlQuran Cloud en.sahih edition, with three invalid foil types (near-neighbor, same-surah far, cross-surah lexical) screened by exact-string, character-overlap, and asymmetric token-containment filters, and the dataset audited in three passes before locking.
3. Baseline evaluations of 12 VLMs. Four closed/API models (GPT-4o, GPT-4.1, GPT-5.5, GPT-5.6-Terra) and eight open-weight models (Qwen2.5-VL-7B/32B, Qwen3-VL-8B/32B, InternVL3.5-8B/38B, MiniCPM-V 4.5, Gemma 4 12B-IT), each under a natural and a de-biased prompt — 24 full model–prompt runs over all 2,400 images.
4. Paired analyses of framing and prompting. Because each underlying item appears in all six visual conditions and under both prompts, the authors compute paired flip counts and McNemar tests to separate genuine discrimination from threshold shifting.

Main Findings

• A sharp closed/open capability split. GPT-5.5 reaches 0.00% false accept and 99.83 discrimination under the natural prompt; GPT-5.6-Terra reaches 4.72% false accept and 95.11 discrimination. GPT-4o and GPT-4.1 show substantial false acceptance (14.89% and 15.94%) with discrimination of 83.44 and 83.73.
• Open-weight models mostly default to VALID. Several produce clean, parseable labels while saying VALID on 90–99% of images. MiniCPM-V 4.5 has 99.17% false accept and 0.50 discrimination (says VALID 99.3%); Gemma 4 12B-IT 98.00% and 0.50; Qwen2.5-VL-7B 96.28% and 0.89; InternVL3.5-8B 94.00% and 0.67. Because the dataset is 25% valid images, such a model scores near 25% raw accuracy with near-zero discrimination.
• Reference reading is not the bottleneck. Model-reported displayed references matched the rendered reference in at least 99.83% of rows for every reported run, and most runs reached 100%. Abstention never exceeded 0.62%. MiniCPM-V 4.5 under the natural prompt produced high-confidence errors on 74.5% of all images while abstaining on none.
• Foil difficulty varies by model. For GPT-4o, natural-prompt false acceptance is 17.67% on near-neighbor, 2.83% on same-surah far, and 24.17% on cross-surah lexical foils. For GPT-5.6-Terra the same cells are 10.50%, 1.67%, and 2.00%; for GPT-5.5 all three are 0.00%.
• Visual framing effects are real but smaller than capability effects, and direction depends on model family. GPT-4o rises from 13.00% false accept on plain invalid images to 17.33% on decorative neutral-authority images (net induced 13, McNemar p = 0.00235). Qwen3-VL-32B drops from 51.33% under plain to 34.00% under Islamic-authority framing, while decorative framing raises it to 66.33% (net induced +45, p = 6.27 × 10⁻⁸ for decorative; −52, p = 5.75 × 10⁻¹¹ for Islamic authority). InternVL3.5-38B goes from 68.67% plain to 89.33% under both Islamic-authority and neutral-authority headers (net induced +62, p = 8.47 × 10⁻¹⁶ and 2.26 × 10⁻¹⁶).
• De-biased prompting rescues errors but does not buy discrimination. The added caution ("Do not treat decorative design, religious-looking borders, or verification headers as evidence that the citation is correct") produces asymmetric paired flips: InternVL3.5-38B nets 603 rescues (p < 10⁻¹⁰), Qwen3-VL-32B nets 162, Qwen2.5-VL-7B nets 61. But for Qwen3-VL-32B, false accept falls from 52.17% to 43.17% while discrimination falls from 12.16 to 10.83 because false rejection rises from 35.67% to 46.00% — the decision threshold moves toward INVALID rather than becoming better calibrated. GPT-5.5 and GPT-5.6-Terra show zero net change.

Methodology in Plain English

The authors start from 100 English Qur'an verses chosen for readability and duplicate safety. For each verse they build four versions of the same quote: one where the displayed surah:ayah reference is correct, and three where it is wrong in a graded way — a nearby verse in the same surah, a farther verse in the same surah, and a verse from a different surah that shares vocabulary. Foils are filtered so the correct verse isn't accidentally contained inside the wrong one, which would make the "invalid" label ambiguous.

Each of these 400 quote–reference pairs is then rendered as an image in six styles: plain, decorative (patterned background), Islamic-authority (header like "Verified Qur'anic Citation"), decorative plus Islamic authority, neutral authority (header like "Verified Citation Record"), and decorative plus neutral authority. That gives 2,400 images. Crucially, the same underlying item appears in every style, so differences between styles can be attributed to the framing rather than to differences in content.

Models see one image and must return VALID, INVALID, or ABSTAIN as JSON. The headline number is the false-accept rate on the 1,800 invalid images, paired with false-reject rate on the 600 valid ones. The authors combine these into "discrimination" (100 − false accept − false reject), which exposes models that look accurate only because they say VALID to almost everything. They compare a natural prompt against a de-biased prompt that explicitly warns against treating design and verification headers as evidence. Every model–prompt pair runs over all 2,400 images; open-weight runs use greedy decoding (do_sample=False, seed 31).

Why This Matters

Impact on research. The paper reframes multimodal hallucination evaluation around attribution rather than object recognition, and shows that a benchmark can distinguish "can't read the image" from "won't doubt the image." The discrimination metric plus paired McNemar analysis gives a template for separating genuine capability from response bias in any yes/no multimodal task. It also contributes a culturally grounded test case to the small but growing set of Islamic-NLP and Muslim-community-relevant AI evaluations.

Real-world applications.

• Fact-checking and moderation tools that process screenshots of religious quotations circulating in messaging apps and social feeds, where a confidently wrong VALID label amplifies misinformation instead of correcting it.
• Educational assistants used by students and teachers who encounter Qur'anic citations in lecture slides and study graphics.
• Retrieval-augmented verification pipelines — the authors propose that such systems should fetch the referenced verse from a canonical source before judging the image, or abstain when they cannot verify, rather than answering the image directly.
• Regression testing during model selection, as a lightweight check for whether a candidate VLM has any citation-verification ability at all before deployment in community-facing products.

Industry relevance. The results are a concrete caution against treating citation verification as generic visual question answering. A model that reads the printed reference perfectly (≥99.83% extraction) can still be nearly useless at judging whether that reference is correct. For teams deploying VLMs in religious, legal, medical, or scholarly contexts where exact source attribution matters, the paper argues for retrieval-grounded verification or abstention as a default, and warns that adding "verified" visual chrome can push some model families toward misplaced trust.

Future Directions

• Move beyond English translations. The current benchmark uses the en.sahih rendering, not Arabic Qur'anic text; an Arabic version would test whether the failure mode is tied to translation or to attribution reasoning generally.
• Test on real screenshots. The paper evaluates rendered static images rather than actual social-media captures, leaving open how compression artifacts, cropping, overlays, and partial screenshots affect false acceptance.
• Retrieval-augmented evaluation. Since prompting alone shifts thresholds rather than improving discrimination, the natural next step is measuring whether retrieving the canonical verse before judging the image actually produces verification rather than learned skepticism.
• Extend to the moving frontier, with proper statistics. The authors note their 12-model set is not exhaustive and that the dataset is clustered by verse, so downstream analyses should include verse-level random effects rather than treating the 2,400 images as independent — a methodological caveat that also invites larger, clustered-aware replication.

Target Audience

Researchers and practitioners working on multimodal factuality, hallucination, and benchmark design; teams building or auditing retrieval-augmented vision systems for religious, educational, or fact-checking use; AI safety and cultural-alignment researchers interested in community-grounded evaluation; and Islamic NLP researchers who want a vision-side complement to existing Qur'anic question-answering benchmarks. Model-selection engineers will find the closed/open capability split and the discrimination metric directly actionable.

Authors’ abstract

Vision-language models (VLMs) are increasingly used to interpret religious, educational, and culturally sensitive visual content, yet their reliability on citation verification remains underexplored. We introduce Visual Sanad, a controlled benchmark for testing whether VLMs falsely accept incorrect Qur'anic verse attributions when the same quote-reference pair is rendered under different visual framings. The benchmark contains 2,400 rendered images built from 100 English Qur'an verses, four factual states per verse, and six visual conditions ranging from plain text to decorative and authority-framed layouts. All quote-reference pairs were source-checked and human-audited before rendering. Across 24 full model-prompt runs, we observe a sharp capability split. Proprietary API-served VLMs range from near-perfect citation verification to moderate false acceptance, whereas recent open-weight VLMs often show weak discrimination and default to VALID on most images. De-biased prompting can reduce false acceptance, but often by shifting the decision threshold toward INVALID rather than by producing strong citation discrimination. Visual Sanad therefore exposes a culturally grounded multimodal factuality failure: fluent VLMs may treat displayed religious citations as self-authenticating unless they have strong citation-verification ability.