BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models

AI summary

Overview

• Research area: Computer vision and natural language processing — specifically scene text recognition (OCR) for Bengali, and evaluation of vision-language models (VLMs) on low-resource scripts.
• Technical level: Intermediate. The paper uses standard OCR metrics (Character Error Rate, Word Error Rate, Normalized Edit Distance) and an LLM-as-a-Judge protocol, plus LoRA fine-tuning; a reader with basic familiarity with machine learning benchmarks will follow it, and the concepts are explained clearly enough for newcomers.
• Scope (1 sentence): The paper introduces BanglaWild, a 2,535-image Bengali in-the-wild scene text benchmark with dual verbatim/standard transcriptions, and uses it to evaluate 15 VLMs, 3 conventional OCR systems, 3 prompting strategies, and 6 LoRA-fine-tuned models under a 15-class error taxonomy.

What This Paper Is About

Existing Bengali text resources target either handwritten documents under controlled conditions or constrained signboard parsing, and they report only aggregate edit-distance metrics while evaluating either conventional OCR or VLMs — never both on the same in-the-wild data. The authors build a natively photographed Bengali scene text benchmark covering public walls, signboards, street vehicles, banners and posters, and product packaging, then measure how both model classes actually fail on that imagery. The goal is to replace marketing claims about multilingual capability with a diagnostic, stratified measurement of real Bengali scene text performance.

Key Contributions

1. The first in-the-wild Bengali scene text recognition benchmark evaluating both conventional OCR and generative VLMs on the same data, stratified by categorical and diagnostic attributes with adjudicated gold transcriptions. The dual verbatim/standard annotation targets two failure modes conventional OCR structurally cannot exhibit: silent over-correction and fluent hallucination.
2. A large-scale evaluation of 15 contemporary VLMs and 3 conventional OCR systems across 3 prompting strategies, plus LoRA fine-tuning of 6 open-source models, complemented by a validated LLM-as-a-Judge protocol on the full test set.
3. A fifteen-class error taxonomy identifying visual mis-recognition, not orthographic confusion, as the dominant failure mode in Bengali scene text recognition — a pattern that holds across VLMs and one conventional OCR baseline, and that bounds what prompt engineering and fine-tuning can achieve.
4. A dataset with dual annotation, where the 3.43% of images whose in-image text deviates from standard spelling also carry an orthographically standard form in a separate field, enabling recognition errors (misreading the image) to be distinguished from correction errors (silently normalizing a misspelling).

Main Findings

• Best system: Gemini 2.5 Flash achieves the best result on BanglaWild, with CER 14.08 and WER 29.97 under the English prompt P1, its best deployment-realistic prompt. Under P3 (the oracle metadata condition) it reaches CER 13.93, but P3 is excluded from headline comparisons because the Context/Content metadata it prepends is annotator-provided and unavailable at inference time.

• Larger models do not outperform smaller ones within a family: Gemini 2.5 Flash outperforms the larger Gemini 2.5 Pro (CER 14.08 vs. 21.65; WER 29.97 vs. 36.42). Within Qwen3.5, the 9B model slightly surpasses the 27B dense and 35B-A3B MoE variants. InternVL3.5 and GPT-5.4-nano collapse across scales with CER above 100. The authors caution that the three Qwen3.5 variants differ architecturally rather than forming a clean scaling series.

• Prompt language helps the mid-tier but hurts the weakest: Bengali-instructed P2 improves CER over English P1 for 7 of 15 models, with the largest gains at mid-tier (Claude Sonnet 4.6: −4.69 CER; Claude Haiku 4.5: −8.30; Gemini 2.5 Pro: −3.20; GPT-5.4-nano: −54.15). The effect reverses for the weakest models: InternVL3.5-4B degrades from 118.44 to 156.92 and InternVL3.5-2B from 132.73 to 176.70 under P2, apparently because these models transcribe the Bengali instruction instead of treating it as a directive. P3 achieves the best CER for 9 of 15 models but rarely the best 1−NED.

• Fine-tuning rescues catastrophic failures but does not raise the ceiling: LoRA on the 50% training split (1,268 images) produces large CER reductions for weak open-source models — Gemma-3-4B-it drops from 134.82 to 50.72 (−62% relative), InternVL3.5-8B from 116.37 to 45.81 (−61%), and the two smallest InternVL variants from 118–133 to the low 50s. By contrast, the strongest open model, Qwen3.5-9B, slightly regresses (20.38 to 21.53). Gemma-3-12B-it improves on CER (63.89 to 28.91) and 1−NED (50.47 to 72.79) but still trails zero-shot Qwen3.5-9B.

• Visual mis-recognition dominates, not orthographic confusion: Lexical word substitution, single-character visual misread, and multi-character word corruption — all vision-encoder failures — account for roughly 56% of Gemini-2.5-Flash errors and roughly 60% of Qwen3.5-9B errors across all prompts. Orthographic categories (matra substitution, consonant homophones, conjunct/hasanta errors, nasal diacritics, compound orthographic edits) contribute only about 13% and 10%, giving visual-to-orthographic ratios of about 4.4 times and 5.8 times. On EasyOCR, the only conventional baseline that reliably reads Bengali, visual errors outnumber orthographic ones by 3.7 times.

• The classical conjunct bottleneck has largely closed: Conjunct/hasanta errors account for only 1.0–1.6% of errors, despite CRNN-era Bengali OCR treating yuktakshara recognition as the primary structural challenge. Even on EasyOCR, conjunct/hasanta errors reach only 4.63% of residual errors, about 3–4 times the VLM share but still secondary to segmentation and lexical substitution.

• Cross-script contamination is the only error category affected by prompting: It drops from 6.8% to 5.5% on Gemini and from 5.6% to 4.8% on Qwen under P2, then rebounds at P3 when English is reintroduced. Lexical substitution, grapheme misread, matra error, and conjunct error stay stable within ±1.5 pp across P1/P2/P3 for both models.

• Context and Content both strongly modulate difficulty: On the Context axis (Gemini 2.5 Flash, P1), CER rises from Printed Material (7.70) to Packaging (10.26), Signboard (13.69), Wall (13.74), and Vehicle (30.35) — a 3.9-times span. On the Content axis, CER rises from Instructional (8.76) to Religious (11.94), Advertisement (12.42), Creative (13.39), Political (17.42), and Colloquial (21.69) — a 2.5-times span. Both rankings hold across Claude Sonnet 4.6 and Qwen3.5-9B, indicating dataset rather than model effects.

• Effects compound in joint cells: The easiest cells are Printed Material × Instructional (CER 2.33) and Signboard × Instructional (5.81). The hardest are Vehicle × Creative (48.22–60.38), Vehicle × Colloquial (33.03–47.48), and Wall × Colloquial (30.05–34.91). Vehicle × Creative at 60.38 under P1 exceeds the sum-minus-mean prediction from the marginals, showing superadditive difficulty from the interaction of hand-painted surfaces and informal language.

• Layout and informality matter more than classic augmentation targets: Curved Text increases CER by 2.2 times for both Gemini 2.5 Flash (13.07 to 28.69) and Qwen3.5-9B (18.88 to 41.98); Heavy Occlusion by 1.6 and 1.3 times; Cluttered Background by 1.8 and 1.4 times. Artistic versus Printed fonts increase CER by only 1.4 and 1.2 times.

• Edit-distance and semantic judgments diverge at the top: Claude Sonnet 4.6 achieves the highest judge score across all fifteen systems (3.82 at P2) despite trailing Gemini 2.5 Flash on CER by about four points (18.46 vs. 14.08 at each model's best deployment-realistic prompt). Sonnet appears to produce more spelling-normalized yet meaning-preserving transcriptions, which edit distance penalizes but the judge does not.

• In-family judge bias was quantified: Gemini 2.5 Flash drops from {3.84, 3.86, 3.86} under the Gemini Pro judge to {3.68, 3.69, 3.70} under a Claude Sonnet 4.6 judge, and Gemini 2.5 Pro from {3.68, 3.72, 3.75} to {3.58, 3.67, 3.63}, implying in-family inflation of 0.16–0.17 points for Flash and 0.05–0.12 for Pro on the 1–5 scale. Budget constraints prevented out-of-family re-scoring of the remaining thirteen models.

• Cosmetic errors inflate WER without reducing readability: Word-boundary segmentation (12–19% of errors) and Unicode/punctuation normalization (5–11%) together account for 22–25% of all errors while leaving readable content unchanged. Qwen3.5-9B produces 2.3 times more Unicode-normalization noise than Gemini-2.5-Flash (about 11% vs. about 5%) despite visually identical output.

• Two rare categories carry disproportionate deployment risk: Semantic hallucination (0.6–1.0%) and numeric/temporal substitution (0.3–1.3%) are individually rare but produce fluent, plausible Bengali that cannot be recognized as incorrect from the output alone. Numeric substitutions are about 4 times more common in Gemini-2.5-Flash than Qwen3.5-9B (1.2% vs. 0.3%), with rates unchanged across prompts.

• Conventional OCR baselines trail badly: EasyOCR reaches CER 48.43, WER 89.26, 1−NED 48.45, judge 1.98. Tesseract reaches CER 98.46, WER 110.05, 1−NED 3.88, judge 1.01, with 62.80% of its errors being truncation or no output. Surya reaches CER 191.32, WER 160.48, 1−NED 11.52, judge 1.02, with 51.83% of its residual errors being script contamination.

• Exact-match accuracy separates the systems sharply: Tesseract 0.16%, EasyOCR 4.69%, Surya 0.63%, Gemini 2.5 Flash 35.6/35.9/35.3 across P1/P2/P3, Claude Sonnet 4.6 25.81/26.09/26.60, Qwen3.5-9B 16.1/16.3/16.1, and Gemma-3-12B-it 4.42/4.54/5.41.

• Annotation quality was verified: Three native Bengali-speaking annotators transcribed every image independently; 18.4% of images required adjudication on at least one character. Mean pairwise CER across all 2,535 images is 3.2%, and Cohen's kappa ranges from 0.81 to 0.93 across all categorical labels.

Methodology in Plain English

The authors first built the dataset. Trained collaborators used consumer mobile-phone cameras to photograph Bengali text on five kinds of public surfaces: walls, signboards, street vehicles, banners and posters, and consumer product packaging. From an initial pool of 3,210 images they discarded 675 (21.0%) that were too low-quality or weathered, unreadable, dominated by code-mixed Bengali-English text, or vulgar, leaving 2,535 images. Deduplication merged near-identical images sharing both text and appearance, while identical text on different surfaces, fonts, layouts, or lighting was kept because it poses distinct recognition challenges. Faces and vehicle registration plates captured incidentally were blurred and verified as non-identifying.

Each image was labeled along two orthogonal categorical axes. Context describes the physical medium (Vehicle, Signboard, Packaging, Wall, Printed Material), while Content captures communicative intent (Religious, Political, Advertisement, Instructional, Creative, Colloquial). Four diagnostic attributes were also recorded: Curved Text, Font Style, Occlusion, and Background Complexity. Three native Bengali-speaking annotators independently transcribed every image so that the gold transcription preserves the text exactly as it appears, including spelling errors and non-standard glyph substitutions; a separate field records the orthographically standard form for the 87 images (3.43%) containing non-standard spellings. For the attribute labels, two annotators worked with disagreements resolved by an expert in Bengali orthography. The data was split 50/10/40 into 1,268 training, 253 validation, and 1,014 test samples, and all models were evaluated on the common test split.

For evaluation, fifteen VLMs were tested zero-shot under three prompting strategies: an English baseline prompt (P1), a Bengali baseline prompt (P2), and a structured metadata prompt (P3) that prepends the gold Context and Content labels. Because those labels are annotator-provided and unavailable at real inference time, the authors treat P3 as an oracle condition and base headline claims on P1 and P2. Six open-source models were then fine-tuned with LoRA, which freezes the pretrained weight matrix and learns a low-rank update, with adapters applied to the attention and MLP projections of the language-model component only while the vision encoder and vision-language connector stayed frozen. Three conventional OCR systems (EasyOCR, Tesseract, Surya) served as non-generative baselines scored through the same pipeline.

Performance was measured with Character Error Rate, Word Error Rate, and 1−NED, supplemented by an LLM-as-a-Judge protocol on a 1–5 rubric-anchored scale, which the authors validated against native-annotator re-scoring (ρ = 0.89). Finally, every error was sorted into fifteen mutually exclusive categories by a deterministic classifier applied to four VLM families and the three OCR baselines, enabling analysis of how rather than merely how far predictions deviate.

Why This Matters

Impact on research. The paper inverts a long-standing assumption in Bengali OCR research: the classical conjunct bottleneck has largely closed, and the real obstacle for modern systems is visual recognition on cluttered, hand-painted, or oblique surfaces. Because the benchmark evaluates conventional OCR and generative VLMs on the same in-the-wild data, it also shows that the dominant failure mode is a property of the imagery rather than of any one model class. The dual verbatim/standard annotation provides a diagnostic capability no prior Bengali scene text benchmark supports, isolating silent over-correction and fluent hallucination that a CRNN structurally cannot produce.

Real-world applications.

• Transcribing Bengali addresses, receipts, and delivery labels from photographs taken in the field.
• Reading election material, protest slogans, and political posters, where numeric and semantic errors silently alter who or what is named.
• Digitizing medical packaging and instructional text, where a substituted number or word changes meaning in a safety-critical way.
• Reading vehicle signage, wall notices, and shop banners for translation, accessibility, or mapping tools in Bengali-speaking regions.

Industry relevance. The flat or inverse scaling results indicate that Bengali scene text recognition depends more on training data composition than on model size, which informs how teams allocate compute and which models they deploy. The finding that WER is inflated by 22–25% purely cosmetic word-boundary and Unicode-normalization differences — and that Qwen3.5-9B produces 2.3 times more Unicode noise than Gemini 2.5 Flash despite visually identical output — means WER-based deployment decisions for Bengali need to account for each model's share of cosmetic errors. The judge-bias measurement (0.16–0.17 points of in-family inflation for Flash) also matters for anyone using an LLM judge to select models.

Future Directions

• Code-mixed Bengali-English text. Code-mixed images were excluded at the filtering stage to keep the evaluation target unambiguous, leaving the code-mixing problem to future purpose-built resources.
• Balanced category coverage. The categorical distribution reflects natural abundance rather than balance: Wall dominates Context, and six of the thirty possible (Context, Content) combinations contain no images, so per-cell results in sparser categories carry higher variance than the macro-averages suggest.
• Larger-scale adaptation. The LoRA experiments adapted six open-source models on about 1,268 training images with a single random seed; the negative result on already-competent models tests targeted small-scale adaptation, not whether full-scale pre-training revision would help.
• Judge evaluation beyond a single family. Budget constraints prevented out-of-family re-scoring of thirteen of the fifteen models, so residual in-family inflation cannot be excluded for any non-Gemini cell,

Authors’ abstract

In-the-wild Bengali scene text recognition is largely unmeasured: existing resources target handwritten documents or constrained sign-board parsing, report only aggregate edit-distance metrics, and evaluate either conventional OCR or VLMs, never both on the same in-the-wild data. To address this gap, we introduce BANGLAWILD, a benchmark of 2,535 Bengali scene text images, each paired with a verbatim gold transcription, two categorical axes, four diagnostic attributes, and an orthographically standard form where the in-image text deviates from canonical spelling. We evaluate fifteen VLMs and three conventional OCR systems under three prompting strategies, fine-tune 6 open-source models with LoRA, and complement edit-distance metrics with an LLM-as-a-Judge evaluation. Our results reveal a persistent gap in which larger models within the same family do not outperform smaller ones. Our fifteen-class error taxonomy shows that visual mis-recognition accounts for ~60% of errors in the strongest systems, while conjunct-related errors contribute under 2%, challenging a long-standing assumption in Bengali OCR research; the same visual dominant profile also holds across architectures, including the one conventional baseline that reads Bengali reliably. Prompt language mainly affects cross-script drift and LoRA reduces catastrophic failures in weak models without lifting the ceiling on already competent ones. Code and data will be publicly released.