Bharat Scene Text: A Novel Comprehensive Dataset and Benchmark for Indian Language Scene Text Understanding

AI summary

Overview

Research area: Computer vision / scene text understanding, specifically multilingual OCR for Indian languages, with a dataset and benchmark contribution.

Technical level: Intermediate. The paper assumes familiarity with scene text detection, recognition architectures (CRNN, PARSeq, DBNet, TextBPN++) and evaluation metrics (F1, WRR, CRR), but the dataset and benchmark framing is accessible to anyone working on Indic NLP or OCR.

Scope: The paper introduces the Bharat Scene Text Dataset (BSTD), a 6,582-image, 1,26,292-word benchmark covering 11 Indian languages plus English, along with a baseline pipeline, open-source toolkit, and evaluations of detection, script identification, cropped word recognition, and end-to-end recognition.

What This Paper Is About

Scene text recognition in English is often treated as nearly solved, but recognizing text in Indian language signage remains an open problem because of script diversity, non-standard fonts, varying writing styles, and, most importantly, the lack of high-quality datasets and open-source models. The authors build BSTD to fill that gap: a manually annotated, multi-task benchmark of Indian street-scene text collected from Wikimedia Commons images and spanning 11 Indian languages plus English. They then adapt existing state-of-the-art English-oriented architectures into a working baseline to establish reference numbers on the new benchmark.

Key Contributions

1. The Bharat Scene Text Dataset (BSTD): 6,582 scene images containing 1,26,292 words in 11 Indian languages and English, manually annotated with polygon-level bounding boxes, transcription, and script labels. Images were sourced from Wikimedia Commons across different Indian linguistic regions.
2. A strong baseline and open-source toolkit: A pipeline that detects, identifies the script of, and recognizes text in English and 11 Indian languages, released as an easy-to-install, replicable toolkit (project page: https://vl2g.github.io/projects/IndicPhotoOCR/).
3. Benchmarks across four tasks: Detailed evaluations with failure-case analysis for scene text detection, script identification, cropped word recognition, and end-to-end scene text recognition, comparing off-the-shelf methods against models fine-tuned on synthetic and/or BSTD training data.
4. A multilingual evaluation protocol: End-to-end reporting using mean WRR and CRR (with reading order relaxed, since reading order in scene images is subjective), plus precision, recall, and F1.

Main Findings

• Detection: TextBPN++ clearly leads on BSTD with P 0.75 / R 0.78 / F1 0.77 (TedEval). Other detectors lag: DBNet (0.71 / 0.50 / 0.59), Hi-SAM (0.75 / 0.19 / 0.46), CRAFT (0.51 / 0.22 / 0.19), EAST (0.55 / 0.10 / 0.17). Hi-SAM is comparably precise but misses many Indian language words.
• Script identification (3-way): The proposed ViT-based model beats AlexNet, CRNN, and CLIP for every regional language tested, ranging from 79.6% (Marathi) to 95.0% (Telugu), with most languages in the 90–95% range.
• Script identification (12-way): The proposed ViT-based approach reaches 80.5% accuracy versus 67.7% for the CLIP baseline. Errors concentrate on Assamese/Bengali confusion and Hindi/Marathi confusion, which the authors attribute to shared Unicode patterns and note is largely harmless for end-to-end performance.
• Cropped word recognition: Fine-tuning PARSeq on real BSTD data raises average word recognition rate from 0.47 (synthetic-only training) to 0.73. English reaches 0.92; a second tier follows with Marathi 0.86, Bengali 0.82, Tamil 0.80, Assamese 0.79, Punjabi 0.75, Odia 0.72, Hindi 0.71; Gujarati 0.61 and Kannada 0.60 are lower, and Telugu 0.56 and Malayalam 0.58 remain in the 50% range.
• Case sensitivity matters: With case-sensitive decoding and evaluation instead of the default case-insensitive setting, English WRR drops from 92% to 84%.
• Off-the-shelf tools struggle on cropped words: CRNN (AI4Bharat) averages 0.16, PaddleOCR 0.29, EasyOCR 0.18, and Tesseract 0.15 across languages.
• End-to-end (WRR): The proposed baseline averages 0.36 versus SuryaOCR 0.14, Tesseract 0.02, ChatGPT 0.13, and Google OCR 0.41. Giving the baseline oracle text detection lifts it to 0.62; oracle detection plus oracle script identification lifts it to 0.71.
• End-to-end (CRR): The proposed baseline averages 0.54, against Google OCR 0.55, SuryaOCR 0.28, ChatGPT 0.21, and Tesseract 0.08. With oracle detection the baseline reaches 0.77, and with oracle detection plus oracle script identification 0.88.
• End-to-end (F-score): The baseline averages F 0.45, compared with Google OCR 0.51, SuryaOCR 0.20, ChatGPT 0.15, and Tesseract 0.03. With oracle detection this rises to 0.64 and with oracle detection plus script identification to 0.70.
• Error analysis shows headroom in the pipeline itself: The authors state that further improving script identification would significantly boost overall end-to-end recognition performance.
• Dataset composition: Of the 6,582 scene images, 53 contain exclusively English text, 1,958 contain only Indian-language text, 4,518 contain English co-occurring with one or more Indian languages, and 53 contain no annotatable scene text (deliberately retained to reflect real collection pipelines).
• Data difficulty: Bounding box areas range from a minimum of 32 pixels to a maximum of approximately 2.49 × 10^6 pixels, with a distribution peaking near a median area of 2,889 pixels; the high proportion of small regions indicates a lot of incidental rather than focused text. Roughly 89% of images were captured in daylight and 11% at night; 91% are outdoor and 9% indoor.

Methodology in Plain English

Building the dataset. The authors first compiled lists of Indian cities and towns where the 11 target languages are spoken, and combined them with public-place keywords such as "railway station," "hospital," "ATM," and "shop entrance" to query the Wikimedia Commons API. That produced roughly 50K images.

Filtering. They ran a pre-trained DBNet scene text detector (fine-tuned on Indic synthetic data) to keep images containing at least one word and discard the rest, then manually filtered further. The final set is 6,582 images.

Annotation. Annotation was semi-automatic. DBnet (fine-tuned on synthetic Indian language data) proposed polygon bounding boxes, which human annotators refined by editing, drawing, or deleting polygons. Annotators with knowledge of each image's geographic origin assigned a script label to every box. PARSeq (fine-tuned on synthetic Indic data) then read the text, and language-expert annotators corrected recognition errors. Each image also carries lighting (day/night) and environment (outdoor/indoor) labels. Three annotators handled image filtering and bounding boxes; eleven language experts, one per target language, handled text correction and script assignment. All annotations passed an additional quality check. The whole process (pseudo-annotation, manual correction, quality verification) took 10 months, and annotators were paid at rates aligned with standard wages for a tier-2 Indian city.

Synthetic data. Because real Indic scene text is scarce, the authors used SynthText with 8,000 background images to generate large-scale synthetic word images per language, drawing vocabulary from the AI4Bharat-IndicNLP Dataset. Synthetic training set sizes ranged from 3,168,386 (Marathi) to 8,475,805 (Bengali) samples, with per-language vocabularies and font counts listed in the paper's Table 5.

The baseline pipeline. Three modules: (1) detection with TextBPN++, chosen after finding modern detectors largely language-agnostic; (2) script identification with a ViT (google/vit-base-patch16-224-in21k, 12 transformer layers, 16×16 patch embeddings, ImageNet-21k pre-trained) with a custom classification head, fine-tuned with AdamW at learning rate 2 × 10^-4 and batch size 16; (3) recognition with PARSeq, trained on synthetic data per language and then fine-tuned on BSTD, using 32×128 input images, a maximum label length of 25, and random augmentation.

Evaluation. An 80-20 image-level split gives 5,263 train and 1,319 test images. For script identification, equal numbers of word images were drawn per language: 1,800 training and 478 testing images per language. The dataset, annotations, models, and toolkit are released under open licenses.

Why This Matters

Impact on research. Indian languages are used by more than 1.4 billion people, roughly 18% of the global population, yet the paper argues that no comprehensive public benchmark existed that supports detection, script identification, and recognition together across 11 Indian languages. Table 1 shows BSTD as the only publicly available dataset that supports all three core tasks end-to-end for Indian languages, compared with earlier efforts (IIIT-ILST, IIIT-TL-STR, IIIT-indicSTR12, IIIT-STR) that are limited to cropped word recognition or fewer languages. The new numbers give the field a common measuring stick.

Real-world applications:

• Assistive technology that reads signage aloud for visually impaired users in Indian cities.
• Search and e-commerce systems that need to index products, shop names, and street addresses from photos.
• Navigation and transport, where signboards, metro signs, and bus station text must be parsed automatically.
• Document and translation tools for public services that operate across multiple Indian scripts.

Industry relevance. The comparison against commercial systems (Google OCR, GPT-4 with vision) and open-source tools (Tesseract, PaddleOCR, EasyOCR, SuryaOCR, CRNN) gives practitioners a concrete picture of what off-the-shelf OCR can and cannot do on Indic scene text, and the released toolkit provides a reproducible starting point rather than a from-scratch build.

Future Directions

• Annotating the "Others" category: Table 3 lists 19,814 word images not among the 11 listed languages (for example Meitei and Urdu); the authors plan to include their annotation in subsequent versions of the dataset.
• Improving script identification to lift end-to-end performance: The paper explicitly states that further gains in script identification would significantly boost overall recognition, and the oracle script-identification experiments quantify that headroom.
• Closing the low-resource recognition gap: Telugu and Malayalam remain in the 50% WRR range and Gujarati and Kannada near 60%, so better recognition for these languages is an open problem.
• Strengthening closed-source baselines: The authors note that more detailed prompting of GPT-4 with vision, such as asking it to locate text or use chain-of-thought reasoning, could improve that baseline but was beyond the scope of this paper.

Target Audience

Researchers and engineers working on scene text detection and recognition, multilingual OCR, and Indic language computing. It is directly useful to anyone building OCR pipelines for Indian languages, to dataset and benchmark designers in computer vision, and to applied teams in assistive technology, e-commerce, and mapping who need to know what current models can achieve on Indian-language signage. Readers without a background in scene text tasks will need to pick up terms like polygon bounding boxes, script identification, WRR, and CRR, but the dataset description and the headline results are readable without deep model expertise.

Authors’ abstract

Reading scene text, that is, text appearing in images, has numerous application areas, including assistive technology, search, and e-commerce. Although scene text recognition in English has advanced significantly and is often considered nearly a solved problem, Indian language scene text recognition remains an open challenge. This is due to script diversity, non-standard fonts, and varying writing styles, and, more importantly, the lack of high-quality datasets and open-source models. To address these gaps, we introduce the Bharat Scene Text Dataset (BSTD) - a large-scale and comprehensive benchmark for studying Indian Language Scene Text Recognition. It comprises more than 100K words that span 11 Indian languages and English, sourced from over 6,500 scene images captured across various linguistic regions of India. The dataset is meticulously annotated and supports multiple scene text tasks, including: (i) Scene Text Detection, (ii) Script Identification, (iii) Cropped Word Recognition, and (iv) End-to-End Scene Text Recognition. We evaluated state-of-the-art models originally developed for English by adapting (fine-tuning) them for Indian languages. Our results highlight the challenges and opportunities in Indian language scene text recognition. We believe that this dataset represents a significant step toward advancing research in this domain. All our models and data are open source.