Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmark

AI summary

Overview

Research area: Computer Vision / Multimodal NLP — specifically OCR-integrated Visual Question Answering (VQA) for low-resource languages.

Technical level: Advanced. The paper assumes familiarity with vision-language models, OCR pipelines, transformer-based language models, and agent-based architectures.

Scope: The paper introduces ViSignVQA, a Vietnamese signboard-oriented VQA dataset of 10,762 images and 25,573 question-answer pairs, together with OCR-integrated baselines and a GPT-4 multi-agent framework.

What This Paper Is About

Most VQA research focuses on English and on general scenes, while answering questions about the text physically printed on shop signs — in Vietnamese — has been almost entirely unaddressed. The authors build the first large-scale Vietnamese dataset dedicated to signboard VQA, then test whether adding OCR text from those signs helps existing models answer questions correctly. They also prototype a multi-agent system that separates the "seeing" step from the "reasoning" step to push accuracy higher.

Key Contributions

1. ViSignVQA dataset. The authors created what they describe as the first high-quality, large-scale VQA dataset for Vietnamese signboards, containing 10,762 images and 25,573 question-answer pairs, with annotations covering signboard relevance (is_related), color dependence (is_color), question/answer length, part-of-speech, named entities, and detected objects.

2. OCR-integrated VQA baselines and a multi-agent framework. They adapted state-of-the-art VQA models (BLIP-2, LaTr, PreSTU, and SaL) by plugging in a Vietnamese OCR model (SwinTextSpotter) and a Vietnamese pretrained language model (ViT5), and additionally proposed a multi-agent VQA architecture combining perception and reasoning agents with GPT-4, reported at 75.98% accuracy via majority voting.

3. Analysis of OCR challenges on signboards. They assessed how well OCR systems extract text from signboard images in order to characterize the dataset's difficulty.

4. Survey of prompt-based fine-tuning. They surveyed and compared prompt-based fine-tuning methods for improving large language model performance on Vietnamese language tasks.

Main Findings

• OCR context matters enormously. Appending OCR-extracted text to the questions produced F1-score improvements of up to 209% across the adapted models.
• Multi-agent voting achieves 75.98% accuracy. The proposed framework, which combines perception and reasoning agents with GPT-4, reached 75.98% accuracy through majority voting.
• The dataset is large by Vietnamese standards. ViSignVQA has 10,762 images and 25,573 QA pairs, compared in Table 1 with ViVQA (10,328 images, 15,000 questions), OpenViVQA (11,199 images, 21,271 questions in the table), ViTextVQA (16,762 images, 50,342 questions), and ViOCRVQA (28,282 images, 123,781 questions).
• Questions are moderately short and varied. Question length in the development set ranged from 4 to 27 tokens, with an average of 9.59 tokens.
• Answers skew short but can be long. Answer lengths concentrate in the 1-10 token range, peaking at around 3-6 tokens, with the longest reaching 43 tokens.
• Nouns dominate, numerals are unusually frequent in answers. Question POS counts include nouns at over 7,077 occurrences, verbs at approximately 3,850, pronouns at more than 3,019, punctuation (CH) at 2,528, and prepositions (E) at around 1,537. In answers, nouns appear about 5,200 times and numerals over 2,110 times — the paper attributes the high numeral count to phone numbers and prices displayed on signs.
• Named entities are rare, and locations outweigh organizations. In questions, non-entity tokens ("O") exceed 19,000, while I-LOC and B-LOC appear 295 and 267 times. In answers, "O" occurs 9,276 times, B-LOC 846, I-LOC 886, B-PER 670, I-PER 131, B-ORG 9, and B-MISC/I-MISC 3 each.
• Color is rarely decisive. Of the development set, 2,266 QA pairs were labeled is_color = 0 and 200 were labeled is_color = 1.
• People dominate the visual scenes. Using YOLOv10 for object annotation, the category "person" appears nearly 4,000 times, far more than any other object; "motorcycle" and "bicycle" occur around 1,000 times each, with a long tail of rare objects such as "umbrella," "dog," "car," "microwave," and "snowboard."
• Annotation agreement was very high. Inter-annotator agreement measured by F1-score was 98.94%, above the required threshold of exceeding 70%.
• The dataset was assembled by filtering existing resources. Images were drawn from OpenViVQA and ViTextVQA, over 14,000 signboard images were collected, and over 10,762 high-quality images were retained after manual filtering. Collection and annotation spanned six weeks with two native Vietnamese speakers.

Methodology in Plain English

The authors did not photograph signs from scratch. They started from two existing Vietnamese VQA datasets (OpenViVQA and ViTextVQA) that already contained plenty of signboard images. They used SwinTextSpotter, an OCR model, to read all the words in every image, then applied two filters: keep images whose questions contain signboard-related keywords such as "Biển" and "ghi," and keep images where the answer contains a word the OCR actually read off the image. Human annotators then removed images with no signboard, non-Vietnamese signboards, signboards that were too small, or signboard content that was not visible — also excluding banners, ribbons, and signs on vehicles. Duplicate images were deleted.

For questions and answers, they reused and refined existing QA pairs rather than writing all-new ones. They added an is_related label (1 if the QA pair concerns the signboard, directly or by inference from surrounding context; 0 otherwise), applied only to the development set, and validated it with the F1-score. The dataset was split 7:1:2 into training, development, and test.

For modeling, they took existing strong VQA models — BLIP-2, LaTr, PreSTU, and SaL — and gave them extra help: they ran a Vietnamese OCR model (SwinTextSpotter) over the image and appended the recognized text to the question, using ViT5, a Vietnamese pretrained language model, on the language side. Separately, they built a multi-agent system in which different agents handle perception (reading the image and its text) and reasoning (answering), with GPT-4 producing answers that are then combined by majority voting.

Why This Matters

Research impact. ViSignVQA fills a clear gap: prior OCR-VQA resources were English and centered on book covers (OCR-VQA-200k, ViOCRVQA), while general Vietnamese VQA datasets like ViVQA, OpenViVQA, and ViTextVQA were not designed around signage. The 209% F1 improvement from appending OCR text gives a quantitative signal that domain-specific scene-text resources, not just bigger general models, drive performance for low-resource languages.

Real-world applications:

• Assisting tourists and newcomers in navigating Vietnamese streets where signs are in Vietnamese only, sometimes bilingual, and often informally worded.
• Supporting accessibility tools for visually impaired users who need spoken descriptions of what a nearby sign says.
• Enabling mapping and street-level data collection from crowd-sourced photos, converting sign content (store names, phone numbers, prices) into structured data.
• Powering delivery, logistics, and autonomous navigation systems that must read storefront signage to confirm a destination.

Industry relevance. Any company deploying vision-language systems in Vietnam — mapping platforms, delivery fleets, e-commerce localizers, or OCR vendors — gets a benchmark tailored to real signage rather than translated English data. The finding that OCR-augmented prompting outperforms canned model behavior suggests a cheap, practical deployment path that avoids full retraining, and the 75.98% multi-agent result shows agent orchestration is a viable accuracy lever.

Future Directions

• Extending the is_related labeling beyond the development set. The authors state that resource constraints limited this labeling to the development set, explicitly noting it was done "to facilitate the testing of certain methods in future experiments."
• Closing the accuracy gap to human level. The strongest reported system reaches 75.98% accuracy, leaving substantial room for improvement on the benchmark.
• Handling the long tail of visual and linguistic cases. Rare objects (e.g., "microwave," "snowboard" in the annotation tail), rare entity types (B-ORG appears only 9 times; B-MISC and I-MISC 3 times each), and infrequent Wh-question types (How, Who, Whose, When, Why) remain underexplored and are likely to be where models fail.
• Leveraging visual attributes beyond text. The is_color split (200 questions requiring color versus 2,266 that do not) indicates that models must integrate non-textual visual cues with OCR, an integration the paper frames as an added challenge rather than a solved one.

Target Audience

Researchers working on multimodal vision-language models, OCR-integrated VQA, and low-resource or Southeast Asian language technology will find this most directly useful, as will NLP groups seeking Vietnamese benchmarks. Practitioners building scene-text understanding products for Vietnamese markets — mapping, accessibility, delivery, and retail analytics — can use ViSignVQA as an evaluation set. The dataset's detailed linguistic statistics (POS, NER, question type, length distributions) also make it valuable to linguists studying Vietnamese signage conventions, including bilingual text, informal phrasing, and layout-dependent meaning.

Note: The paper content provided was truncated. Detailed per-model benchmark tables (beyond the reported 209% F1 improvement and 75.98% multi-agent accuracy) are not fully reported in the available text, nor are the specific per-model scores for BLIP-2, LaTr, PreSTU, and SaL. The related-work section states OpenViVQA comprises 11,199 images and 37,914 QA pairs in the narrative text while Table 1 lists 21,271 questions; both figures appear in the source.

Authors’ abstract

Understanding signboard text in natural scenes is essential for real-world applications of Visual Question Answering (VQA), yet remains underexplored, particularly in low-resource languages. We introduce ViSignVQA, the first large-scale Vietnamese dataset designed for signboard-oriented VQA, which comprises 10,762 images and 25,573 question-answer pairs. The dataset captures the diverse linguistic, cultural, and visual characteristics of Vietnamese signboards, including bilingual text, informal phrasing, and visual elements such as color and layout. To benchmark this task, we adapted state-of-the-art VQA models (e.g., BLIP-2, LaTr, PreSTU, and SaL) by integrating a Vietnamese OCR model (SwinTextSpotter) and a Vietnamese pretrained language model (ViT5). The experimental results highlight the significant role of the OCR-enhanced context, with F1-score improvements of up to 209% when the OCR text is appended to questions. Additionally, we propose a multi-agent VQA framework combining perception and reasoning agents with GPT-4, achieving 75.98% accuracy via majority voting. Our study presents the first large-scale multimodal dataset for Vietnamese signboard understanding. This underscores the importance of domain-specific resources in enhancing text-based VQA for low-resource languages. ViSignVQA serves as a benchmark capturing real-world scene text characteristics and supporting the development and evaluation of OCR-integrated VQA models in Vietnamese.