ViInfographicVQA: A Benchmark for Single and Multi-image Visual Question Answering on Vietnamese Infographics

AI summary

Overview

Research area: Computer Vision / Multimodal Document Understanding — specifically Visual Question Answering (VQA) over infographics in Vietnamese.

Technical level: Intermediate. The paper introduces a dataset and benchmark; understanding the results requires familiarity with VQA, OCR, layout understanding, ANLS scoring, and vision-language model fine-tuning, though the core ideas are explained accessibly.

Scope (one sentence): The paper builds and evaluates ViInfographicVQA, the first Vietnamese InfographicVQA benchmark, covering both single-image question answering and cross-image (multi-image) reasoning over real-world infographics.

What This Paper Is About

Infographics combine text, charts, icons, tables, and layout design, so answering questions about them requires reading, locating, and reasoning over several modalities at once — far more than answering questions about natural photos. Existing infographic benchmarks such as InfographicVQA are English-centric, and existing Vietnamese VQA datasets (ViVQA, EVJVQA, OpenViVQA, ViTextVQA, and others) target natural images, scene text, or synthetic renderings rather than infographics. This paper fills that gap by introducing a Vietnamese infographic VQA benchmark with two evaluation settings: a Single-image task and a Multi-image task that requires combining evidence across several semantically related infographics.

Key Contributions

1. A new benchmark: ViInfographicVQA, described as the first Vietnamese InfographicVQA dataset, with over 6747 infographics and nearly 20409 QA pairs, released with official train/test splits. The paper states the splits total 6096 train images and 651 test images, and 18399 train QAs and 2010 test QAs.
2. Two evaluation settings: a Single-image task following the traditional setup, and a Multi-image task that requires synthesizing evidence across multiple semantically related infographics — described as the first Vietnamese evaluation of cross-image reasoning in VQA.
3. A structured annotation framework: standardized answer-source categories (Image-span, Question-span, Multi-span, Non-extractive for single-image; Multi-image span, Cross-image synthesis, Non-span for multi-image), with VLM-assisted generation followed by human verification, and rules to avoid rationale leakage.
4. Baselines and fine-tuning analysis: systematic evaluation of recent open-source vision-language models (VLMs) plus a fine-tuned Qwen2.5-VL-7B variant, including an ablation over supervision strategies.

Main Findings

• Single-image performance is moderate, not solved: Ovis2.5-9B achieves the highest overall ANLS on the Single-image task (71.02), followed by Qwen2.5-VL-7B (67.42), InternVL3.5-8B (67.02), Qwen2.5-VL-7B fine-tuned (67.80), MiniCPM-o2.6-8B (57.03), VideoLLaMA3 Image-7B (49.60), Phi-4-multimodal-5B (41.91), and Molmo D-7B (38.26).
• Span questions are easiest: Image-span and Question-span categories remain the easiest, often exceeding 75% ANLS, while Non-extractive (numeric reasoning such as counting, arithmetic, rank, or percentage) is consistently lower across models. Ovis2.5-9B leads on Multi-span and most element types, particularly tables and graphs, while InternVL3.5-8B performs best on Non-extractive reasoning.
• Multi-image performance drops sharply: Multi-image accuracy falls relative to Single-image, typically by about 12–32 ANLS points. Best overall on the Multi-image task is Qwen2.5-VL-7B fine-tuned (55.47), then base Qwen2.5-VL-7B (54.92), MiniCPM-o2.6-8B (40.55), InternVL3.5-8B (35.50), and Phi-4-multimodal-5B (20.95). Cross-image synthesis is the weakest category for several models (for example, InternVL3.5-8B scores 10.97 there versus 24.67 on Non-span).
• Fine-tuning helps, but only modestly: QLoRA fine-tuning (r = 8, alpha = 32, dropout 0.05, 5 epochs, learning rate 2×10⁻⁴) yields moderate gains for Qwen2.5-VL, slightly improving arithmetic and reasoning-oriented categories.
• Combined supervision is best overall: In the ablation, zero-shot inference scores 63.43 combined, single-image-only fine-tuning 63.64, multi-image-only fine-tuning 63.52, and combined fine-tuning 64.41. Multi-image-only fine-tuning boosts cross-image reasoning (54.74 to 55.47 on Multi-image), while joint training gives the best overall trade-off.
• Dataset composition: Single-image questions account for 68.1% of QA pairs and Multi-image questions 31.9%. In the Single-image setting, the four answer-source categories appear with comparable frequencies; in the Multi-image setting, Multi-image span is the largest proportion, followed by Non-span and Cross-image synthesis. The top-5 image categories are Economy-Integration, Healthcare-Community, Culture-Society, Disaster-Accident, and Sports-Arts.

Methodology in Plain English

The team built the dataset in three main phases.

Data curation. They crawled real Vietnamese infographics from infographics.vn, a portal maintained by the Vietnam News Agency, covering topics such as Economics, Healthcare, Culture-Society, Disaster-Accident, and Sports-Arts. They kept only images with an aspect ratio within [0.33, 3.00] and a shorter side of at least 512 pixels, so that text and layout remain readable. They then computed multimodal representations combining OCR text and its locations with visual and layout cues, and clustered images within each topic (3 clusters per topic) using cosine similarity to form semantically related sets. Entire Multi-image sets were assigned to a single split to prevent cross-split leakage, and splits were stratified by topic × answer-source.

Annotation. For single infographics, a VLM-assisted scan proposes informative regions (text boxes, chart panels, legends, icons, map areas), and rule-based prompt templates generate questions along two dimensions: element type (table, graph/diagram, map, text, layout/visual) and answer-source category (Image-span, Question-span, Multi-span, Non-extractive). For multi-image sets, they first define the relation linking images using four types — Identical Replication; Content Augmentation/Variation within the Same Specific Subject; Content Replicated, Presentation Varies; and Thematic Connection with Diverse Specific Subjects/Focus — then generate questions that genuinely require cross-image evidence, categorized as Multi-image span, Cross-image synthesis, or Non-span.

Verification and refinement. Gemini 2.0 Flash was used semi-automatically to expand candidate QAs and produce brief validation explanations; only answers (without rationales) are released. Human reviewers filtered out questions answerable without the image, refined wording, and checked naturalness and faithfulness. An automated checker removed duplicates and verified that each multi-image question truly depends on at least two images. Finally, deduplication, automated VLM validation, and human review on a web platform produced the released dataset.

Evaluation. Models are scored with Average Normalized Levenshtein Similarity (ANLS), which credits near-matches and penalizes semantically incorrect answers, scoring 0 for normalized Levenshtein distance of 0.5 or above.

Why This Matters

Impact on research: The paper argues that current VLM pipelines rely on shallow OCR–language fusion and lack explicit mechanisms for layout-aware reasoning and cross-document information integration (such as memory over sets, retrieval and alignment, or planning-based composition). It also highlights that ANLS, while robust to minor orthographic variation, does not capture semantic correctness, numerical exactness, grounding faithfulness, or calibration — limitations most visible on non-extractive and multi-image questions. The benchmark gives the community a low-resource-language testbed for both problems.

Real-world applications:

• Automated question answering over government and news agency infographics from portals such as Vietnam News Agency's infographics.vn, in Vietnamese.
• Document and report assistants for economics, healthcare, and education materials where figures and charts carry the key information.
• Accessibility and information-retrieval tools that let users ask questions about data-rich visuals instead of reading dense charts and tables.
• Multilingual document AI systems that need to handle low-resource languages where English-centric benchmarks offer no signal.

Industry relevance: Organizations deploying document AI in Vietnamese or other low-resource markets can use ViInfographicVQA to measure whether a model actually reasons over charts, tables, and layouts rather than just extracting text. The multi-image setting reflects real workflows where an analyst must combine several related reports, and the paper's finding that fine-tuning helps but leaves a substantial multi-image gap signals where engineering effort is still needed.

Future Directions

• Layout-aware and cross-image modeling: Developing architectures with explicit mechanisms for layout reasoning and for integrating information across multiple related documents, since cross-image synthesis is identified as the key bottleneck.
• Richer evaluation beyond ANLS: Complementing ANLS with task-specific measures such as exact and relative numeric accuracy, unit normalization, faithfulness and attribution checks, and targeted human assessment for ambiguous cases.
• Broader language and domain coverage: Extending the benchmark to multilingual and cross-lingual settings, auditing source and template biases in the collected infographics, and probing domain transfer.
• Data-efficient adaptation without rationale leakage: Exploring training strategies that improve reasoning under limited supervision while keeping explanations internal, as the paper's annotation pipeline does.

Target Audience

Researchers and practitioners in multimodal document understanding, VQA, and chart/infographic reasoning; teams building Vietnamese or low-resource-language document AI; and anyone evaluating vision-language models on layout-heavy, numerically grounded questions. Readers interested in dataset construction methodology — VLM-assisted generation with human verification, leakage prevention, and stratified splits — will also find it useful.

Authors’ abstract

Infographic Visual Question Answering (InfographicVQA) evaluates a model's ability to read and reason over data-rich, layout-heavy visuals that combine text, charts, icons, and design elements. Compared with scene-text or natural-image VQA, infographics require stronger integration of OCR, layout understanding, and numerical and semantic reasoning. We introduce ViInfographicVQA, the first benchmark for Vietnamese InfographicVQA, comprising over 6747 real-world infographics and 20409 human-verified question-answer pairs across economics, healthcare, education, and more. The benchmark includes two evaluation settings. The Single-image task follows the traditional setup in which each question is answered using a single infographic. The Multi-image task requires synthesizing evidence across multiple semantically related infographics and is, to our knowledge, the first Vietnamese evaluation of cross-image reasoning in VQA. We evaluate a range of recent vision-language models on this benchmark, revealing substantial performance disparities, with the most significant errors occurring on Multi-image questions that involve cross-image integration and non-span reasoning. ViInfographicVQA contributes benchmark results for Vietnamese InfographicVQA and sheds light on the limitations of current multimodal models in low-resource contexts, encouraging future exploration of layout-aware and cross-image reasoning methods.