ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures

AI summary

Overview

Research area: Multimodal AI evaluation for scientific figure comprehension, benchmark and competition design in computer vision and document analysis, applied to Atomic Layer Deposition/Etching (ALD/E) materials science.

Technical level: Intermediate. The paper is a competition overview rather than a new model architecture, but it assumes familiarity with vision-language models, LoRA/QLoRA fine-tuning, ROUGE, BERTScore, TEDS, and relative mapping similarity.

Scope (one sentence): The paper reports the organization, dataset construction, task design, and results of the ICDAR 2026 Sci-ImageMiner competition, the first expert-annotated benchmark dataset and community competition for end-to-end information extraction and reasoning over ALD/E scientific figures.

What This Paper Is About

Scientific figures such as charts, plots, spectra, and band diagrams carry knowledge that the surrounding text of a publication often does not state explicitly, but understanding them requires both visual perception and domain-specific reasoning. Existing chart datasets and prior ICDAR competitions on scientific figure understanding either target general real-world or synthetic charts, or cover material science with limited scope, figure types, and task formulations. This paper describes the Sci-ImageMiner benchmark and its accompanying ICDAR 2026 competition, which curate an expert-annotated ALD/E figure dataset across four end-to-end tasks and measure how well state-of-the-art multimodal models perform on each.

Key Contributions

1. A new expert-annotated benchmark dataset (Sci-ImageMiner). 205 ALD/E research publications were collected, spanning experimental and simulation-based studies, yielding 1,951 figures organized under a 49-figure taxonomy and split into train (128 papers, 1,180 figures), dev (20 papers, 201 figures), and test (57 papers, 570 figures) partitions.

2. A four-task competition design covering progressive scientific comprehension. The competition defines Task 1 Classification, Task 2 Data Extraction, Task 3 Summarization, and Task 4 Visual Question Answering, each with its own evaluation protocol and leaderboard.

3. A custom multi-user annotation platform and annotation protocol. The authors built a web-based platform supporting subfigure-level bounding-box labeling and a unified JSON schema, used Qwen2.5-VL-7B-Instruct for automated pre-annotation of classification, data extraction, and summarization (with VQA reserved for manual expert annotation), and recruited 10 domain experts with postgraduate to postdoctoral qualifications, measuring inter-annotator agreement of 0.46 Fleiss' Kappa on a shared 5-paper subset.

4. Baseline evaluations and a comparative analysis of competing approaches. The paper establishes LVLM baselines and summarizes the methods of top-5 leaderboard teams, showing that no single baseline performed well across all tasks and that state-of-the-art models perform well on classification and summarization but struggle with data extraction and visual question answering.

Main Findings

• Participation was substantial. The competition ran from 9th January 2026 to 8th April 2026, attracting 68 active participants and 1,263 public and private submissions. By task, the reported figures are: Classification 81 participants and 501 submissions, Data Extraction 53 and 335, Summarization 43 and 212, and Visual Question-Answering 47 and 215.

• Classification was the strongest area of performance. Ricoh_SRCB ranked first with 0.79 Accuracy and 0.81 F1-score, followed by BOE AI+ (0.76 / 0.77), IIT_PATNA_CV_1 (0.76 / 0.77), DocMiner (0.75 / 0.76), and VLMinators (0.74 / 0.75). The best baseline, Gemma 4 E4B 8b, scored 0.67 Accuracy and 0.68 F1-score.

• Data extraction scores were low in absolute terms for the RMS component. TeleOCR-VL ranked first with RMS 17.23, TEDS 66.39, and a weighted score of 41.81, ahead of VLMinators (17.29 RMS, 64.31 TEDS, 40.80 weighted), Ricoh_SRCB (16.23 / 61.12 / 38.67), Vassilis Sioros (14.94 / 55.20 / 35.07), and DocMiner (12.67 / 53.72 / 33.19). The Qwen 3 VL 8b baseline reached RMS 14.08, TEDS 57.86, and weighted 35.97, outperforming two of the five listed top teams.

• Summarization scores clustered closely among top teams, with a clear margin over baseline. DeepVitminC ranked first with R1 0.37, R2 0.17, RL 0.30, BERT-F1 0.84, and weighted 0.56, followed by Ricoh_SRCB (0.37 / 0.16 / 0.29 / 0.83 / 0.55), TeleOCR-VL (0.35 / 0.15 / 0.28 / 0.82 / 0.54), BOE AI+ (0.35 / 0.15 / 0.28 / 0.82 / 0.54), and Vassilis Sioros (0.34 / 0.14 / 0.27 / 0.82 / 0.53). The Intern VL 3.5 8b baseline scored 0.26 / 0.09 / 0.2 / 0.79 / 0.48.

• Visual question answering exposed the largest weaknesses, especially for List answers. DeepVitminC ranked first with a weighted score of 0.31, then Ricoh_SRCB (0.27), DocMiner (0.27), Vassilis Sioros (0.26), and VLMinators (0.25). DeepVitminC's Yes/No F1 was 0.92 and its Paragraph BERT-F1 was 0.85, but its Factoid Exact Match was 0.09 and its List set-based F1 was 0.09. All top-5 teams had List set-based F1 between 0.05 and 0.09. The Molmo2 8b baseline scored a weighted 0.2.

• No single baseline model performed well across all tasks. The paper evaluated Gemma 4 E4B 8b, Qwen3-VL-8B-Instruct, GLM-4.6V-Flash, and Intern VL 3.5 8b, along with task-specific baselines Qwen 3 VL 8b and Molmo2 8b, and reports that no single baseline performed well across all tasks.

• Winning methods converged on a common pattern. Top-performing approaches predominantly used Qwen-based backbones, with Qwen2.5-VL-7B-Instruct and Qwen3.5-9B appearing most often, and combined them with pipelines that integrate context, structure, and task-specific constraints. Ricoh_SRCB placed 1st in Task 1, 3rd in Task 2, 2nd in Task 3, and 2nd in Task 4; TeleOCR-VL placed 1st in Task 2 and 3rd in Task 3; DeepVitminC placed 1st in both Task 3 and Task 4.

• Key techniques across teams included hierarchical decomposition, context injection, preference alignment, and ensembling. Examples reported include Ricoh_SRCB's coarse-grained 6-class to refined 49-class hierarchical fine-tuning with dual global/cropped image inputs, DocMiner's multi-agent consensus inference, VLMinators' QLoRA fine-tuning with selective label hints, TeleOCR-VL's structure-aware training and synthetic data generation, and Ricoh_SRCB's DPO with hard negative samples. The paper also notes that ensemble methods across heterogeneous architectures proved ineffective for VLMinators due to high inter-model disagreement.

Methodology in Plain English

The authors first gathered a corpus of 205 ALD/E research publications covering experimental and simulation-based studies. They used MinerU to extract textual content as structured JSON alongside high-resolution figures in JPEG format, preserving both semantic structure and visual fidelity. They then built a 49-figure taxonomy and used it to label figures at the subfigure level.

Because existing annotation tools could not support the domain-specific end-to-end workflow, they developed a custom web-based, multi-user annotation platform with bounding-box coordinates and a unified JSON schema. Qwen2.5-VL-7B-Instruct was used for automated pre-annotation of classification, data extraction, and summarization, while VQA was annotated entirely by hand to ensure grounded reasoning quality. Ten domain experts were screened and recruited, each assigned 25 papers, with 5 papers shared across annotators to compute inter-annotator agreement. Automated scripts ran bi-weekly validations, and inconsistencies were reported back to annotators for alignment with the guidelines.

The competition was run in two phases: a development phase starting 9 January 2026 with progressive batch releases of training data, and an evaluation phase from 16 March 2026 to 8 April 2026 on a blind test set released 6 March 2026. Submissions were managed through the CodaBench platform with separate tracks per task and automated evaluation scripts. Four tasks were scored with distinct metric combinations: Accuracy/Precision/Recall/F1 for classification (F1 for ranking), a half-RMS plus half-TEDS weighted score for data extraction, a half-average-ROUGE plus half-BERTScore-F1 score for summarization, and an equal quarter-weight combination of Paragraph, Factoid, List, and Yes/No scores for VQA.

Why This Matters

This work provides the first benchmark dataset and competition dedicated to scientific comprehension and reasoning over ALD/E figures, addressing a documented gap: prior chart datasets target general real-world or synthetic data, and prior material science datasets are limited in scope, range of quantitative figures, and task formulations. It shifts evaluation from natural-image performance toward authentic, domain-grounded scientific figures, and it shows quantitatively where current multimodal systems break down, notably in extracting chart data and in producing List-type answers.

Real-world applications:

• Semiconductor process development: ALD and ALE are described as foundational technologies in materials science for advanced electronic and functional materials and next-generation nanoelectronics at the atomic scale, so extracting quantitative data from published process figures supports process comparison and replication.

• Materials science literature mining: automatically converting charts and spectra in publications into structured Markdown tables and summaries lets researchers aggregate results at scale across the 205-paper corpus and beyond.

• Domain expert assistance: the expert-annotated VQA set (four question-answer pairs per figure, labeled at subfigure level, with question types covering Process-Oriented, Comparative/Trend, Structure-Property, and Application/Performance) supports building question-answering tools that reason about chemical composition and material properties.

• Benchmarking and model selection: the leaderboards and baselines give practitioners a way to compare vision-language models and fine-tuning strategies on a complex, domain-specific task rather than on general chart benchmarks.

Industry relevance: the competition attracted participation from both academia and industry, and industrial teams appear on the leaderboards, indicating direct commercial interest in automated extraction from specialized scientific documents. The dataset and evaluation scripts are released under a CC BY 4.0 license, and winning teams were invited to publish system papers through TIB Open Conference Publishing free of charge.

Future Directions

• Improving data extraction and VQA. The paper states that significant challenges remain in data extraction and visual question-answering, which require robust multimodal reasoning and precise scientific understanding, and that these are the weakest tasks in the results.

• Building unified frameworks rather than isolated improvements. The comparative analysis concludes that future improvements will likely arise from unified frameworks that jointly optimize data engineering, prompt design, model adaptation, and inference-time optimization, instead of relying on isolated advancements.

• Addressing answer-type-specific failures. Factoid Exact Match and List set-based F1 scores were very low across all top teams, raising the question of what methods could improve precise numerical and set-formatted answers.

• Expanding domain-specific datasets and reasoning capabilities. The paper frames the lack of large-scale, expert-curated datasets that support end-to-end multimodal reasoning on authentic, domain-grounded scientific figures as a critical gap, and asks via RQ1, RQ2, and RQ3 how well current multimodal models reason over such figures, which approaches significantly improve their capabilities, and which tasks remain most challenging.

Target Audience

Researchers and practitioners in multimodal AI and document analysis who work on scientific figure understanding, chart-to-table extraction, and domain-grounded visual question answering; materials science and semiconductor process researchers interested in automated literature mining; benchmark and competition organizers in computer vision and document analysis; and industry teams evaluating vision-language models and fine-tuning strategies such as LoRA, QLoRA, DPO, and GRPO for specialized scientific content.

Authors’ abstract

Scientific figure comprehension and reasoning using multimodal AI requires integrating visual perception with domain-specific reasoning to extract meaningful knowledge, often not presented in the text of a research publication. The Sci-ImageMiner benchmark dataset, accompanied by a community-driven competition, raises the bar over prior scientific competitions by curating a comprehensive, expert-annotated dataset across four end-to-end complementary tasks. The competition attracted 68 active participants and 1,263 public/private submissions from 9th January 2026 to 8th April 2026. Our results show that state-of-the-art multimodal models perform well on classification and summarization tasks but struggle with data extraction and scientific reasoning, particularly in visual question-answering. These findings reveal key limitations and highlight challenges and opportunities for improving domain-aware multimodal AI systems. Overall, the Sci-ImageMiner benchmark and competition establish a rigorous platform for advancing research in scientific figure comprehension and reasoning and demonstrate the potential of state-of-the-art approaches for a challenging and complex research area.