ChartComplete: A Taxonomy-based Inclusive Chart Dataset

AI summary

Overview

Research area: Chart understanding (ChartQA) and multimodal AI benchmarking, specifically the datasets used to evaluate multi-modal large language models on chart images.

Technical level: Beginner-Friendly. The paper is a dataset and resource description; the collection pipeline uses standard, well-documented tools (Google ViT, FAISS, T-SNE, CKA), and no new model or training algorithm is proposed.

Scope: The paper introduces ChartComplete, a taxonomy-derived dataset of classified chart images spanning thirty chart types, intended to broaden the chart types available for evaluating chart-understanding models.

What This Paper Is About

Existing chart-understanding benchmarks cover only a narrow set of chart types, mostly bar, pie, line, and scatter charts. That narrowness limits how well a benchmark can measure whether a model genuinely understands charts. The authors build ChartComplete, a dataset organized around a modified version of Borkin's visualization taxonomy, covering thirty chart types with fifty images each.

Key Contributions

1. A taxonomy-based chart dataset with thirty chart types. ChartComplete is built on a modified version of Borkin's taxonomy of chart types, spanning twelve categories: Area, Bar, Circle, Diagram, Distribution, Matrix, Line, Map, Point, Text, Tree, and Combination. Each chart type contains fifty images.

2. An expansion of the chart types covered by prior benchmarks. The paper reviews FigureSeer, FigureQA, PlotQA, ChartQA, Chart-to-text, OpenCQA, and ChartFC and shows they are limited to chart types such as bar plots, line plots, pie plots, scatter plots, area plots, flow charts, and graph plots, whereas ChartComplete adds rarer types such as parallel coordinates charts, box and whisker charts, and choropleth map charts.

3. A documented collection and quality-control protocol. The authors define collection guidelines covering image quality, content quality, complete information, absence of prior visual knowledge, complexity, and image format (JPEG or PNG), with a minimum allowed image dimension of 300 pixels. The pipeline is iterative, uses version control, and applies an author-side quality check before merging.

4. An open release of the dataset and code. The dataset is distributed under the CC BY license, and the self-contained source code including the chart collection is available on GitHub (AI-DSCHubAUB/ChartComplete-Dataset).

Main Findings

• Scale and composition: ChartComplete contains 30 chart types at 50 images each. Of the images, 951 (63.4%) were collected manually and 549 (36.6%) were scraped.
• Collection sources: Charts were scraped from Statista (12,635 images, from the chart-of-the-day page on 27 January 2025) and Our World in Data (4,113 images, from the "Browse By Topic" menu under "Charts"), with a sizable remaining portion collected by hand because some chart types are not available in large collections in one place.
• Selection procedure: Because the scraped images came without chart-type labels, the authors extracted features with a Google ViT (google/vit-base-patch16-224), built a FAISS index, sampled a random image per target type, retrieved its 100 nearest neighbors, and manually filtered correct matches, repeating until 50 images per type were reached.
• Collection-method distribution: 18 chart types were purely collected manually, 8 were purely scraped, and 4 were both scraped and manually collected.
• Taxonomy modifications: The authors added "stacked" variants for area, line, scatter, and bubble charts (renaming the simple-versus-complex distinction as "vanilla" versus "stacked"), removed less common types such as the Venn diagram, circular bar chart, and contour line chart, and added combination charts such as the bar and line chart.
• Visual feature space: A T-SNE projection of Google ViT features (Figure 1) is segmented, which the authors read as evidence of both diversity and clusterization, and of the model's ability to capture discriminative features. The authors note imperfections, such as some area charts clustering near circle charts and other area chart groups near diagram charts, suggesting a more specialized feature extraction model would help.
• Chart-type similarity: Centered Kernel Alignment (CKA) computed between all chart types (Figure 2) shows all chart types differ from one another feature-wise.
• Image size distribution: The distribution of image sizes (Figure 3) is not uniform, because image size and shape were not part of the collection guidelines beyond the 300-pixel minimum; in general the images are large in resolution.
• Quality filtering: The final pass removed duplicate images and images for which the answer to "Can we ask meaningful questions about the chart?" was "No."
• Stated limitation: ChartComplete is a collection of classified chart images and does not include a learning signal, such as supervision for summarization or question answering.

Methodology in Plain English

The authors started from an existing taxonomy of chart types used by the visualization community (Borkin's), adapted it, and used it as the target list of chart types to populate. They then assembled images in two ways. For chart types that appear abundantly online, they scraped large collections from Statista and Our World in Data — 12,635 and 4,113 images respectively — but those sources did not label the chart type of each image. To sort them, they ran every scraped image through a Google ViT to get a feature vector, stored those vectors in a FAISS index, and then, for each desired chart type, picked a random image, found its 100 nearest neighbors in feature space, and manually inspected those neighbors to keep the correct ones. This loop ran until each type had 50 images. For chart types that scraping could not supply in sufficient numbers, they ran a manual collection campaign with explicit written guidelines: images had to be at least 300 pixels on both dimensions, readable, glitch-free, complete (title, labeled axes with units, legends where needed), free of logos or symbols requiring prior knowledge, neither overly simple nor overly fancy, and in JPEG or PNG format. A first collection round produced low-quality images, so the authors formalized the guidelines and repeated the campaign. Collections were pushed to version control, quality-checked by the authors before merging, and any failing images were removed. A final pass removed duplicates and images that could not support meaningful questions. They then characterized the dataset visually: T-SNE over ViT features to inspect how chart types spread out in feature space, CKA to measure how similar chart types are to one another, and a plot of image size distribution.

Why This Matters

Benchmarks shape what models are optimized for. If every widely used chart benchmark contains only bar, line, pie, and scatter charts, then strong benchmark scores say little about whether a model can read a Sankey diagram, a waterfall chart, a heat map, or a choropleth map. ChartComplete broadens the set of chart types that the research community can draw on when building evaluation suites, and it points to the gap between what models are tested on and what people actually encounter in reports, dashboards, and publications. The dataset itself carries no training signal, so its immediate role is as raw material for classification and for building downstream question-answering and summarization benchmarks rather than as a finished evaluation set.

Real-world applications of more inclusive chart understanding:

• Business intelligence and reporting tools: charts in reports and dashboards span waterfall, treemap, heat map, and combination types that current benchmarks largely ignore.
• Document and figure understanding pipelines: scientific papers, news articles, and government reports embed choropleth maps, box and whisker plots, and parallel coordinates charts.
• Automated chart summarization and chart-to-text generation: systems that describe a figure in words need exposure to a wider range of visual encodings.
• Chart search, retrieval, and classification: organizing or searching large chart repositories requires recognizing many chart types, not just four.

Industry relevance: companies building multimodal assistants, accessibility tools, and document-processing products need models that generalize past the small set of chart types found in existing benchmarks. A taxonomy-organized dataset gives those teams a way to test coverage gaps directly. The paper also reports a practical engineering detail useful to industry: the ViT-plus-FAISS nearest-neighbor loop for sorting unlabeled scraped images into chart types.

Future Directions

• Adding a learning signal. The authors state that ChartComplete currently lacks a direct training signal such as those used for summarization or question-answering tasks, and that they plan to address this in future work while encouraging community contributions.
• Improving feature extraction for chart-type discrimination. The T-SNE space shows area charts overlapping with circle and diagram charts, which the authors attribute to the general-purpose ViT features and suggest would benefit from a more specialized feature extraction model.
• Extending coverage further. The authors removed chart types such as the Venn diagram, circular bar chart, and contour line chart from the taxonomy, leaving open whether those should eventually be reintroduced.
• Turning the dataset into benchmarks. Since the images are classified but unlabeled for tasks, a natural next step is building question-answering or classification benchmarks on top of the thirty chart types, particularly for the rare types that existing benchmarks omit.

Target Audience

Researchers and practitioners working on chart understanding, multimodal large language models, and visual question answering over figures will benefit most, especially those who build or critique ChartQA benchmarks. Dataset curators and annotation teams will find the collection guidelines and the ViT-plus-FAISS selection procedure directly reusable. Engineers building document understanding, business intelligence, or chart retrieval products will find the taxonomy useful as a coverage checklist, and students entering the chart-understanding area will find the comparison of prior benchmarks in Table 1 a compact orientation to the field.

Authors’ abstract

With advancements in deep learning (DL) and computer vision techniques, the field of chart understanding is evolving rapidly. In particular, multimodal large language models (MLLMs) are proving to be efficient and accurate in understanding charts. To accurately measure the performance of MLLMs, the research community has developed multiple datasets to serve as benchmarks. By examining these datasets, we found that they are all limited to a small set of chart types. To bridge this gap, we propose the ChartComplete dataset. The dataset is based on a chart taxonomy borrowed from the visualization community, and it covers thirty different chart types. The dataset is a collection of classified chart images and does not include a learning signal. We present the ChartComplete dataset as is to the community to build upon it.