ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams

AI summary

Overview

Research area: Artificial intelligence and multimodal machine learning for database engineering — specifically, evaluating Vision–Language Models (VLMs) on Entity–Relationship Diagrams (ERDs), with connections to conceptual database design, schema recovery, and multimodal document understanding.

Technical level: Intermediate. Readers benefit from familiarity with database schemas (entities, attributes, keys, relationships) and with standard evaluation metrics (F1, BLEU), but the paper is written so that the core message — VLMs read ERD text well but fail to reconstruct ERD structure — is accessible without deep expertise.

Scope in one sentence: The paper introduces ERUnderstand, a benchmark of 2,960 ER diagrams paired with machine-readable JSON ground truth, and uses it to systematically measure how ten state-of-the-art VLMs perform on structured schema reconstruction.

Venue and availability: The paper is published under CC BY 4.0 and is associated with the Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2026. The benchmark, dataset, evaluation toolkit, and generation code are released at https://github.com/salinaria/ERUnderstand. All six authors are affiliated with the Department of Computer and Information Sciences at Temple University, Philadelphia, PA, USA.

What This Paper Is About

ERDs are the standard abstraction for conceptual database design, but in practice they exist almost exclusively as rendered images in textbooks, slides, PDFs, and screenshots rather than as machine-readable schemas. This creates a bottleneck for AI-assisted database engineering: AI systems can process symbolic schemas deterministically, but cannot directly interpret the visual artifacts that humans actually circulate.

The paper's goal is to build the first large-scale benchmark that evaluates whether modern Vision–Language Models can recover executable, machine-readable schemas from ERD images. This requires going beyond reading the text labels in a diagram and instead reasoning jointly about graphical syntax, notation conventions, spatial connectivity, and structural topology.

Key Contributions

1. The ERUnderstand benchmark. A collection of 2,960 ER diagrams drawn from three complementary sources — educational/curated ERDs, production database schemas, and synthetically generated ERDs — spanning diverse domains, multiple ER notations (including Chen and Silberschatz-style), varying complexity levels, and Extended Entity–Relationship (EER) constructs.

2. A standardized evaluation framework. Each diagram is paired with a unified JSON representation encoding entities, attributes, relationships, cardinalities, primary keys, weak entities, identifying relationships, inheritance hierarchies, and other EER constructs. The paper provides structure-aware metrics, an open-source evaluation toolkit, and reproducible protocols. Annotation reliability was checked via a double-annotation study on 30 curated ERDs (15 web-sourced, 15 textbook diagrams), yielding an average agreement score of 0.897 on the structure-aware Macro-F1 metric; exactly 21 diagrams required joint review, mainly due to ambiguous notation or low-resolution images.

3. The first systematic benchmarking of VLMs on conceptual database diagram understanding. Ten state-of-the-art VLMs are evaluated under identical zero-temperature settings, establishing baselines for future work.

4. A failure-mode analysis of multimodal structural reasoning. Through controlled experiments, the authors identify recurring weaknesses — spatial proximity bias, reliance on linguistic priors, and degradation under increasing structural complexity — that appear consistently across both open- and closed-source model families.

Main Findings

• Basic elements are recovered reliably, but EER constructs fail. Entities and regular attributes reach F1 around 0.89–0.93 on curated ERDs and 0.94–1.00 on synthetic ERDs. The abstract states that common ERD elements are recovered with F1 above 0.74, but performance drops sharply on weak entities (as low as 0.28 F1), multivalued attributes (0.14 F1), and n-ary relationships (0.07 F1).

• Reasoning models help but do not solve the problem. Reasoning-augmented models improve overall Macro-F1 by 15–25% relative to non-reasoning models. On Table 3, curated Macro-F1 is 0.59 for non-reasoning models versus 0.68 for reasoning models; on synthetic ERDs it is 0.59 versus 0.80.

• A gap between semantic extraction and structural consistency. On synthetic ERDs, models reach BLEU between 0.69 and 0.81 and Macro-F1 between 0.52 and 0.73. On curated ERDs, BLEU drops to 0.42–0.64 while Macro-F1 stays comparatively stable at 0.57–0.64. The asymmetry suggests text extraction succeeds more than structural reconstruction.

• The production-schema tier performs better on aggregate but still degrades with size. The Schema tier averages 46.4 structural elements per diagram and reaches Macro-F1 of 0.72–0.75, above the Web and Instructional tiers, because it contains zero weak entities, weak relationships, n-ary relationships, IS-A hierarchies, or derived/composite/multivalued attributes by construction. Within the tier, the Pearson correlation between element count and Macro-F1 is −0.66, and GPT-5.4-pro drops from 0.911 on the 10-element movielens schema to 0.226 on the 579-element adventureworks schema.

• Spatial bias rather than visual fidelity explains the curated–synthetic gap. Curated diagrams underperform synthetic ones despite higher visual quality (Macro-F1 0.59 vs. 0.67; BLEU 0.35–0.41 vs. 0.40–0.48). Regenerating curated ERDs from the same JSON with Graphviz in three notational styles changed results only narrowly: Web ERDs varied between 0.62 and 0.66 Macro-F1, and Instructional ERDs between 0.60 and 0.67. Component-level performance stayed stable, with entity extraction up to 0.97, relationship detection up to 0.85, and attribute recognition up to 0.94 regardless of notation.

• Models rely on linguistic priors over diagrammatic structure. Across 10 thematic categories spanning 20 domains, Macro-F1 varied minimally: Easy tier ranged from 0.77 (Legal) to 0.83 (Education), averaging 0.80, with Medium at 0.72 and Hard at 0.67. In controlled perturbations on 200 medium-complexity ERDs, replacing labels with random 4-letter strings (Context-Free) produced drops such as −0.08 for GPT-5.4 and −0.20 for Claude-4.6 among reasoning models, versus −0.35 for Claude-4.5 and −0.65 for Gemini-3.0-flash among non-reasoning models. Label permutation produced drops ranging from −0.27 (Gemini-3.0) to −0.49 (Claude-4.6), with Qwen-3.5 at −0.51.

• Spatial proximity bias drives relationship errors. Models achieve roughly 0.90 F1 on entities but drop about 15% on relationships for curated data and 10% for synthetic data. Error analysis shows hallucinated relationships between spatially proximate entities and missed relationships between distant but correctly connected entities. An attention-map analysis of Qwen shows the model failing to attend to a relevant entity and connecting edge; a distance analysis confirms that hallucinated relationships (EXTRA) occur predominantly between nearby entities while missed relationships (MISSED) are more likely between distant ones. The illustrative example: Qwen hallucinates "Train has Seat Availability" while ignoring the true connection "Zone has Station."

• Hierarchy tracing collapses without semantic cues. On an IS-A-only subset of 32 diagrams containing only superclass–subclass links, GPT-4o drops from 0.96 to 0.12 and Gemini-2.5 from 0.94 to 0.07, while entity recognition remains high. Table 3 shows the notation-independent version of this: curated IS-A F1 is 0.09 for non-reasoning and 0.28 for reasoning models, versus 0.74 and 0.91 on synthetic ERDs.

• Primary key identification follows naming conventions, not visual cues. Altering naming conventions increased missed PKs from 40.75 to 105.50. Removing visual cues led to widespread hallucination (up to 479.62 extra PKs). Misleading naming amplified errors further (979.88 missed, 799.25 extra).

• Complexity causes sharp degradation with sharp model differences. On 19 diagrams with 20+ entities and relationships, Macro-F1 dropped to 0.035–0.048 for most models with relationship scores as low as 0.009, while Gemini-2.0 reached 0.428 Macro-F1 and Gemini-2.5 reached 0.695. Across five element-count buckets on curated ERDs, Macro-F1 fell monotonically from 0.80 to 0.67 for Chen notation and from 0.82 to 0.72 for Silberschatz — relative declines of 16.8% and 13.9%. On the Schema tier, GPT-5.4-pro's Macro-F1 fell from 0.918 (≤30 elements) to 0.742 (31–60) to 0.364 (61–100) to 0.240 (101+).

• OCR is not the bottleneck. All models achieved high label recognition accuracy (average 95.5%, maximum 99.6%, minimum above 87%).

• GED corroborates the F1 trends. GPT-5 achieves normalized GED of 0.074, 0.119, and 0.157 on Easy, Medium, and Hard ERDs; Grok-2-Vision degrades from 0.363 to 0.727 to 0.891 across the same settings. Several models exceed normalized GED above 1.0 on IS-A-only diagrams. On the Schema tier, GED correlates with Macro-F1 at Pearson r = −0.98 and with element count at r = +0.63; GPT-5.4-pro's mean normalized GED rises from 0.173 (≤30 elements) to 0.862 (101+). The paper treats GED as complementary rather than primary, because it offers limited additional insight beyond Fractional F1 and is sensitive to local mismatches in dense ERDs.

• Interactive prompting helps with misses, not with reasoning failures. Across 30 diagrams spanning difficulty levels and three models (Grok, Gemini, Claude), Grok reduced missed attributes by 25.9% and missed relationships by 57.1%, while Claude achieved the largest overall error reduction at 39.9%, including an 84.2% decrease in hallucinated primary keys. Errors tied to structural reasoning — incorrect connections and misinterpretation of visual notation such as multivalued attributes — persisted after interaction, and additional dialogue sometimes introduced new errors.

• Open-source models lag, with an exception on the Schema tier. Open-source models consistently underperformed closed-source models with larger gaps on complex ERDs. On the Schema tier, GLM-4.5 (open-source) landed within 0.002 Macro-F1 of GPT-5.4-pro, and Llama-3.2 outperformed several closed-source models, suggesting the gap narrows when EER constructs are absent. Repeated evaluations across model versions showed minimal variation, with Macro-F1 differences under 4%.

Methodology in Plain English

The researchers assembled the benchmark from three sources designed to complement one another:

• Curated ERDs. They collected 182 ER diagrams from publicly available educational resources including textbooks, course materials, and online repositories. After removing duplicates and manually filtering low-quality examples, 148 unique high-quality conceptual ERDs remained. To respect copyright, they released source references and metadata rather than the original images. Rendering each underlying schema in multiple graphical styles, including original source representations and standardized Graphviz renderings, produced 740 diagram instances covering notations such as Chen and Silberschatz-style variants.

• Real database schemas. They automatically extracted 179 production schemas from widely used database benchmarks and open-source systems — Spider, BIRD, Northwind, AdventureWorks, Sakila, Chinook, Pagila, Employees, and ClassicModels — plus representative property-graph schemas from domains such as recommendation systems, social networks, fraud detection, and knowledge graphs. All were converted to the unified JSON representation and rendered in a single standardized Graphviz style.

• Synthetic ERDs. They built a four-stage generation pipeline: generate domain-specific database scenarios, convert scenarios into structured JSON schemas, render ER diagrams using Graphviz, and assign difficulty levels based on structural complexity and EER characteristics. The synthetic collection systematically varies schema size, relationship arity, hierarchy depth, spatial layout, and advanced constructs. Easy and medium subsets follow the scale and complexity distribution of common educational ERDs, while hard subsets introduce substantially larger schemas with richer EER structures. Fixed random seeds make the released benchmark reproducible.

Annotation. Every diagram was converted into a unified JSON representation. Two annotators independently converted a sample of 30 curated ERDs using identical guidelines, and agreement was measured with the same Macro-F1 metric used throughout evaluation.

Evaluation. The primary metric is a structure-aware fractional F1-score that grants partial credit, assigning 0.5 weight to structural errors (incorrect relationships, cardinalities, or attribute assignments) as both false positives and false negatives. Modeling choices aside, the design is straightforward in intent: greedy matching aligns predicted elements to ground truth by label similarity before structural properties are checked, and F1 is computed separately for entities, relationships, attributes, weak entities/relationships, and primary keys. Relationship scoring explicitly accounts for missed, extra, misconnected, and incorrect predictions. Invalid or unparsable JSON outputs are scored as empty predictions, with valid-JSON coverage tracked separately. BLEU provides a sequence-level complement, computed after deterministically sorting entities, relationships, and attributes to reduce ordering sensitivity.

Controlled probes. Rather than only reporting aggregate scores, the authors ran targeted experiments: regenerating curated diagrams in different notations to isolate visual fidelity, testing across 20 semantic domains to isolate domain familiarity, constructing an IS-A-only subset of 32 diagrams to isolate hierarchy tracing, and applying two label-perturbation strategies — Context-Free (replacing all labels with random 4-letter strings) and Label Permutation (shuffling labels across entity, attribute, and relationship roles while preserving vocabulary) — on 200 medium-complexity ERDs.

Models evaluated. Ten state-of-the-art VLMs under identical zero-temperature settings, separated into reasoning models that employ explicit reasoning or reinforcement-based inference and non-reasoning models using standard single-pass decoding. Commercial systems include ChatGPT-5.4, Gemini-3.0, Claude-4.6-Sonnet, and Gro

Authors’ abstract

Entity-Relationship Diagrams (ERDs) are central to conceptual database design, yet they are typically available only as rendered images rather than machine-readable schemas, limiting AI-assisted database engineering. We introduce ERUnderstand, the first large-scale benchmark for structured understanding of ER diagrams, comprising 2,960 diagrams collected from curated educational sources, real-world schemas, and synthetically generated examples spanning diverse domains, notations, complexity levels, and Extended Entity-Relationship (EER) constructs. Each diagram is paired with a standardized machine-readable representation for fine-grained evaluation of schema elements. Evaluating state-of-the-art Vision-Language Models (VLMs), we find that while common ERD elements are recovered reliably (F1 > 0.74), performance drops sharply on weak entities (as low as 0.28 F1), multivalued attributes (0.14 F1), and N-ary relationships (0.07 F1). Reasoning-augmented models improve overall performance by 15-25% but remain sensitive to linguistic priors and increasing diagram complexity. ERUnderstand provides a standardized benchmark for evaluating multimodal understanding of conceptual database schemas. The benchmark, dataset, evaluation toolkit, and generation code are publicly available at https://github.com/salinaria/ERUnderstand.