Enginuity: Building an Open Multi-Domain Dataset of Complex Engineering Diagrams

AI summary

Overview

Research area: Computer vision and multimodal AI for scientific discovery, specifically the parsing and understanding of engineering diagrams.

Technical level: Beginner-Friendly. The paper is a dataset proposal (a position/roadmap paper) rather than an empirical study; it contains no model architectures, no equations, and no experimental results.

Scope: This paper proposes "Enginuity," a planned open, large-scale, multi-domain dataset of complex engineering diagrams with structural annotations, beginning with the automotive domain, intended to support automated diagram parsing, relationship extraction, and diagram question answering.

What This Paper Is About

Engineering diagrams hold essential technical knowledge, but most remain locked inside proprietary archives, so AI systems cannot learn from them. Existing datasets are either small and narrow (for example, limited to electrical circuits or isolated symbol classification) or restricted, and no public dataset reportedly exists with more than 10K real-world engineering diagrams that include both component annotations and structural relationship annotations. The authors propose Enginuity, a dataset of 50K annotated engineering diagrams built with an automotive AI industry partner, along with a four-stage annotation pipeline, a defined task suite, evaluation metrics, and plans for a CVPR 2026 workshop, shared task, and leaderboard.

Key Contributions

1. A proposed open, multi-domain engineering diagram dataset. Enginuity is described as the first open, large-scale, multi-domain engineering diagram dataset with comprehensive structural annotations, targeting 50K+ diagrams across powertrains, chassis, and body components from 500+ vehicle models, standardized to ISO/IEEE ontologies.

2. A two-pronged data collection strategy for openness plus realism. The plan combines annotation of public-domain automotive diagrams (declassified government vehicles and older public-domain vehicles, plus associated technical procedure manuals) with an industry engagement framework in which private contributors such as OEMs can contribute "older vehicle parts diagrams" (5-15 years old) without disclosing proprietary information.

3. A four-stage human-in-the-loop annotation pipeline. Stage 1 uses domain-specialized LLMs and vector embeddings for AI-driven preprocessing of PDF, DXF, SVG, and raster scans; Stage 2 uses a dedicated annotation team for bounding box adjustments, OCR verification, and component alignment; Stage 3 uses domain experts to build a "golden set"; and Stage 4 is an active learning loop that reuses validated annotations to auto-label later batches.

4. A defined task suite, metrics, and community infrastructure. The paper specifies tasks from component and symbol recognition through relationship extraction, functional context interpretation, diagram question answering, and diagram-to-digital-twin alignment, proposes mAP and a graph accuracy metric as baselines, and plans open release on Kaggle with baseline models in PyTorch, leaderboards, tutorial notebooks, and community support channels.

Main Findings

• Relationship extraction is the stated bottleneck. The paper reports that current methods achieve 85%+ accuracy on symbol detection but that performance drops by 25%+ when moving to relationship extraction (attributed to Stürmer et al., 2025). This gap is presented as the core motivation for the dataset.

• No sufficiently large annotated public dataset is reported to exist. According to the paper, there is no public dataset with more than 10K real-world engineering diagrams containing both component and structural relationship annotations.

• Existing datasets are described as too narrow. The paper contrasts Enginuity with prior resources such as handwritten engineering diagrams (circuits) (Bayer, 2025), which it characterizes as focused exclusively on small-scale electrical circuits, and SiED: Symbol Classification Engineering Diagrams (Elyan et al., 2020), which it says provides isolated symbol classification without higher-order structure, system-level relationships, or domain generalization.

• Automotive diagrams are the chosen starting domain. The rationale is that exploded parts diagrams used by technicians tightly couple visual structure, text, and functional knowledge; technicians navigate them with natural-language queries such as "front-left brake caliper." The industry collaborator, Predii, is described as processing 2B+ repair jobs monthly.

• Planned annotation scope and cost reduction. Annotations are planned to include hierarchical component relationships, spatial connections, part numbers, functional roles, part-number relationships, hierarchy tagging of components, sub-components, assemblies and one-time-use parts, specifications and usage types, and system-subsystem clustering. The pipeline is stated to integrate active learning to reduce costs by 65%, with expert validation on 10% of samples in the main text and a domain expert team reviewing 5-10% of annotations for complex components in Appendix A.4.

• Planned partitioning includes a withheld test set with domain shift. The corpus is to be split four ways: training, validation, testing, and a withheld testing set reserved for the organized competition. The withheld set will additionally include diagrams from private organizations outside the automotive domain to introduce domain shift.

• Planned metrics. Mean Average Precision (mAP) at standard IoU thresholds is proposed for symbol and component recognition, and a graph accuracy metric, computed as the proportion of correctly predicted edges and node labels compared to ground-truth graphs, is proposed for relationship extraction and diagram graph construction.

• Planned budget and timeline. The estimated budget is $150K for data collection and annotations, $30K for infrastructure support, and $20K for baselines and evaluation, approximately $200K USD total. By the end of month 12 the authors expect to have 50K annotated images available for the research community.

• No empirical results are reported. The paper presents no experiments, no trained models, no measured baselines, and no dataset statistics from an actual released corpus. All quantitative claims about Enginuity itself are targets or plans.

Methodology in Plain English

The authors do not run experiments. Instead they lay out a plan for building a dataset and a benchmark.

For sourcing, they take two routes. First, they collect and annotate diagrams that are already public: diagrams from declassified government vehicles and older vehicles, including exploded parts diagrams and the technical procedure manuals that reference them. Second, they create a framework through their industry collaborator so that companies such as OEMs can contribute diagrams of older vehicles (5-15 years old) without revealing proprietary information, since the age of the material lowers the sensitivity.

For labeling, they describe a tiered pipeline. An AI system does a first pass, detecting lines, arrows, text regions, and component clusters and normalizing inconsistent file formats (PDF, DXF, SVG, raster scans) into a uniform digital representation. A dedicated annotation team then cleans up simpler items such as bounding boxes, OCR text, and component alignment. Domain experts such as technicians and engineers build a "golden set" and validate a portion of the annotations. Finally, an active learning loop uses the validated data to train models that automatically label the next batches, with human review concentrated on uncertain or novel cases. Labels are mapped to or extend existing engineering ontologies (IEEE, ISO, ISA) so the data can be reused across domains.

The annotated corpus is divided into training, validation, testing, and a withheld competition test set. Baselines will be implemented in PyTorch, and the dataset will be released under an open license on Kaggle with a datacard, leaderboards, tutorial notebooks, and community channels. The paper notes that the release location is described as Kaggle in Appendix A.9 and as Hugging Face in Appendix A.11.

Why This Matters

Impact on research. The paper argues that diagram understanding is a long-studied, difficult computer vision problem that requires progress in perception, structured representation, and multimodal reasoning, and that no single group can solve it alone. A shared benchmark and leaderboard is presented as the way to make sustained community progress. The authors frame the missing open, expert-labeled, cross-disciplinary dataset as a major bottleneck that limits how well AI can generalize, generate, or link diagrams to digital twins.

Real-world applications (as described by the authors):

• Design optimization — tools that automatically suggest structural improvements or highlight inefficiencies in complex assemblies.
• Simulation integration — converting diagrams into machine-readable representations suitable for digital twin simulations, improving prediction accuracy and reducing experimental overhead.
• Automated documentation — generating consistent metadata and structured component lists from legacy diagrams for interoperability across industry and research archives.
• Knowledge preservation and harmonization — bridging generational gaps in notation, style, and conventions across decades of archived diagrams.

Industry relevance. The project is built around an industry partnership. Predii, an automotive AI company and a coauthor, supplies domain knowledge and data, and the engagement framework is designed to give OEMs a way to participate in research without exposing proprietary material. Practical relevance is anchored in real repair workflows, where technicians already depend on exploded parts diagrams and natural-language queries to identify parts. The authors also link the effort to Industry 4.0 transformation.

Future Directions

1. Expansion beyond automotive. Enginuity v1.0 is centered on automotive diagrams, but the authors state the dataset will later be expanded to other domains such as mechanical and process engineering, supported by versioning protocols that keep benchmark splits stable while allowing incremental additions.

2. A shared task, workshop, and leaderboard. The initiative is planned to culminate in a CVPR 2026 workshop and shared task, with a leaderboard arena modeled on LMSYS where frontier AI models would be tested on the benchmark.

3. Refining evaluation protocols for higher-level tasks. The mAP and graph accuracy metrics are described as initial baselines; the authors intend to refine and extend them with the research community, particularly for diagram question answering and diagram-to-digital-twin alignment, where specialized evaluation protocols may be needed.

4. Building a sustainable academic-industry infrastructure. Open questions include how to fund and maintain the resource long term, how to keep growing it with new contributors without inheriting the variability of industrial documentation practices, and whether further industry support can be secured to launch the arena.

Target Audience

This paper is most useful to researchers and practitioners who need engineering diagram data: computer vision and document AI researchers working on object detection, graph extraction, or diagram parsing; multimodal LLM researchers interested in spatial and structural reasoning benchmarks; AI-for-science and digital twin researchers; and engineering informatics groups. It is also relevant to automotive OEMs, technical documentation teams, and dataset curators evaluating how to release sensitive industrial material responsibly, and to workshop or shared-task organizers planning similar benchmark initiatives. Readers looking for experimental results or a completed dataset will not find them here, since the paper describes a proposed resource rather than a finished one.

Authors’ abstract

We propose Enginuity - the first open, large-scale, multi-domain engineering diagram dataset with comprehensive structural annotations designed for automated diagram parsing. By capturing hierarchical component relationships, connections, and semantic elements across diverse engineering domains, our proposed dataset would enable multimodal large language models to address critical downstream tasks including structured diagram parsing, cross-modal information retrieval, and AI-assisted engineering simulation. Enginuity would be transformative for AI for Scientific Discovery by enabling artificial intelligence systems to comprehend and manipulate the visual-structural knowledge embedded in engineering diagrams, breaking down a fundamental barrier that currently prevents AI from fully participating in scientific workflows where diagram interpretation, technical drawing analysis, and visual reasoning are essential for hypothesis generation, experimental design, and discovery.