Secu-Table: a Comprehensive security table dataset for evaluating semantic table interpretation systems

AI summary

Overview

Research area: Semantic Table Interpretation (STI) and cybersecurity knowledge representation, with a focus on benchmarking LLM-based table annotation systems.

Technical level: Intermediate. Familiarity with knowledge graphs, RDF/SPARQL, and standard STI tasks (CEA, CTA, CPA) helps, but the paper is largely about data construction rather than algorithms.

One-sentence scope: The paper introduces and releases Secu-Table, a publicly available security-domain tabular dataset built from CVE and CWE sources and annotated against Wikidata and SEPSES CSKG, designed to benchmark semantic table interpretation systems powered by large language models.

What This Paper Is About

Semantic Table Interpretation (STI) systems — particularly those built on large language models — cannot be meaningfully evaluated without high-quality, domain-specific datasets to test against. While general-purpose tabular datasets exist for STI benchmarks, no publicly available security-domain tabular dataset has been released for this purpose before. The paper fills this gap by constructing Secu-Table: over 1,500 tables of cybersecurity data extracted from CVE and CWE feeds, manually annotated against two knowledge graphs, and packaged for the SemTab@ISWC 2025 challenge.

Key Contributions

1. A new security-domain tabular dataset (Secu-Table). 1,554 tables built from Common Vulnerabilities and Exposures (CVE) and Common Weakness Enumeration (CWE) data, split into 76 ground-truth tables and 1,478 hidden test tables.
2. Multi-knowledge-graph annotation. Tables are annotated against both Wikidata (general-purpose, ~100M entities) and SEPSES CSKG (a cybersecurity-specific KG with ~3.8M entities), giving STI systems exposure to both broad and domain-specific linking challenges.
3. Controlled error injection for realistic benchmarking. The dataset deliberately mixes 20% clean data with 26% missing context, 26.6% misspelling errors, and 26.26% annotation errors to test LLM robustness under noisy real-world conditions.
4. Open release and baseline evaluation. Full dataset on Hugging Face and code on GitLab, plus a preliminary baseline using Falcon3-7b-instruct, Mistral-7B-Instruct, and GPT-4o mini.

Main Findings

• The dataset is large and multi-layered. Over 150,000 entities, 1 million rows, and 20,000 columns in the test set, with an average of 8.13 columns and 291.63 rows per table. The ground-truth subset alone holds more than 8,900 entities, 55,000 rows, and 1,000 columns.
• Annotation was performed manually. Because the dataset is intended for rigorous evaluation at SemTab@ISWC 2025, two annotators (with expert review) manually produced the ground truth rather than relying on automated pipelines.
• Wikidata yielded limited usable annotations. Many Wikidata entries produced ambiguity or empty fields, so this version only provides Cell Entity Annotation (CEA) for Wikidata; SEPSES CSKG covers all three tasks (CEA, CTA, CPA).
• Knowledge graph incompleteness leaves empty cells. Some cells remain unannotated due to gaps in the source KGs — the authors frame this as intentional realism rather than a defect.
• A SemTab challenge baseline is provided. Two open-source LLMs (Falcon3-7b-instruct, Mistral-7B-Instruct) and one closed-source LLM (GPT-4o mini) were evaluated using a two-shot prompting setup on the 76 ground-truth tables, with precision, recall, and F-score as metrics.

Methodology in Plain English

The authors first recruited data curators (master's students and a professor, all with semantic web backgrounds) and trained them on annotation tasks. Then they collected CVE and CWE data — originally in JSON, XML, and CSV — and converted it into flat CSV tables, where each column corresponds to a structured metadata field (like "CWE-ID", "Name", "Description").

To annotate, they loaded the SEPSES CSKG into a Jena triple store and ran SPARQL queries to find matching entities, column types, and column properties. For Wikidata, they searched manually because SPARQL queries returned too many irrelevant or ambiguous results. Annotations were cross-checked by a second annotator. Finally, they injected controlled errors (misspellings, missing context, wrong labels) using Pandas to simulate noisy real-world data, and split the tables into ground-truth and test sets for the SemTab challenge.

Why This Matters

Impact on research: This is the first publicly available tabular dataset tailored to cybersecurity STI evaluation. It gives researchers a shared, reproducible benchmark for comparing LLM-based annotation methods in a domain where data has historically been scattered across incompatible formats.

Real-world applications:

• Vulnerability management: Automatically linking CVE records to standardized weakness types (CWE) and external knowledge bases speeds up triage and prioritization.
• Threat intelligence integration: Mapping disparate security feeds (ATT&CK, OWASP, SCAP) into unified knowledge graphs so analysts get a holistic view.
• Automated security report enrichment: Annotating internal vulnerability tables against public KGs helps security tools reason about risk context.
• Training data for security LLMs: Realistic noisy annotations help fine-tune models to survive misspellings and incomplete context.

Industry relevance: Security operations teams and SIEM vendors constantly struggle with fragmented formats; a standardized KG-linked benchmark is a step toward interoperable tooling and LLM-assisted security analysis.

Future Directions

1. Expanding source coverage. Quarterly releases starting in January are planned to add ATT&CK, CCE, CPE, CVSS, OWASP, and SCAP data, plus additional knowledge graphs such as DBpedia.
2. Semi-automatic annotation. Because manual annotation is slow and expensive, the authors plan to combine LLMs (e.g., Falcon3-7b-instruct) with human-in-the-loop verification to scale up curation.
3. Fixing the Wikidata CTA/CPA gap. Only CEA was produced for Wikidata due to sparse annotations — future versions aim to close this so all three STI tasks are supported against both KGs.
4. Stronger LLM baselines. The current evaluation is preliminary; more systematic studies across prompting strategies, model sizes, and robustness-to-noise conditions are needed.

Target Audience

Researchers and practitioners working on semantic table interpretation, knowledge graph construction, semantic web technologies, and LLM evaluation — especially those focused on cybersecurity applications. Data scientists and engineers building security knowledge graphs or threat intelligence platforms will also find the dataset directly useful as a training and benchmarking resource.

Authors’ abstract

Evaluating semantic tables interpretation (STI) systems, (particularly, those based on Large Language Models- LLMs) especially in domain-specific contexts such as the security domain, depends heavily on the dataset. However, in the security domain, tabular datasets for state-of-the-art are not publicly available. In this paper, we introduce Secu-Table dataset, composed of more than 1500 tables with more than 15k entities constructed using security data extracted from Common Vulnerabilities and Exposures (CVE) and Common Weakness Enumeration (CWE) data sources and annotated using Wikidata and the SEmantic Processing of Security Event Streams CyberSecurity Knowledge Graph (SEPSES CSKG). Along with the dataset, all the code is publicly released. This dataset is made available to the research community in the context of the SemTab challenge on Tabular to Knowledge Graph Matching. This challenge aims to evaluate the performance of several STI based on open source LLMs. Preliminary evaluation, serving as baseline, was conducted using Falcon3-7b-instruct and Mistral-7B-Instruct, two open source LLMs and GPT-4o mini one closed source LLM.