Ontology Learning with LLMs: A Benchmark Study on Axiom Identification

AI summary

Overview

• Research area: Ontology learning and knowledge representation, specifically the use of Large Language Models (LLMs) for automatically identifying axioms in formal ontologies.
• Technical level: Intermediate. Readers need basic familiarity with RDF/OWL ontology components (classes, properties, axioms) and with LLM prompting terminology (zero-, one-, few-shot), but no deep logic background is required.
• Scope: The paper introduces a nine-ontology benchmark called OntoAxiom and reports a systematic evaluation of twelve LLMs across two prompting strategies and three shot settings for identifying five types of axioms.

What This Paper Is About

Ontologies are structured domain models that make knowledge machine-readable, but building and maintaining them is slow, expert-heavy work. Automated "ontology learning" has advanced for low-complexity tasks such as extracting terms and concepts, while the hardest layer of the ontology learning layer cake, axioms, has received very little attention. This paper asks how well current LLMs can infer five specific axiom types from a list of an ontology's classes and properties, and whether the way the model is prompted changes the answer.

Key Contributions

1. A new benchmark (OntoAxiom): Nine medium-sized ontologies selected under explicit criteria (15 to 100 classes, at most 500 properties, at least 20 and at most 500 instances of one of the five target axioms, and topical diversity), together containing 17.118 triples and 2.771 axioms, released through an open-source repository.
2. A focused axiom taxonomy: Five commonly used RDFS/OWL axioms are targeted, covering both class axioms and property axioms: rdfs:subClassOf, owl:disjointWith, rdfs:subPropertyOf, rdfs:range, and rdfs:domain.
3. A controlled comparison of prompting strategies: Two strategies are defined and compared, a Direct approach that queries all axioms at once and an Axiom-by-Axiom (AbA) approach that queries one axiom type per prompt, each tested under zero-, one-, and five-shot settings.
4. A broad model sweep: Twelve instruction-tuned LLMs (open-source and proprietary, large and small, general-purpose and reasoning/coding-specialised) are evaluated, producing 3,240 results, plus a three-run robustness check on output consistency.

Main Findings

• Subclass axioms are easiest, domain and range are hardest: Mean F1 across all other factors is 0.359 for subclass (precision 0.399, recall 0.326), 0.106 for subproperty, 0.095 for disjoint, 0.038 for domain, and 0.030 for range. Precision exceeds recall for every axiom except domain.
• The ontology's domain strongly shapes performance: Mean F1 is highest for FOAF (0.221), then Pizza (0.172), gUFO (0.170), GoodRelations (0.119), Music (0.117), Time (0.101), NordStream (0.100), SAREF (0.065), and lowest for ERA (0.055).
• Best and worst per-domain results diverge sharply: The FOAF ontology reaches an F1 of 0.642 for the subclass axiom, while the music ontology reaches only 0.218. The paper attributes FOAF's advantage to its general domain and its frequent appearance in training data, and ERA's weakness to specialised concepts such as Siding and PhaseInfo with less public text.
• Axiom-by-Axiom prompting wins overall: The AbA approach reaches a mean F1 of 0.130 at one-shot, 0.126 at few-shot, and 0.124 at zero-shot, versus 0.126 (few-shot), 0.122 (one-shot), and 0.122 (zero-shot) for the Direct approach. AbA is better for subclass, domain, and range; Direct is better for disjoint; performance is nearly equal for subproperty.
• Shot settings matter only modestly: Few-shot leads for the Direct approach and one-shot leads for AbA, but the spread across shot settings is small, which the authors read as evidence that task difficulty, not prompt design, is the binding constraint.
• Proprietary and larger models lead: Category averages show closed models at F1 0.124 versus open models at 0.077, large models at 0.118 versus small at 0.077, and reasoning/coding models at 0.110 versus general-purpose at 0.101 (a difference of 0.009).
• o1 is the top scorer in Table 9: F1 0.197 with precision 0.249; o4-mini follows at F1 0.190 (precision 0.206, recall 0.176), and GPT-4.1 has the highest recall at 0.185 with F1 0.176. The lowest scorers are GPT-4o-mini (0.065), Qwen2.5-Coder (0.068), DeepSeek R1 (0.083), and Qwen3 (0.087). The body text reports F1 values of 0.164 for o1 and 0.197 for o4-mini, which differ from the values in Table 9.
• Best open-source models are competitive: Llama 3.3 reaches F1 0.121 and Mistral 3.1 Small 0.107, occasionally outperforming proprietary alternatives.
• Results are reasonably stable across runs: Precision is 0.139 in run 1 and run 2 and 0.145 in run 3; recall 0.113, 0.114, 0.118; F1 0.125, 0.125, 0.130.
• Overall performance does not support full automation: The authors conclude LLMs are better used as candidate-axiom generators that a human ontology engineer reviews.

Methodology in Plain English

The researchers first defined what counts as an axiom and settled on five widely used ones, each described in both description logic and plain language. They then gathered nine existing ontologies that met size, axiom-count, and topic-diversity criteria: ERA (railways, 7021 triples), FOAF (persons, 631), GoodRelations (products, 1834), gUFO (foundational knowledge, 766), the Music Ontology (2141), the Nord Stream Pipeline ontology (362), Pizza (1848, with six contradicting educational classes removed), SAREF (smart appliances, 1151), and OWL-Time (1364).

For the experiment, the model receives only the ontology's classes and properties, not its existing axioms, and must output the axioms. In the Direct setup, a single prompt requests all five axiom types at once as a JSON object with five keys, each mapping to a list of lists. In the AbA setup, each axiom type gets its own prompt and the outputs are aggregated afterwards, mirroring the two branches shown in the paper's Figure 2. Each setup is run with zero, one, or five labelled examples in the prompt.

Twelve instruction-tuned models were tested at a temperature of 0.2: Qwen2.5-Coder (14.8B), Qwen3 (30B), Llama 3.3 (70.6B), Mistral 3.1 Small (23.6B), DeepSeek R1 (32.8B), GPT-4o and GPT-4o-mini (parameter counts not publicly disclosed; estimated at approximately 200B and 8B), GPT-4.1 and GPT-4.1-mini (estimated approximately 1.8T and 8B), o1 (approximately 300B), o1-mini and o4-mini (approximately 100B each). The paper classifies models as large above 30B parameters and small below.

Predictions are scored automatically against the ontologies' real axioms using precision, recall, and F1. For disjointness, a prediction is counted as correct if the ontology states the relation in either direction. Because LLM outputs are not deterministic, the full experiment was repeated three times as a robustness check.

Why This Matters

For research, this is one of the few quantitative, multi-model studies of the top layer of the ontology learning layer cake. The authors state that, to their knowledge, the only prior work combining axioms with LLMs is Mateiu and Groza's GPT-3 fine-tuning study, whose evaluation was limited to a brief qualitative assessment of selected examples. OntoAxiom therefore gives the field a reusable benchmark and a baseline, and it corroborates earlier findings that expert-validated automatic axioms can be useful (Bühmann and Lehmann reported experts deeming 48.2% of proposed axioms useful extensions).

Real-world applications, drawn from the domains represented in the benchmark:

• Rail and transport infrastructure: the ERA ontology models European railway infrastructure, where consistency constraints on classes and properties matter for data exchange between operators.
• E-commerce product data: GoodRelations describes commodity products and services for web sales, where domain and range constraints keep product feeds well-typed.
• Smart appliances and IoT: SAREF provides shared vocabulary for smart applications, where disjointness and domain/range axioms help detect contradictory device descriptions.
• Temporal and scheduling data: OWL-Time covers temporal concepts, where property restrictions support reasoning about intervals and dates.
• Safety and news-derived knowledge: the Nord Stream Pipeline ontology was built from a news item, illustrating how quickly a lightweight ontology can be extended with candidate axioms after an event.

Industry relevance centres on cost. Ontology development in the paper is described as complex and resource-intensive, requiring both modelling and domain expertise, and it is described as hard for humans too. The authors' own results show that smaller models may remain viable in resource-constrained settings, that Mistral 3.1 Small and Llama 3.3 can occasionally beat some proprietary alternatives, and that proprietary models are more expensive to run with undisclosed technical details. The practical proposition is not replacement of ontology engineers but a shorter review-and-refine loop, and the paper's funding comes from the TrustLLM project with authors at TNO, Leiden University Centre for Linguistics, and Utrecht University.

Future Directions

• Broaden the axiom space: Extend beyond the five RDFS/OWL axioms to additional axioms, complex axiom structures, and full logical rules, and enlarge the benchmark with more ontologies as well as more axiom types.
• Soften and deepen evaluation: Since subclass and subproperty are transitive, a prediction such as "Pizza is a Thing" implied via "Pizza is a Food and Food is a Thing" could count as correct under more lenient scoring; the authors also suggest competency-question-based or human evaluation to judge quality and usefulness rather than only overlap with existing axioms.
• Try other prompting and input designs: Chain-of-thought prompting, other few-shot permutations, and different inputs such as domain-specific texts, documentation, requirements, or competency questions could guide the model towards more relevant axioms.
• Build an interactive workflow: Move toward a hybrid, semi-automatic process in which an LLM proposes candidate axioms and a developer selects among them for the specific use case, with ontology engineers remaining essential for complex axioms.

Target Audience

Ontology engineers and knowledge-representation researchers who want to know how much of axiom authoring can be delegated to LLMs; NLP and LLM evaluation researchers interested in a documented, reproducible benchmark with a full hyperparameter sweep; and applied teams in specialist domains (rail, retail, IoT, temporal data) weighing whether open-source models can substitute for proprietary APIs in a resource-constrained pipeline. Readers looking for a fully automated solution will not find one here, but they will find a clearly scoped baseline and a set of candidate-axiom use cases.

Authors’ abstract

Ontologies are an important tool for structuring domain knowledge, but their development is a complex task that requires significant modelling and domain expertise. Ontology learning, aimed at automating this process, has seen advancements in the past decade with the improvement of Natural Language Processing techniques, and especially with the recent growth of Large Language Models (LLMs). This paper investigates the challenge of identifying axioms: fundamental ontology components that define logical relations between classes and properties. In this work, we introduce an Ontology Axiom Benchmark OntoAxiom, and systematically test LLMs on that benchmark for axiom identification, evaluating different prompting strategies, ontologies, and axiom types. The benchmark consists of nine medium-sized ontologies with together 17.118 triples, and 2.771 axioms. We focus on subclass, disjoint, subproperty, domain, and range axioms. To evaluate LLM performance, we compare twelve LLMs with three shot settings and two prompting strategies: a Direct approach where we query all axioms at once, versus an Axiom-by-Axiom (AbA) approach, where each prompt queries for one axiom only. Our findings show that the AbA prompting leads to higher F1 scores than the direct approach. However, performance varies across axioms, suggesting that certain axioms are more challenging to identify. The domain also influences performance: the FOAF ontology achieves a score of 0.642 for the subclass axiom, while the music ontology reaches only 0.218. Larger LLMs outperform smaller ones, but smaller models may still be viable for resource-constrained settings. Although performance overall is not high enough to fully automate axiom identification, LLMs can provide valuable candidate axioms to support ontology engineers with the development and refinement of ontologies.