Generative Ontology: When Structured Knowledge Learns to Create

AI summary

Overview

Research area: Artificial intelligence — specifically the intersection of knowledge representation (ontologies), large language model generation, and creative design automation.

Technical level: Advanced. The paper assumes familiarity with Pydantic schemas, DSPy signatures, retrieval-augmented generation, multi-agent pipelines, and intraclass correlation statistics.

Scope: The paper proposes "Generative Ontology," a framework that turns passive domain ontologies into executable schemas that constrain LLM generation, demonstrated on tabletop game design through a system called GameGrammar and evaluated with three empirical studies.

What This Paper Is About

Traditional ontologies can describe a domain precisely but cannot create anything new, while large language models generate fluently but produce outputs that "sound" valid without being structurally valid — a failure the author calls structural hallucination, such as a game design that mentions a mechanic without ever specifying its cards or end condition. The paper argues these two weaknesses are complementary and introduces Generative Ontology, which encodes domain knowledge as executable Pydantic schemas that constrain LLM output, orchestrated through DSPy signatures and a multi-agent pipeline. The goal is to show that constraints do not limit creativity but enable it, and to demonstrate this empirically in the domain of tabletop game design.

Key Contributions

1. The Generative Ontology framework. A method for converting an ontology into executable Pydantic schemas and DSPy signatures, so that every ontology class becomes a typed, validated field the LLM must satisfy. The ontology functions both as a generation grammar and as a validation contract.

2. A five-agent ontology pipeline with "professional anxieties." Rather than one LLM call designing everything, the pipeline assigns specialized roles to ontology domains: a Mechanics Architect, a Theme Weaver, a Component Designer, a Balance Critic, and a Fun Factor Judge, each carrying a professional worry (for example, the Balance Critic asks "What breaks when optimized?") intended to prevent shallow, agreeable output.

3. Retrieval-augmented grounding in design precedents. A two-phase retrieval strategy — ontology filtering on mechanism type, then semantic ranking by embedding similarity — grounds novel designs in a BoardGameGeek corpus of 1,767 games with 384-dimensional embeddings, dispatched through ChromaDB vector search or a SQLite structured fallback.

4. Three empirical studies with evaluator validation. An ablation over 120 designs and 4 conditions, a benchmark comparison of 30 generated designs against 20 published board games, and a test-retest ICC study of 50 evaluations that validates the LLM-based evaluator rather than assuming its reliability.

Main Findings

• Schema validation eliminates structural errors. In the ablation study (4 conditions × 10 prompts × 3 replications = 120 designs), mean consistency errors dropped from 5.03 (C1, raw LLM with no schema) to 0.10 (C2, schema validation only), an effect of d = 4.78, p < .001. Adding ontology constraints (C3, C4) eliminated errors entirely (0.00).

• Multi-agent specialization, not constraints alone, drives creative quality. Constraints alone (C2, C3) did not improve creative scores; the largest gains appeared from C3 to C4 (full pipeline) on strategic depth (d = 1.59, p < .001), elegance (d = 1.14, p < .001), and fun (d = 1.12, p < .001), with a moderate effect on tension (d = 0.79, p < .05). Replayability and social interaction showed no reliable differences across any conditions.

• Single-agent ontology generation sacrifices completeness. C3 achieved perfect consistency (0.00 errors) but only 83% completeness, because single-agent generation misses detail fields — resolved by multi-agent specialization in C4, which returned to 100% completeness.

• Structural parity with published games, but a bounded creative gap. Comparing 30 C4-generated designs against 20 published board games, both groups reached 100% completeness, and generated designs had fewer consistency errors (1.27 vs. 2.80, d = 0.76, p < .01). Published games scored significantly higher on 5 of 7 creative dimensions, with the largest effects on elegance (d = 1.72) and fun (d = 1.86); generated designs scored 7–8 while published games scored 8–9.

• Two dimensions already reach parity. Tension/drama (d = 0.35, ns) and social interaction (d = 0.19, ns) showed no significant differences between real and generated designs, suggesting the pipeline handles these comparably.

• Generated designs cluster in a mid-complexity band. Generated designs used a similar number of mechanisms (6.7 vs. 6.1, d = −0.60) but fewer component types (7.8 vs. 11.6, d = 1.12, p < .01), and produced neither ultra-light party games nor epic-scale strategy games.

• The LLM evaluator is reliable for most metrics. In the test-retest study (10 designs, fun ratings 6–9, each evaluated 5 times, 50 evaluations total), using ICC(2,1), 7 of 9 metrics achieved Good-to-Excellent reliability (ICC 0.836–0.989). Four were Excellent: tension & drama (0.989), social interaction (0.983), thematic cohesion (0.975), and elegance (0.934). Three were Good: fun rating (0.873), strategic depth (0.846), and player agency (0.836). Two were Moderate: replayability (0.578) and engagement variance (0.598). Across all metrics the range was 0.578–0.989 with a median of 0.873.

• Cost scales steeply with quality. C4 used approximately 27k tokens per design versus approximately 3.1k for C1, an 8.5× increase. C4 achieved 3,369 tokens per fun point versus 398 for C1, establishing a Pareto frontier.

• The constraint paradox. Constraints alone suppress richness; constraints combined with architectural specialization enable it. The author proposes a Constraint–Architecture Interaction Model in which creative quality is a function of constraint expressiveness multiplied by architectural specialization, not simply additive.

• The creative gap is framed as a feature. Because published games reflect decades of iterative playtesting and community feedback, the author argues the one-point gap reflects the absence of iterative refinement rather than a fundamental limitation — and notes that the parity dimensions (tension/drama, social interaction) may be those most amenable to specification-time design, while the gap dimensions (elegance, replayability) may require playtesting.

• Case study output is specific rather than vague. For the theme "bioluminescent fungi competing for dominance in a deep cave ecosystem" (2–4 players, medium complexity, cooperative, 60–90 minutes), the pipeline produced Mycelium: The Deep with three mechanisms (resource management, area control, engine building), a concrete component list (30 hex tiles, 3 card types, 60 growth cubes, 12 spore tokens, 20 flood markers), explicit victory and defeat conditions, and a Fun Factor Judge rating of 7/10.

Methodology in Plain English

The author starts from a prior tabletop game ontology that could decompose games like Catan into mechanisms, components, and player dynamics, then converts each ontology class into a Pydantic BaseModel. Fields carry minimum-length requirements and enum constraints — for instance, a game's goal must be at least 20 characters, and its mechanisms must come from a fixed MechanismType vocabulary of eight recognized types (worker placement, action points, deck building, resource management, area control, engine building, set collection, hidden information). Because these are enums, the model cannot invent mechanisms outside the taxonomy.

Generation is driven by DSPy, treating each LLM interaction as a typed operation. The signature's docstring becomes the system prompt and its output field is bound to the GameOntology schema, so DSPy builds the prompt, sends it, parses the response into the model, and validates it. A separate validation layer checks mechanism–component dependencies — deck building requires cards, area control requires a board, worker placement requires tokens — and DSPy's Assert mechanism retries generation with the error message as feedback when validation fails.

On top of this sits a multi-agent pipeline in four phases: sequential generation (Mechanics Architect, then Theme Weaver, then Component Designer), parallel critical evaluation (Balance Critic), refinement if moderate or severe issues are found, and experience assessment (Fun Factor Judge). Retrieval augmentation supplies precedents.

Evaluation uses two lenses. Structural metrics are computed deterministically (completeness, alignment, consistency). Creative metrics are scored by an LLM evaluator called the "Design Coach" on seven dimensions — strategic depth, tension/drama, player agency, replayability, social interaction, elegance, and thematic cohesion — each on a 1–10 scale, plus an overall fun rating and engagement variance. Generation used Claude Sonnet 4 (model ID claude-sonnet-4-20250514, temperature 0.7); the evaluator used the same model at temperature 0.

Why This Matters

Impact on research. The paper reframes ontology from a descriptive artifact into a generative one, and provides an ablation that isolates why structured generation works: schema validation solves validity, while specialization supplies creativity. It also makes a methodological contribution by reporting ICC values for every LLM-scored metric, letting readers distinguish "non-significant due to noise" from "non-significant due to genuine equivalence" — directly addressing the "who validates the validators" question raised in prior work.

Real-world applications:

• Tabletop and video game design. Rapid prototyping of complete, playable designs with explicit goals, end conditions, and mechanism–component coherence, rather than vague pitch documents. The GameGrammar system is available at https://github.com/bennycheung/GameGrammarCLI.

• Music composition. Scales, progressions, and forms as ontology elements; original compositions as output; harmonic validity and structure as validation criteria.

• Architecture and software design. Spaces, materials, flows, and building codes — or design patterns, components, and interfaces — as ontologies producing building designs or system architectures checked against structural integrity, compliance, consistency, coupling, and coverage.

• Legal documents, culinary arts, narrative, and scientific protocols. The paper's Table 10 lists eight applicable domains, including contracts validated for legal validity and completeness, recipes validated for flavor balance and feasibility, and experimental designs validated for reproducibility.

Industry relevance. The cost analysis is a practical signal: the full pipeline costs roughly 8.5× more tokens than raw generation and 3,369 tokens per fun point versus 398, so the framework comes with a measurable efficiency trade-off. The generality argument — that any domain with expert vocabulary, checkable validity constraints, and accumulated exemplars is a candidate — makes this relevant to any organization that needs to generate valid artifacts at volume, not just games.

Future Directions

1. Human calibration. The ICC analysis shows the evaluator is consistent but not that it is correct; LLM scores may be consistently biased. Comparing LLM scores against expert human judgment remains future work.

2. Iterative design loops. Running multiple generation–evaluation–refinement cycles to close the one-point creative gap, since the dimensions where generated designs fall short (elegance, replayability) are those that plausibly emerge from playtesting rather than specification.

3. Cross-model evaluation. All generation and evaluation used Claude Sonnet 4; testing other models would assess robustness of the findings.

4. DSPy optimization and temperature sensitivity. Using the ontology validation function as a reward signal for automatic prompt tuning, and testing whether evaluator reliability holds at temperatures above the 0 used here.

Beyond these, the paper leaves open whether the framework's parity dimensions and gap dimensions generalize across the eight candidate domains, and whether a larger, more diverse benchmark set would change the magnitude of the creative gap measured against the convenience sample of 20 titles.

Target Audience

This paper is most valuable to AI researchers working on structured generation, knowledge representation, or hallucination mitigation — particularly those interested in the reverse direction of the usual ontology-plus-LLM work, where the LLM builds the ontology. It also suits practitioners building multi-agent generation pipelines who want a concrete ablation showing which components actually move quality, and game designers or design-tool developers interested in automated design generation. Evaluation researchers will find the ICC test-retest analysis useful as a template for validating LLM-as-judge instruments, and readers interested in the philosophical framing (the paper grounds the framework in Whitehead's distinction between eternal objects and actual occasions) will find that discussion in Section 3.2 and Section 12.3.

Authors’ abstract

Traditional ontologies describe domain structure but cannot generate novel artifacts. Large language models generate fluently but produce outputs lacking structural validity, hallucinating mechanisms without components, goals without end conditions. We introduce Generative Ontology, a framework synthesizing these complementary strengths: ontology provides the grammar; the LLM provides the creativity. Generative Ontology encodes domain knowledge as executable Pydantic schemas constraining LLM generation via DSPy signatures. A multi-agent pipeline assigns specialized roles: a Mechanics Architect designs game systems, a Theme Weaver integrates narrative, a Balance Critic identifies exploits, each carrying a professional "anxiety" that prevents shallow outputs. Retrieval-augmented generation grounds designs in precedents from existing exemplars. We demonstrate the framework through GameGrammar, generating complete tabletop game designs, and present three empirical studies. An ablation study (120 designs, 4 conditions) shows multi-agent specialization produces the largest quality gains (fun d=1.12, depth d=1.59; p&lt;.001), while schema validation eliminates structural errors (d=4.78). A benchmark against 20 published board games reveals structural parity but a bounded creative gap (fun d=1.86): generated designs score 7-8 while published games score 8-9. A test-retest study (50 evaluations) validates the LLM-based evaluator, with 7/9 metrics achieving Good-to-Excellent reliability (ICC 0.836-0.989). The pattern generalizes beyond games. Any domain with expert vocabulary, validity constraints, and accumulated exemplars is a candidate for Generative Ontology.