EvoOntology: A Self-Evolving Ontology Layer for Data Agents

AI summary

Overview

Research area: AI agents for data analysis — specifically bridging the gap between large language model (LLM) agents and heterogeneous data sources (databases, tables, files, documents).

Technical level: Intermediate. Readers should be comfortable with LLM agent scaffolds (ReAct), tool-use protocols such as MCP, text-to-SQL, and basic ontology concepts. No formal logic or ontology-engineering background is required.

Scope: The paper proposes EvoOntology, an automatically constructed and continuously self-improving ontology layer exposed to data agents as an MCP server, and evaluates it on three data-agent benchmarks across six LLM backbones.

What This Paper Is About

Data agents must answer natural-language questions over data that sits outside them — relational databases, spreadsheets, and unstructured files — which they can only reach through generic tools like SQL interfaces and file readers. Because the agent does not know the schema or vocabulary of these sources in advance, it wastes turns blindly probing for the right columns and concepts, a problem the authors call the agent–data gap.

The paper's goal is to insert a semantic middle layer between the agent and the data that persists what was learned, is exposed through callable tools rather than stuffed into the prompt, and adapts itself over time to both the data and the specific agent using it.

Key Contributions

1. The first autonomous interactive ontology layer for data agents. The ontology is encapsulated as a Model Context Protocol (MCP) server, so agents actively query it at runtime instead of passively consuming it as prefixed context. It is the only ontology content placed in the prompt; detailed records are fetched on demand.

2. A three-part ontology architecture. The layer separates a Content Layer (a typed semantic graph of Terms, Mappings, Constraints, and Evidence, connected by Semantic Relations and Structural References), a Schema Layer (defining the object model and permitted reference patterns), and a Tool Layer (two MCP tools, browse for retrieving top-n semantic matches and resolve for fetching records and linked objects, plus a session manifest).

3. A builder agent plus a self-evolution loop. A builder agent constructs an initial ontology from the training workload without seeing gold answers, using probe queries to verify every committed entry against real data. An evolution agent then refines the ontology through a four-step loop: diagnose, attribute, patch, and gate.

4. Attribution-guided typed edits gated by paired validation. Every proposed change is tagged as Content-, Tool-, or Schema-level, modifies exactly one level, and is accepted only if it beats its parent on a held-out validation set under the same backbone, decoding, and interaction budget.

Main Findings

• Large gains on multi-source data research. On DDR-Bench (10-K scenario), EvoOntology improves Trajectory-Wise accuracy on all six backbones by an average of +17.8 points over a no-ontology ReAct baseline, ranging from +4.8 (Qwen3.5-Flash) to +26.7 (GPT-5.5).

• Static semantic-layer injection is not just weaker — it is often harmful. Prepending the builder's semantic layer as a fixed prompt fragment ("Baseline + SL") drops Trajectory-Wise accuracy by −15.0 points on Claude-Sonnet-5, because it competes with other instructions and cannot be pruned per turn. Tool-mediated access to the same content avoids this.

• Episodic memory does not substitute for typed structure. A ReAct + Memory baseline that stores past trajectories as retrievable episodes lifts Trajectory-Wise from 69.5 to 75.8, still 13.7 points below EvoOntology's 89.5.

• Consistent but smaller gains on business analytics. On InsightBench, EvoOntology improves Overall score on every backbone, with a mean gain of +1.9 points and the largest improvement on DeepSeek-V4-Flash (+6.1). The authors attribute the smaller margin to saturation on short reference-style findings.

• Improvements in both correctness and efficiency for text-to-SQL. On BIRD under Oracle Knowledge, EvoOntology raises Execution Accuracy by an average of +7.4 points and Valid Efficiency Score by +8.6 points across all six backbones.

• The gate is the single most important design element. Ablating the validation gate costs −11.2 Trajectory-Wise points, more than removing attribution (−6.3), diagnosis (−4.8), or the typed patch (−1.7). Unfiltered candidates admit regressions that later rounds cannot always undo.

• All three editable levels are complementary. Restricting evolution to one level gives +8.7 (Content only), +13.2 (Tool only), and +3.6 (Schema only), none approaching the +20.0 of the full three-level loop. Tool-level edits account for 57% of cumulative accepted gain, Content-level 34%, Schema-level 9%.

• Mappings and Evidence are the load-bearing object families. Masking Mappings costs −13.4 points and masking Evidence −8.7, versus −3.5 for Constraints and −2.1 for Relations. This justifies requiring every committed entry to be anchored in an executable probe rather than a natural-language description.

• Backbones evolve divergent ontologies. Pairwise Jaccard overlap of accepted Term identifiers never exceeds 0.62, and applying one backbone's evolved store to another drops performance by at least 6.6 points, indicating that backbone-specific evolution is genuinely beneficial.

• The ontology reduces total cost. Average turns per task fall from 14.6 to 8.4 and total tokens per task from 52.6K to 42.0K — roughly 20% below baseline — despite a larger per-turn context, because repeated schema rediscovery is eliminated.

• Evolution converges rather than growing unboundedly. Content grows mostly in the first three accepted rounds (Terms rise from 61 to 80), with per-round growth dropping below 5% thereafter, flattening in step with performance.

Methodology in Plain English

The researchers treat the ontology as a small, queryable service rather than a block of text. Three pieces of the method matter:

Building the starting ontology. The builder agent reads the training workload (the questions users actually ask) and extracts recurrent entities, metrics, operations, and conditions. For each candidate concept, it fires probe queries against the raw data to check whether the declared type, filters, and value distributions hold up. Only candidates that pass verification are committed, and the supporting records are kept alongside them as Evidence. Nothing here requires gold answers.

Evolving it from real usage. After agents run tasks, the evolution agent clusters their trajectories to find recurring failure signatures, decides whether each signature stems from missing content, poor exposure through tools, or a representational limit of the schema, and proposes a patch at exactly one of those three levels. Keeping the edit to a single level isolates the hypothesis being tested and limits collateral damage.

Accepting only what helps. Each candidate ontology is evaluated against its parent on the same held-out validation set, with identical decoding settings and interaction budgets, under the same backbone. It is kept only if the improvement clears a margin threshold. Everything — accepted or rejected — is logged so ineffective edits are not retried. Evaluation itself uses a reciprocal two-fold split: the ontology is built and evolved on 70% of one fold, validated on the remaining 30%, then frozen before testing on the other fold, with folds swapped and scores averaged.

Why This Matters

This work reframes the semantic layer from a static artifact that humans author and paste into prompts into a runtime service that agents query and that improves from the agents' own behavior. It also supplies quantitative evidence that naive context injection of semantic metadata can actively degrade agent performance — a finding with direct implications for how retrieval-augmented and tool-augmented systems are built.

Real-world applications:

• Enterprise analytics on messy data estates. Business-intelligence agents answering questions that span data warehouses, exported CSVs, and PDF filings can amortize schema discovery across the whole workload instead of restarting per question.

• Text-to-SQL over unfamiliar or legacy databases. The BIRD results suggest an ontology layer improves both query correctness and execution efficiency, useful where schemas are poorly documented and column names are cryptic.

• Regulatory and financial document research. The DDR-Bench 10-K scenario mirrors the task of synthesizing answers across lengthy, heterogeneous corporate filings.

• Multi-agent or multi-model deployments. Backbone-specific evolution means a platform serving different models to different teams can let each model's ontology adapt to its own interaction patterns from a shared initialization.

Industry relevance: The roughly 20% reduction in total tokens per task addresses a real cost concern for production agent deployments, and framing the layer as an MCP server makes it compatible with the emerging tool-integration ecosystem rather than requiring a bespoke prompt pipeline.

Future Directions

• Reconciling divergent, backbone-specific ontologies. Since cross-backbone transfer costs at least 6.6 points, an open question is whether a shared core ontology plus per-backbone overlays could capture most of the benefit while avoiding N independently maintained stores.

• Scaling beyond the evaluated workloads. The experiments cover three benchmarks; whether the evolution loop remains stable when data sources are orders of magnitude wider, or when the workload distribution shifts over time, is untested.

• Reducing dependence on held-out validation data. The paired gate requires a labeled validation set with per-backbone scoring. Weaker-signal or fully unsupervised acceptance criteria would broaden applicability.

• Richer schema-level evolution. Schema edits contributed the smallest share of gain (9%) and the smallest single-level improvement (+3.6), yet they address limitations content edits cannot. Understanding when structural change is genuinely warranted is unresolved.

• Combining the ontology with memory mechanisms. Memory-based persistence captured only about a third of EvoOntology's gain; a principled integration of episodic traces and typed semantic structure may outperform either alone.

Target Audience

The paper is most valuable to researchers and engineers building LLM-based data agents, text-to-SQL systems, or tool-augmented analytics pipelines — particularly those who have hit scaling limits with prompt-injected schema descriptions. It is also relevant to practitioners working on semantic layers, data catalogs, and metadata management who want to understand how LLM agents change the requirements for those artifacts, and to MCP/tooling developers interested in a concrete case of encapsulating domain knowledge as a queryable server rather than context.

Authors’ abstract

Data agents aim to fulfill natural-language instructions over heterogeneous data, including tables, files, and databases. However, data agents face a challenging agent-data gap: heterogeneous data resides outside the agent, while the agent can access it (e.g., column names and file paths) only through generic tools. Existing approaches either let agents directly explore raw data sources or inject manually constructed semantic layers into prompts. However, neither scales well to large heterogeneous data sources nor adapts to different agent behaviors. In this paper, we introduce EvoOntology, a self-evolving ontology layer for data agents. EvoOntology encapsulates the ontology as an MCP server comprising a schema layer, a content layer, and a tool layer, enabling agents to actively query and interact with the ontology at runtime. To this end, we introduce a builder agent for autonomous ontology construction and a self-evolution loop that continuously refines the ontology through attribution-guided typed edits that are accepted only after a backbone-conditional paired evaluation. Experiments on three well-adopted data-agent benchmarks with four LLM backbones demonstrate that EvoOntology consistently outperforms strong baselines and existing semantic-layer approaches, effectively bridging the agent-data gap and enabling more effective interaction with heterogeneous data. Code: https://github.com/ruc-datalab/EvoOntology