GraphMERT: Efficient and Scalable Distillation of Reliable Knowledge Graphs from Unstructured Data

AI summary

Overview

Research area: Neurosymbolic AI — specifically, automatic construction of reliable domain-specific knowledge graphs (KGs) from unstructured text, combining encoder-only transformer models with symbolic graph structures.

Technical level: Intermediate. The paper discusses transformer architecture, masked language modeling, graph encodings, and ontology constraints, but presents its central argument (LLMs produce unreliable KGs) in accessible terms.

Scope: The paper proposes GraphMERT, an 80M-parameter encoder-only model that distills factual, ontology-consistent KGs from small, high-quality domain text corpora, and reports FActScore/ValidityScore comparisons against a 32B-parameter LLM baseline.

Index terms listed by the authors: hallucinations, interpretability, knowledge graphs, language models, neurosymbolic methods, retrieval-augmented generation.

What This Paper Is About

Automatically turning text into a reliable knowledge graph is still an open problem, and the currently dominant approach — prompting large language models to emit triples — is brittle, prone to hallucination, and blind to domain ontologies. The authors argue that a KG must be both factual (with provenance back to source sentences) and valid (relations consistent with a domain ontology) to count as reliable, and that off-the-shelf LLMs fail on both counts. Their goal is a small, domain-agnostic, scalable pipeline that produces reliable domain-specific KGs from small amounts of high-quality data, where no large expert-annotated triple-extraction corpus exists.

Key Contributions

1. GraphMERT, an 80M-parameter encoder-only model that jointly learns semantic representations from a small expert-curated seed KG (e.g., 100+ triples per relation) and syntactic representations from unstructured sentence-level text, by minimizing a standard masked language modeling (MLM) loss together with a proposed masked node modeling (MNM) loss.

2. A complete KG-extraction pipeline in which GraphMERT predicts semantic token completions for "chain graphs," an LLM converts those raw token completions into grammatically well-formed triple tails, and the resulting triples are filtered by similarity to the source syntactic context and de-duplicated. Triples are extracted at the sentence level, so each one can be traced back to its source sequence (provenance).

3. The first framework the authors claim to satisfy all six requirements for effective KG construction: factuality and provenance, validity, automation, scalability, domain generality, and global integration (cross-document linking rather than isolated text spans).

4. An empirical demonstration that LLM-generated domain KGs are unreliable, contrasted with GraphMERT on PubMed diabetes text, plus evidence that KG cleaning and treating the augmented KG as a new seed KG further improves factuality (with human expert editing and auditing possible on top).

Main Findings

• GraphMERT beats a far larger LLM on factuality: On text from PubMed papers related to diabetes, the KG produced by the 80M-parameter GraphMERT pipeline reaches a 69.8% FActScore, while a 32B-parameter baseline LLM (Qwen3-32B) reaches only 40.2%.

• GraphMERT also wins on ontology alignment: The GraphMERT-extracted KG achieves a ValidityScore of 68.7% versus 43.0% for the LLM-generated baseline. The introduction reports this as 68.8% for the GraphMERT KG, so the paper states two slightly different values for the same metric.

• KG cleaning improves factuality further: After cleaning, GraphMERT reaches 76.9% FActScore, compared with 55.6% for the LLM baseline.

• Iterative refinement is possible: GraphMERT can treat the augmented (cleaned) KG as a new seed KG and refine it further; human experts can then edit and audit the extracted KGs to raise reliability — something the authors describe as nearly impossible with purely neural representations.

• Frontier LLMs fail a simple ontology-consistency test: In a "reverse test" using a UMLS ground-truth triple — ⟨chronic kidney disease, has_finding_site, kidney structure⟩ — only Claude Sonnet 4.5 among Gemini 2.5 Pro, Claude Sonnet 4.5, GPT-5, and Grok 4 produced the correct tail. Gemini 2.5 Pro instead answered with "cerebellar gray matter," which the authors note should be linked via associated_with, not finding_site. GraphMERT, using the top-20 predicted tokens (top-5 softmax probabilities: 0.789, 0.173, 0.01, 0.003, 0.003) and Gemini 2.5 Pro as a helper LLM, recovered "kidneys," which corresponds to the UMLS gold answer.

• Scale mismatch motivates the approach: The largest public KGs are on the order of 10^8 entities (Wikidata 118M+ entities, PubGraph 385M+ entities), whereas top-tier LLMs in 2024 reported training datasets up to 10^13 tokens and pretraining corpora collectively surpassing 700TB — a mismatch that motivates a data-quality-first strategy.

• The authors explicitly warn against LLM-generated KGs in high-stakes domains such as medicine, law, business, and education, citing prompt sensitivity, shallow domain expertise, and hallucinated relations.

Methodology in Plain English

GraphMERT's name stands for Graphical Multidirectional Encoder Representations from Transformers. The approach rests on a two-part recipe.

First, the model is trained on a fusion of two kinds of data. A small, expert-curated seed KG supplies semantic knowledge, and a modest high-quality text corpus (roughly 100M tokens) supplies syntactic knowledge. These are combined into chain graphs: roots hold the syntactic knowledge from text (shown in orange in the framework figure), sparse leaves hold semantic examples from the seed KG (blue), and edges encode the semantic relations (purple arrows). The model is trained with two objectives at once — the standard masked language modeling loss on text and a new masked node modeling loss on graph nodes — so it learns to align semantic examples with their syntactic context.

Second, extraction runs in stages. GraphMERT predicts novel semantic token completions for chain graphs that have no injected semantic examples, using their syntactic information as context. Because those raw completions are just token lists, a helper LLM is used to combine them into grammatically well-formed triple tails (stage III in the figure). The candidate triples are then filtered by similarity to the source syntactic context and duplicates are dropped, yielding the final KG. Because extraction happens at the sentence level, every triple can be traced back to its source sentence.

The transformer's native self-attention only handles sequential input, so the authors do two things: they encode relation embeddings into the input graph sequences, and they modify attention weights to reflect spatial distance in the input graphs. They take inspiration from Graphormer but design alternative graph encodings tailored to language tasks.

The baseline for comparison is an LLM-based KG generation method: an off-the-shelf LLM (Qwen3-32B) prompted to generate the domain KG, following the currently dominant approach. The input domain in the reported experiment is text from PubMed papers related to diabetes, with the Unified Medical Language System (UMLS, and its SNOMED CT United States Edition vocabulary) providing the underlying ontology in the motivational example.

The paper's later sections — experimental setup (Sec. 5), results (Sec. 6), limitations (Sec. 7), and conclusion (Sec. 8) — are referenced but the provided content is truncated before those details, apart from the headline numbers in the abstract and introduction. The code is released at https://github.com/jha-lab/graphmert_umls, and the paper is reviewed on OpenReview (https://openreview.net/forum?id=tnXSdDhvqc).

Why This Matters

Impact on research. The paper takes a direct position in a longstanding debate: it argues that generative LLM prompting is the wrong tool for building knowledge graphs in specialized domains, and demonstrates that a tiny encoder-only model trained on a small, carefully curated corpus can outperform a model roughly 400 times larger on factuality and ontology validity. It also supplies a checklist (the six requirements) against which future KG-construction methods can be judged, and it reframes KG reliability as two distinct properties — factuality with provenance, and ontological validity — rather than a single accuracy number. The neurosymbolic stack it proposes (neural abstraction learning plus a symbolic KG for verifiable reasoning) is notable because the authors claim prior neurosymbolic frameworks largely failed to scale.

Real-world applications:

• Clinical and biomedical knowledge bases: The reported experiments are on PubMed diabetes literature; a reliable, provenance-linked KG could support clinical decision support where every relation traces to a source sentence and every error can be audited and corrected.
• Legal and regulatory knowledge bases: The paper names law as a high-stakes domain where editable, auditable, and forget-able knowledge representations matter — information can be deleted from a KG on legal or user request, which is described as far harder to do inside LLM weights without risking catastrophic unlearning.
• Enterprise and financial workflows: GraphMERT can seed agentic workflows driven by interaction with an editable knowledge base, and its extracted KGs can feed retrieval-augmented generation systems (including GraphRAG-style hierarchical summarization), where answer quality depends directly on KG coverage, validity, and factuality.
• Education and domain-specific assistants: The paper lists education among the high-stakes domains where hallucinated relations are unacceptable, and where a small, domain-adaptable extraction pipeline is more practical than retraining a large model.

Industry relevance. The core selling point is cost and control. The approach requires no pretraining on large unverified text, no manual feature engineering or rule handcrafting, and no human expert in the loop for the base extraction — and it runs at 80M parameters rather than billions or trillions. For organizations working in regulated or proprietary domains where high-quality data is scarce and copyright or privacy concerns restrict access to training corpora, a compact model trained on a small curated corpus plus a seed KG is described as far more practical than fine-tuning or continued pretraining a frontier LLM.

Future Directions

• Scaling behavior: The authors state the framework "can be scaled when provided with more data and given extra compute resources," but do not report how factuality, validity, or runtime behave as data volume and compute grow — the scalability claim is stated as a property, not measured in the provided content.
• Error propagation from the helper LLM: The pipeline uses an LLM to turn GraphMERT's raw token candidates into well-formed triple tails. The motivational example shows an LLM's own reasoning can be unreliable on ontology questions, so how much the helper stage limits end-to-end reliability is an open question the paper's provided content does not resolve.
• Generalization beyond the tested domain: The paper claims domain generality because no domain-specific parameters are hard-coded and it reports the method is domain-agnostic, but the reported experiments use PubMed diabetes text with UMLS. Transfer to other ontologies, other relation vocabularies, and cold-start entities remains to be demonstrated.
• Human-in-the-loop refinement loops: The paper notes that experts can edit and audit extracted KGs, and that the augmented KG can be re-used as a seed KG, but the provided content does not report how much these interventions improve reliability or how much expert effort they require.
• Standardized evaluation: The paper asserts that LLM-generated KGs should not be used in high-stakes domains, but how FActScore and ValidityScore behave across different corpora, seed KG sizes, and ontologies is not reported in the available content.

Target Audience

Researchers and practitioners in neurosymbolic AI, knowledge graph construction, and NLP who are evaluating whether to build domain knowledge bases with prompted LLMs or with dedicated smaller models. It is also relevant to applied teams in medicine, law, finance, and education who need auditable, editable, provenance-carrying knowledge representations, to engineers building retrieval-augmented and GraphRAG systems on top of KGs, and to readers interested in the quality-versus-quantity argument for training data. Readers seeking full architectural and experimental detail should consult the paper's Sections 5 and 6, which are not contained in the provided content.

Authors’ abstract

Researchers have pursued neurosymbolic artificial intelligence (AI) applications for nearly three decades. A marriage of the neural and symbolic components can lead to rapid advancements in AI. Yet, the field has not realized this promise since most neurosymbolic AI frameworks fail to scale. In addition, the implicit representations and approximate reasoning of purely neural approaches limit interpretability and trust. Knowledge graphs (KGs), a gold-standard representation of explicit semantic knowledge, can address the symbolic side of the problem. However, automatically deriving reliable KGs from text corpora remains an open problem. We address these challenges by introducing GraphMERT, a tiny graphical encoder-only model that distills high-quality KGs from unstructured text corpora and its own internal representations. GraphMERT and its equivalent KG form a modular neurosymbolic stack: neural learning of abstractions; symbolic KGs for verifiable reasoning. GraphMERT + KG is the first efficient and scalable neurosymbolic model to achieve state-of-the-art benchmark accuracy along with superior symbolic representations relative to baselines. Concretely, we target reliable domain-specific KGs that are both (1) factual (with provenance) and (2) valid (ontology-consistent relations with domain-appropriate semantics). When a large language model (LLM), e.g., Qwen3-32B, generates domain-specific KGs, it falls short on reliability due to prompt sensitivity, shallow domain expertise, and hallucinated relations. On text obtained from PubMed papers on diabetes, our 80M-parameter GraphMERT yields a KG with a 69.8% FActScore; a 32B-parameter baseline LLM yields a KG that achieves only 40.2% FActScore. The GraphMERT KG also attains a higher ValidityScore of 68.8%, versus 43.0% for the LLM baseline.