Ontology-Driven Structural Regularization for Document-Level Relation Extraction

AI summary

Overview

Research area: Natural Language Processing, specifically Document-Level Relation Extraction (DocRE) and knowledge graph construction, with an emphasis on data quality and ontology-based (OWL) constraint checking.

Technical level: Advanced. The paper assumes familiarity with the DocRE task, the DocRED/ReDocRED benchmarks, RDF triple semantics, OWL property axioms (domain/range, owl:inverseOf, owl:AsymmetricProperty, cardinality restrictions), and transformer-based DocRE architectures such as ATLOP and DREEAM.

One-sentence scope: The paper proposes an ontology-driven framework that measures and removes structural inconsistencies in DocRE training data, and shows that doing so reduces logically ill-formed model predictions and improves generalization.

What This Paper Is About

DocRE systems depend on large, distantly supervised datasets such as DocRED distant, but that data is noisy because distant supervision assumes that whenever a knowledge graph asserts a relation between two entities, every document mentioning the pair expresses it. The authors argue that beyond the well-studied semantic noise, there is an overlooked category of structural noise: triples that violate ontology constraints, omit required inverse relations, contradict asymmetry, or break cardinality limits. The goal is to quantify this structural noise under OWL semantics and to remove it through a lightweight, model-agnostic pre-processing step before training.

Key Contributions

1. An ontology-driven diagnostic framework. The authors formalize DocRE as knowledge graph construction under the RDF model and define four OWL-grounded structural constraints: invalid triples (domain/range violations), missing inverse relations (owl:inverseOf), asymmetric relation violations (owl:AsymmetricProperty), and cardinality violations (functional properties and qualified cardinality restrictions).

2. A first quantitative audit of structural inconsistencies in DocRE data. They measure these violations across the DocRED manual training set, the DocRED distant dataset, and the ReDocRED train/dev/test splits, reporting previously unreported violation rates.

3. A model-agnostic cleaning pipeline. They build a structurally corrected version of DocRED distant by removing entities involved in violations and automatically adding missing inverse triples, without changing model architectures, loss functions, or inference procedures.

4. An empirical study of structural regularization. They train ATLOP and DREEAM (with BERT and RoBERTa backbones) on both the original and corrected distant data, showing fewer ill-formed predictions and better performance, with random-subsampling controls to isolate the cause. The framework is released as open source at the repository listed in the paper.

Main Findings

• Distant supervision is structurally much noisier than manual data. Compared to the ReDocRED manual training split, the DocRED distant dataset contains three times more invalid triples, nearly five times more missing inverse relations, and twice as many asymmetric inconsistencies.

• Absolute violation counts (Table 1). DocRED distant holds 1,505,638 triples with 46,211 invalid triples (3.07%), 85,065 missing inverse relations (40.26%), 12,408 asymmetric inconsistencies (0.84%), and 25 cardinality violations (0.01%). ReDocRED train holds 85,932 triples with 810 invalid (0.94%), 947 missing inverse (7.89%), 284 asymmetric (0.34%), and 20 cardinality violations (0.40%). ReDocRED dev (17,284 triples) shows 122 (0.71%), 243 (8.92%), 56 (0.33%), 3 (0.32%); ReDocRED test (17,448 triples) shows 141 (0.81%), 242 (9.27%), 82 (0.48%), 6 (0.65%). DocRED manual train (38,180 triples) shows 602 (1.58%), 2,696 (64.36%), 154 (0.41%), 10 (0.28%).

• A known annotation flaw is confirmed and repaired. The DocRED manual training set has the worst missing-inverse rate (64.36%); ReDocRED reduced it to 7.89%. The paper attributes this to the corrections introduced by Huang et al. (2022) and Tan et al. (2022b).

• Distant data unexpectedly has the fewest cardinality violations. DocRED distant reports 0.01%, the lowest across datasets, while the other datasets range from 0.28% to 0.65%. The authors attribute this to the construction method, which aligns Wikipedia documents with Wikidata triples and thereby preserves their cardinalities.

• Evaluation data is not structurally clean either. ReDocRED splits contain invalid triples between 0.71% and 0.94%, missing inverse relations below 10%, asymmetric violations from 0.33% to 0.48%, and cardinality violations reaching 0.65% on the test set.

• Most invalid triples come from entity errors, not relation errors. A simple random sample of 400 invalid triples from DocRED distant, annotated manually and analyzed with a 95% Highest Posterior Density credible interval (α = 0.05), shows that 69% (±4.5%) of invalid triples stem from entity annotation errors and 31% from incorrect relation annotations.

• Structural noise propagates into predictions. Models trained on DocRED distant and evaluated on ReDocRED test produce invalid triples at 0.85% to 1.61% (about one third of the training-data rate), missing inverse relations at 41.18% to 45.22% (close to the 40.6% rate in training data), and asymmetric violations at 0.53% to 0.79%, matching the 0.84% training rate. ATLOP appears slightly more sensitive than DREEAM; the authors suggest DREEAM's self-training with a teacher trained on manual data gives it more robustness.

• Cleaning sharply reduces ill-formed predictions. On average, the proportion of invalid triples in predictions falls by −62.92%. Missing inverse relations drop by −87.44% for ATLOP-RoBERTa and −86.97% for ATLOP-BERT, and by −74.13% for DREEAM. Asymmetric violations are nearly eliminated: ATLOP-RoBERTa goes from 0.60% to 0.00%.

• Cleaned data improves generalization. Relative to DocRED distant, ATLOP gains on average +1.65% precision, +3.10% recall, +2.69% F1, +3.83 ignPrec, and +4.59% ignF1. DREEAM gains on average +14.07% recall (with DREEAM-BERT up +19.32%), +10.02% F1, +1.96% ignPrec, and +10.67% ignF1, while precision slips slightly by −0.45%.

• Random removal does not reproduce the gains. Removing the same 167,234 triples at random (ten seeds, standard deviation always below 1%) yields only marginal improvements and even degrades some metrics: ATLOP-RoBERTa recall and F1, precision for both DREEAM variants, and ignPrec for DREEAM-BERT. The authors conclude the benefit comes from correcting structural errors, not from training on less data.

• Overall headline numbers. The conclusion reports an average gain of +6.36% in F1 and +7.62% in ignF1 from training on structurally consistent data.

Methodology in Plain English

The authors treat each predicted relation as an RDF triple of the form ⟨subject, predicate, object⟩ and ask whether that triple would be legal under an ontology.

Building the rules. DocRED relations are each mapped to a Wikidata property. From Wikidata's property descriptions the authors read off the permitted "subject type constraint" (domain) and "value type constraint" (range), and they supplement these with the most common subject and object entity types observed in the ReDocRED training data. Entity types come from the six DocRED types: PER, ORG, LOC, TIME, NUM, and MISC. To avoid overly restrictive rules, MISC is included in the domain and range of every relation. For example, P19 (place of death) has domain "PER|MISC" and range "LOC|MISC". A triple is invalid if the subject type is outside the domain or the object type is outside the range.

Three further rules. A missing inverse relation occurs when a triple is present but its declared owl:inverseOf counterpart (for instance P1376 capital of, the inverse of P36 capital) is absent. An asymmetric violation occurs when both ⟨e_s, r, e_o⟩ and ⟨e_o, r, e_s⟩ appear with the same relation r; only P26 (spouse) and P3373 (sibling) are symmetric, so all other relations are treated as asymmetric. A cardinality violation occurs when a subject entity has more distinct objects for a relation than the maximum cardinality k_r allows. Domain knowledge initially flagged 11 relations as strictly functional (k_r = 1), but 7% of those instances in ReDocRED training have cardinality 2, and 15% of P570 (date of death) instances have k_r > 1, so the limit was relaxed to k_r = 2.

Measuring the noise. They apply these rules to DocRED manual train, DocRED distant, and ReDocRED train/dev/test, reporting both absolute counts and percentages computed per rule type.

Diagnosing the cause. Because annotating all invalid triples in DocRED distant would be infeasible, they draw a simple random sample of 400 invalid triples, manually label each as an entity annotation error or a relation annotation error, and estimate the population proportion with a 95% HPD credible interval.

Cleaning the data. Correcting every invalid triple by hand would require revising 58,619 instances (46,211 invalid triples plus 12,408 asymmetric inconsistencies). Instead they take a conservative, scalable route: for each invalid triple or asymmetric violation, they remove the entity involved (and therefore all triples tied to that entity in the document). This removes 208,665 triples plus 3 documents for invalid triples and 31,178 triples for asymmetric violations. Missing inverse relations are added automatically, contributing 72,609 new triples. The corrected dataset has 1,338,404 training instances (−167,234 relative to DocRED distant) across 101,870 documents (−3).

Training and evaluation. ATLOP and DREEAM are each trained with BERT and RoBERTa backbones on DocRED distant and on the corrected version, validated on ReDocRED dev, and evaluated on ReDocRED test. Because DREEAM follows a teacher-student scheme, the authors use the student right after self-training, with the teacher trained on the ReDocRED training set. As a control, models are also trained on a random subsample of DocRED distant with the same 167,234 triples removed, repeated across ten seeds.

Scope decisions. The authors also examined biomedical DocRE datasets but found none suitable: BioRED and BC5CDR contain only one relation type ("associated_with"), and GutBrainIE had already been preprocessed to remove structural inconsistencies. Results on the denoised distant datasets produced by UGDRE and DOREMI are described as reported in an appendix that is not included in the content reviewed here.

Why This Matters

Impact on research. The paper reframes structural consistency as a missing axis of supervision in DocRE, alongside the semantic correctness that existing denoising methods target. It also argues that entity extraction is a bottleneck: since roughly 69% of invalid triples trace back to entity misannotation, relation extraction quality is capped by entity recognition quality. The authors note their gains should be read as conservative estimates because they do not touch entity extraction at all.

Real-world applications.

• Knowledge graph construction from large document collections, where ill-formed triples degrade downstream graph quality.
• Information retrieval and question answering systems that consume structured knowledge extracted from text.
• Data mining and recommendation systems that rely on graph-structured inputs, as cited in the paper's list of downstream DocRE applications.
• Large-scale distant supervision pipelines, which can now be audited and cleaned automatically before training rather than manually curated.

Industry relevance. Distant supervision is attractive precisely because manual annotation is costly, and the paper shows a cheap, model-agnostic pre-processing step can make that data usable. Because the pipeline requires no changes to architecture, loss function, or inference, it plugs into existing sequence-based or graph-based DocRE systems. The released toolkit works with any dataset in the DocRED schema and accepts custom constraints.

Future Directions

• Jointly addressing entity-level noise. Since entity misannotations dominate invalid triples, combining structural cleaning with improved named entity recognition or targeted entity denoising could yield larger gains than the conservative estimates reported here.
• Re-annotation instead of entity removal. The current strategy discards affected entities; systematic re-annotation could preserve more of the 208,665 and 31,178 removed triples while improving data quality.
• Building a structurally clean evaluation benchmark. The authors acknowledge that ReDocRED test itself contains logical inconsistencies, and that re-annotating it could be perceived as tailoring the benchmark to their approach.
• Completing the empirical error analysis. The paper states its error analysis is primarily empirical and, in the truncated content, does not yet quantify a further aspect; extending this analysis is left open.
• Extending constraint-based methods to DocRE more broadly. The paper notes that constraint-based approaches in sentence-level relation extraction have not been extended to the document level.

Target Audience

Researchers and practitioners working on document-level relation extraction, distant supervision and denoising, and knowledge graph construction, particularly those who train models on DocRED distant or similar automatically aligned corpora. It is also useful for dataset curators and benchmark maintainers interested in ontology-based data quality auditing, and for engineers who want a drop-in, model-agnostic pre-processing step that requires no architectural changes.

Authors’ abstract

Document-Level Relation Extraction (DocRE) relies heavily on costly manually annotated datasets, while large distant supervision resources such as DocRED distant remain underexploited due to noise. We show that a critical yet overlooked source of noise lies in structural inconsistencies within relational triples, including violations of ontology constraints and logical contradictions. We introduce an ontology-driven framework to quantify and enforce structural consistency in DocRE datasets. Our analysis reveals substantial structural noise in DocRED distant and demonstrates that such inconsistencies propagate to model predictions. Enforcing structural well-formedness during training significantly reduces logical contradictions and consistently improves generalization performance. These findings establish structural consistency as a missing axis of supervision in DocRE and highlight structural regularization as an effective strategy for leveraging distant data at scale.