Relations, Events, and Slot Extraction

A report may mention a supplier, a device, a date, and a failure without stating that the supplier caused the failure. Linking nearby entities as a relation would invent a claim.

That inference is not a hypothetical error. It is a published method's stated premise, written down by its own authors. Distant supervision, set out by Mintz, Jurafsky and colleagues in 2009, assumes it outright: “The distant supervision assumption is that if two entities participate in a relation, any sentence that contain those two entities might express that relation.” The sentence reads “any sentence that contain” in the original. Applied deliberately and at scale the assumption is productive: they report 10,000 extracted instances of 102 relations at 67.6% precision. It is also priced. 67.6% is what treating co-occurrence as evidence costs.

What trained annotators do with that output has been counted too. DocRED's builders used the technique to pre-fill relation candidates for human review, then reported the survival rate: “Finally 57.2% relation instances from entity linking and 48.2% from RE models are reserved”. The team that later re-annotated DocRED restated the same pipeline per document: “On average for each document, 19.9 triples are suggested from distantly supervised data and 7.8 triples are recommended by relation extraction models… an average of 12.5 triples per document are kept in the DocRED dataset.” Roughly 27.7 machine-proposed triples per document. About 12.5 survive a human reading the sentence.

Structured extraction must therefore identify the textual predicate or evidence that supports a relation, then represent uncertainty and alternatives. Database-shaped output is not automatically database-quality evidence.

TACRED drew that line by making the absence of a relation a label in its own right. Zhang, Manning and colleagues built it in 2017. It is trained over 42 classes: forty-one relations plus a no relation class, and its authors “minimize cross-entropy loss over all 42 relations”. The payoff they report is operational rather than academic. Their model replaced the relation-extraction component of the best TAC KBP 2015 slot filling system, and “its F1 score increases markedly from 22.2% to 26.7%”.

Each layer adds commitments somebody has to validate separately.

The event layer is not an abstraction invented for this lesson. It was fixed in writing on 1 July 2005, by the Linguistic Data Consortium, before any of the systems that now populate it existed. The ACE annotation guidelines for events, version 5.4.3, define 8 event types with 33 subtypes — LIFE 5, MOVEMENT 1, TRANSACTION 2, BUSINESS 4, CONFLICT 2, CONTACT 2, PERSONNEL 4, JUSTICE 13. Every event has to carry a trigger word plus typed arguments.

The guidelines also refuse to let uncertainty live in a comment field. Four separate attributes hang off every annotated event: POLARITY, TENSE, GENERICITY and MODALITY. The last of them is defined outright: “An Event is ASSERTED when the author or speaker makes reference to it as though it were a real occurrence.” Negation, genericity and mere report are columns in the schema. They are not advice to the annotator.

The corpus carrying that schema, LDC2006T06, was released on 15 February 2006. It holds 303,833 English words, 334,121 Chinese characters and 112,233 Arabic words. Viet Dac Lai's survey of event extraction reproduces the type table and confirms the shape of it: “The ACE-2005 defines 8 event types and 33 event subtypes as presented in table 2. This dataset annotates 599 documents”. Every layer below is a design decision somebody wrote down and versioned. A record that does not name which version it was built under cannot be checked against it.

The best choice depends on schema size, document length, overlap, data, and control requirements.

The document-level column is the one with a measured size. DocRED, published in 2019, annotates 132,375 entities and 56,354 relational facts over 5,053 Wikipedia documents with 96 relation types. Its introduction says why the corpus exists: “According to the statistics on our human-annotated corpus sampled from Wikipedia documents, at least 40.7% relational facts can only be extracted from multiple sentences, which is not negligible.” A sentence-level pipeline on this data is not merely less convenient. At least 40.7% of the facts are outside anything it can see.

The cost of that reach is combinatorial rather than conceptual. A DocRED document averages 19.5 entities. Across the 4,053 non-blind documents, the Re-DocRED team computed an average of 393.6 candidate entity pairs per document. A document-level model buys cross-sentence coverage by classifying hundreds of pairs where a sentence-level model classifies a handful. On DocRED's original labels, 97.1% of those pairs are marked no_relation. That ratio is what “higher context cost” and “needs document splits” cost in practice. It is also why a document-level score is so sensitive to which negatives are real.

A high-quality corpus must contain cases where proximity and surface cues fail. Four of the cases below are not merely hard. ACE 2005 gives them dedicated fields — POLARITY, TENSE, GENERICITY and MODALITY — precisely because an annotator who resolves them silently destroys the distinction.

The final consumer should be able to inspect why each field exists. ACE 2005 is a worked example of steps one and two: types and subtypes enumerated, a trigger required for every event, argument roles typed, and POLARITY, TENSE, GENERICITY and MODALITY recorded alongside the event rather than folded into it.

Entity F1 cannot reveal whether the correct supplier was linked to the correct outage. The BioNLP'09 shared task on event extraction made that separation the design of the task. Every participating system was handed the gold protein entities. The organizers conceded the cost in one line: “This is the only feature of the task setting that notably detracts from its realism.” With the entity layer free and perfect, the best system, UTurku, scored 51.95 F overall — recall 46.73, precision 58.48. The breakdown is the point: 70.21 F on simple events, 44.41 on binding, 40.11 on regulation. The best ensemble of participating systems reached 55.96. The Turku group's own later system, reported in Bioinformatics in 2010, does not change the picture: “The event extraction system (without the NER component) achieves an F-score of 52.86% (precision 58.13% and recall 48.46%) on the BioNLP'09 Shared Task test set.” Perfect entities, and the structure over them is still wrong about half the time.

So measure trigger, argument, relation, coreference, normalized value, attribution, and complete-record validity. Use strict and relaxed span scores where appropriate, but report exact operational fields separately.

Document-level evaluation should group by source to avoid repeated entities leaking across splits. It should also be read against the completeness of the gold standard, which two independent teams have now measured on DocRED. In 2022 two experts at Peking University relabelled 96 DocRED documents from scratch. Their group reported what the original revision had missed: “However, it turns out that 95.7% of these missing instances are still left out even after revision.” The recommend-and-revise workflow did not recover what the recommender never proposed. The Re-DocRED team, also in 2022, re-annotated the same 4,053 non-blind documents and raised the triple count from 50,503 to 120,664. The share of entity pairs labelled no_relation fell from 97.1% to 94.0%, and model F1 rose by around 13 points. A model scored against a key that a later re-annotation more than doubled is being penalised for relations that are in the text and not in the labels.

Select a paragraph that contains an event, two possible causes, one negated relation, and a quoted statement. Annotate triggers, arguments, modality, source, and normalized time. Borrow the ACE 2005 attribute set rather than inventing one: POLARITY, TENSE, GENERICITY and MODALITY, with a trigger word required for every event.

Have a second annotator work independently. Then write adjudication rules for each disagreement. Identify which uncertainties should remain in the output rather than being forced into one record. Record the normalized time separately from the expression that produced it, since on gold spans a strong normalizer still returned the wrong VALUE 23.9 times in a hundred at TempEval-3 scale.