Calibrated Typed Decisions for Entity, Relation, and Event Extraction

AI summary

Overview

Research area: Natural Language Processing — information extraction (named entity recognition, relation extraction, event extraction) with schema-conditioned models; also touches on reinforcement-learning-style training objectives and probability calibration.

Technical level: Intermediate. Readers should know what an encoder is, what F1 scores mean, and roughly how span-based extraction and softmax classification work. The reinforcement-learning and calibration sections assume some familiarity with policy-gradient ideas and expected calibration error.

Scope: The paper introduces CalOn, a single 0.6B-parameter encoder trained to extract entities, relations, and events under user-specified type schemas, evaluated zero-shot across 40 benchmarks, with an added reward term on sampled extraction decisions and a correction for a probability bias introduced by down-weighting null candidates.

What This Paper Is About

Information extraction models tend to work well on the data they were trained on but degrade when moved to new domains or new annotation guidelines, and simply letting a user type in the desired types does not close that gap. The authors build a compact extractor that reads the text together with the requested entity and relation types (and optional descriptions) and then decides, for each candidate span or span pair, whether it is one of the requested types or nothing at all. The goal is a single model that transfers across datasets and schemas without being retrained per schema, while producing confidence scores that are actually usable after calibration.

Key Contributions

1. A unified schema-conditioned encoder for three tasks. CalOn is a 0.6B-parameter encoder built on Qwen3-0.6B with the causal mask removed. It handles entities, relations, and events through one common formulation: every candidate span (or ordered span pair) is assigned a probability distribution over the queried types plus a null class. The same span-pair formulation is reused for tasks beyond conventional IE, including aspect sentiment triplet extraction, pronoun resolution, coreference, and temporal/causal links between events.

2. A training objective combining likelihood with relative rewards on sampled decisions. For each candidate, Gaussian noise is added to the type logits to produce several sampled decisions. A sample scoring a gold type gets reward 1, a non-gold type gets −1, and the null class gets 0. Each sample's advantage is computed against the mean reward of the other samples (GRPO-style, with no separate critic), and this signal is added to the supervised loss.

3. A label-status scheme for mixing heterogeneous training sources. Each queried type in a training instance is marked positive, negative, closed-world negative, or unknown, so that partially annotated sources (e.g., RAMS, GENEVA) are not treated as if unannotated meant absent, while exhaustively annotated sources (e.g., Few-NERD) are.

4. A prior correction for the null-class down-weighting. Because null candidates are given a weight ω ≤ 1 in the loss, the model's null-vs-type odds are reported ω times too low. Adding log(1/ω) to the null logit before temperature scaling fixes this without changing candidate rankings at a fixed temperature, improving calibration with little change in F1.

Main Findings

• Broad benchmark lead. Across 40 benchmarks, CalOn outperforms the evaluated baselines on most benchmarks. It leads every reported relation extraction benchmark in the paper. Entity averages: CrossNER 67.8 versus GLiNER2 62.4, GLiFormer 67.6, GNER 63.1, InstructUIE 48.5, GoLLIE 52.7; specialised/other domains 56.6 versus 41.1, 41.5, 29.3, 30.8, 36.3; personal information 58.5 versus 50.2, 54.2, 26.1, 44.8, 47.5.

• Relation extraction is the strongest result. With gold entities, CalOn averages 45.4 on CrossRE (best baseline 11.6) and 36.0 end-to-end (best baseline 10.6). On the four ASTE datasets it averages 83.0 with gold entities versus 4.9 for GLiNER2 and 17.1 for GLiREL; end-to-end 55.0 versus 4.6.

• Relation rejection matters as much as typing. When only pairs with an annotated relation are supplied, GLiREL's average relation typing accuracy across the 18 RE benchmarks is 40% versus 79% for CalOn. When every pair of gold entities is considered, average F1 is 2.6 versus 49.8.

• Event extraction leads on average, with a caveat. CalOn has the highest average F1 for trigger classification and for argument classification with gold triggers (35.1 versus 4.5, 33.2, 10.0). InstructUIE's average end-to-end argument F1 exceeds CalOn's because of its CASIE score (62.6 with gold triggers), but the CASIE training split was included in that checkpoint's training. Excluding CASIE, CalOn is highest on all three EE metrics, with end-to-end argument F1 of 10.9 versus InstructUIE's 8.4.

• Some per-dataset losses. GLiFormer leads on CrossNER Music (76.2 vs 72.0) and Science (69.0 vs 64.4). InstructUIE's MIT Restaurant score (80.9) lifts its MIT average (66.9) above CalOn's (62.3), and both MIT datasets were in that checkpoint's training data.

• Descriptions help most for relations. On CrossRE, type descriptions raise average F1 with gold entities by about 16 points, while NER group averages change by one to three points.

• Frequent type names do not explain the gains. Every evaluated entity type also occurs by name in the training data, and there is no consistent per-type F1 increase with how often a type is annotated during training. CalOn's largest margin over the strongest baseline occurs for the least frequently annotated third of entity types.

• Joint distribution beats independent scores. Replacing the joint softmax-over-types-plus-null with independent sigmoid scores drops average NER F1 from 61.5 to 53.4, the largest NER decrease among the ablations. Thresholding the most probable type instead of total non-null probability also lowers NER F1.

• Calibration improves, F1 barely moves. After correction and refitting, average expected calibration error falls from 0.25 to 0.10 for relations, 0.33 to 0.23 for entities, and 0.51 to 0.39 for event arguments. Event arguments retain the highest calibration error.

• Large throughput advantage, higher memory. On a GH200 GPU with bf16, CalOn processes 120 sentences per second on MIT Movie with all 12 types, 5 to 25 times the baseline throughputs (GLiNER2 22, GLiFormer 18, InstructUIE 16, GoLLIE 10.8, GNER 4.9). Peak GPU memory is 8.0 GB, higher than GLiNER2 (2.9 GB) and GLiFormer (3.2 GB).

• Reward term helps mainly relations. With type descriptions, the decision reward mainly benefits RE; treating unknown types as negatives lowers F1 on all three tasks. An MLP head variant slightly improves NER but lowers RE and event-argument F1.

Methodology in Plain English

Input construction. The model's input is a flat token sequence: special markers followed by each requested entity type and an optional short description, then each relation type with its description, then the text itself. The text is tokenized separately so every token keeps its character offsets — this is what keeps every extracted mention anchored to actual input text, avoiding the boundary drift and hallucinated spans the paper attributes to generative extractors.

Candidate representation. Any contiguous span of at most W text tokens becomes a candidate, described by its first token, last token, the average of its tokens, and a learned width embedding. A relation candidate is an ordered pair of entity spans (at most 24 spans per instance in training, with gold relation participants placed first), described by the two span vectors, their element-wise product, and the average of the tokens lying between them. Because (a, b) and (b, a) are different candidates, direction is predicted directly.

Interaction and scoring. A small stack of Transformer blocks without position information lets the queried types, the candidates, and a learned null vector attend to one another, so a type's score for a span depends on which other types were requested and which overlapping candidates exist. A bilinear form of rank r then produces one logit per type plus a null logit, converted by softmax into a distribution.

Two-stage decision. Extraction and typing are deliberately separated. A span is extracted if 1 − p(0), the total probability mass on all requested types, clears a threshold τ; it is then labelled with the most probable requested type. This avoids rejecting a span whose evidence is split between a coarse and a fine type.

Training. The supervised loss maximises the total probability of the gold type(s) for each candidate, with null candidates down-weighted by ω because they dominate. On top of that, for each candidate the logits are perturbed with Gaussian noise S times, each perturbed sample takes an argmax decision, and the decision is rewarded as the evaluation metric would score it (+1 gold type, −1 wrong type, 0 null). The advantage over the other samples' mean reward, multiplied by the log-probability of that perturbation, is added with weight ρ. Training uses 3.1M instances from 84 sources (human-labelled, LLM-annotated, and synthetic), 40,000 updates, a global batch size of 128, and a 5.12M-draw sampling budget distributed across sources with UniMax, yielding roughly 35% entity, 41% relation, and 23% event examples.

Data hygiene. Synthetic annotations are cross-checked: for most sources a second model (Nemotron-3-Nano-Omni-30B-A3B) re-annotates and gpt-oss-120b adjudicates disagreements; for some synthetic sources up to five models vote, requiring three agreeing votes, with unresolved disagreements removed from the queried schema. Near-duplicates are removed at cosine similarity 0.9 using all-mpnet-base-v2 embeddings, and any training text sharing 80% of its word 5-grams with an evaluation sentence is dropped.

Inference settings. For entities, temperature T minimises the multi-class Brier score and τ maximises F1, both fitted on calibration splits of the training sources; for relations T = 1 and τ is fitted on REBEL with gold entities. No labelled examples from any target dataset are used for training, model selection, or calibration.

Why This Matters

Impact on research. The paper argues against two prevailing assumptions: that flexible schema input alone solves cross-domain transfer, and that generative extraction is the natural route to general-purpose IE. It shows that a small discriminative encoder with a well-designed joint decision layer, mixed-source training, and a reward signal can match or beat 6.7B–11.4B generative baselines at 5–25 times their throughput, and it contributes a reusable fix for a bias that arises whenever null classes are down-weighted in span-based extraction — a setting common well beyond this paper.

Real-world applications:

• Recruitment and job-market analytics — pulling skills, qualifications, and experience from job descriptions, the example the paper opens with.
• Biomedical and scientific literature mining — BioRED and SciER are evaluated, and CalOn's largest entity gains are reported on scientific, biomedical, and measurement text.
• Privacy and compliance scanning — the ai4privacy and spy (legal, medical) benchmarks target personal-information detection in documents.
• Public-health and news event monitoring — SPEED++ mpox and zika, plus M2E2 and CASIE, cover event triggers and roles in news-like text.
• Review analytics — aspect sentiment triplet extraction is handled through the same span-pair formulation without a task-specific head.

Industry relevance. A 0.6B encoder at 120 sentences per second on a single GPU is deployable where 7B–11B generative extractors are not, and a schema passed at inference time means one deployed model can serve many customers with different type systems. Calibrated confidences are what make such a system usable in a human-review pipeline or behind a decision threshold. The 8.0 GB peak memory is the practical trade-off to budget for.

Future Directions

• Multilingual transfer. The authors state that training and evaluation are English-only and that transfer to other languages is untested — an obvious extension given the multilingual backbones and datasets now available.

• Longer mentions and cross-document relations. Candidate spans are capped at 32 tokens and relations are predicted only between spans inside the same input, leaving long mentions and document-level or cross-input links out of scope. The LitBank and MAVEN-ERE conversions already measure within-input span-pair extraction rather than the original document-level task, so closing that gap is a natural next step.

• Better event calibration and single-stage event extraction. Event arguments retain the highest average calibration error (0.39) even after correction, and events are handled by a two-pass pipeline (triggers first, then roles over predicted triggers). Improving argument confidence and unifying the passes are open problems.

• Robustness and annotation bias. Results come from a single seed, and much of the training data is LLM-annotated, so voting and judge checks reduce but do not eliminate shared bias. Multi-seed evaluation and stronger human verification would test how much of the margin depends on that pipeline.

Target Audience

Researchers and engineers working on information extraction who need one model to serve many schemas, and who care about the trade-off between generative and discriminative approaches. It is also useful for practitioners deploying extraction behind review queues or thresholds, since the calibration and efficiency sections address deployment concerns directly, and for anyone training span-based models with a null class who wants a ready-made fix for the resulting probability bias. Readers new to IE will find the setup accessible but should expect the training-objective and calibration sections to require background reading.

Authors’ abstract

Information extraction models often perform well on the datasets they are trained on, but transferring to new domains and annotation guidelines remains challenging. Supporting user-specified types alone does not resolve this problem. We present CalOn, a unified 0.6B-parameter encoder for entity, relation, and event extraction across datasets and schemas. The model jointly encodes text and type definitions, then directly identifies entity spans, relations between spans, and event triggers and arguments. Its extraction decisions consider the requested types together, using their combined evidence to identify relevant candidates before assigning a type. This gives all three tasks a common probabilistic formulation while keeping every extracted mention anchored to the input text. We train on diverse sources, and supplement supervised learning with a reward that encourages correct extractions and penalises incorrect ones. Across 40 benchmarks, CalOn outperforms the evaluated baselines on most benchmarks. It leads all reported relation extraction benchmarks and achieves the highest average F1 for event triggers and arguments with gold triggers. The same model also performs strongly on aspect sentiment triplet extraction and pronoun resolution through its span-pair formulation. A correction for the probability bias introduced during training improves confidence estimates for entities, relations, and event arguments after calibration, with little change in F1.