A systematic review of relation extraction task since the emergence of Transformers

AI summary

Overview

Research area: Natural Language Processing and Information Systems — specifically Relation Extraction (RE), the task of identifying semantic relations between entities in text to produce (subject, relation, object) triples, with connections to the Semantic Web and knowledge graph construction.

Technical level: Intermediate. The paper is a literature review, not a new model, but it assumes familiarity with standard NLP concepts (NER, entity linking, coreference, Transformer-based language models) and with knowledge graph formalisms such as RDF, ontologies, and SPARQL.

Scope in one sentence: A systematic literature review of 34 surveys, 64 datasets, and 104 models published between 2019 and 2024, published as arXiv:2511.03610v2 [cs.CL].

What This Paper Is About

Relation extraction research spans more than three decades, but the arrival of Transformer-based pre-trained language models marked a turning point that reshaped the field. Existing surveys of RE were either outdated (covering pre-2020 work) or narrow in scope, and almost none of them released the annotated corpus behind their analysis.

This paper addresses those gaps by building a large, openly available, systematically collected and finely annotated corpus of RE papers, then analyzing trends, benchmark resources, and the integration of Semantic Web technologies across surveys, datasets, and models.

Key Contributions

1. An updated and detailed overview of RE research, highlighting the transformative impact of large pre-trained language models.
2. An application of the SciLEx framework, described as an extended systematic literature review methodology ensuring transparency, reproducibility, and open availability of results.
3. A fine-grained annotation along more than thirty analytical dimensions across resources, models, and surveys — 32 annotation dimensions in total, 5 of which are derived from PaperWithCode — released openly to support future investigations.
4. Open release of the annotated Zotero bibliography and the SciLEx software as a unified framework for collecting and analysing papers in systematic literature reviews.

Main Findings

• Scale of the collected corpus: Starting from 2,160 automated queries, the authors collected 3,894 papers: 77 survey articles, 250 dataset papers, and 3,567 model papers. Selection criteria C1–C7 reduced this to a consistent set of resources, and after applying impact criteria plus manual verification, 202 papers remained: 34 surveys, 64 datasets, and 104 models.

• Survey landscape is mostly not systematic: Of the 34 curated surveys (44% of the surveys initially collected), 12 are purely qualitative, 13 combine performance results with descriptive analysis, and 5 focus exclusively on benchmarking. Only 7 surveys (20%) apply principles consistent with a Systematic Literature Review.

• Surveys lack shared, reusable annotations: The reviewed systematic reviews describe core subtasks (NER, coreference resolution, entity linking, relation classification) and dimensions such as language, granularity, and domain, but none provide direct access to their annotated corpora; that information must generally be inferred from the text. Most also focus on classical neural architectures with embeddings that are increasingly outdated, and analyse only roughly twenty models and datasets each.

• Dataset granularity skews toward paragraphs: Half of the 64 datasets (33) are designed exclusively for paragraph-level RE, while 11 target the document level. The remaining 20 datasets support both paragraph- and document-level annotations. Restricting RE to individual sentences limits the capture of context-long and interdependent sentential relations.

• Language coverage is dominated by English: Approximately 80% of the datasets are dedicated to English. A growing minority incorporate additional languages such as Arabic, Chinese (in 5 papers), and Spanish (in 4 papers) alongside English.

• Annotation strategies trade scale for quality: Distant supervision is a popular way to build large training sets but introduces label noise. In the corpus, 12 datasets were partially annotated to assess data quality, whereas 29 were entirely manually annotated.

• Wikipedia is the dominant text source: Wikipedia underpins 23 datasets in the collection, typically paired with structured knowledge bases such as DBpedia, YAGO, Wikidata, or Freebase that provide the ground truth for relation labels.

• Relation values are mostly strings: 54 datasets use string-based relations, 13 cover date-related relations, and only 7 focus on extracting numerical values. 28 datasets explicitly use object properties — only 1/3 of the datasets proposed.

• Domain distribution: Half of the datasets contain encyclopedic knowledge and a quarter are news-related. Scientific and biology-related datasets are also well represented, alongside contextual resources such as the conversational ACE dataset and the cybersecurity dataset CASIE, plus multi-domain resources like CROSS-RE and MultiCROSSRE.

• Model publication trends: The 3,567 collected model papers span 2009 to 2024. About 200 papers were published between 2014 and 2017, rising to 400 in 2019, and peaking at over 600 in 2021. Models are rarely published in journals — most appear at conferences such as ACL, EMNLP, IJCAI, or AAAI. Not all models have a DOI, and citation network data could only be retrieved for approximately half of the papers.

• Benchmarking is consolidated but uneven: 18 surveys report performance metrics, but only 2 offer substantial benchmarking depth. Nayak et al. (2021) enriches 4 datasets (NYT11-HRL, TACRED, WebNLG, and SemEval-2010 Task 8) with results for 36 models, while Ye et al. (2022) extends evaluation on the NYT dataset with 11 models. This review consolidates results from PapersWithCode for broader coverage and consistent comparisons.

• Annotation agreement is good: Across annotation dimensions, annotators showed perfect agreement on the type of survey (Survey methodology) and substantial agreement on the pre-trained model used (PTM) and the number of relation types represented by datasets or targeted by models. Moderate agreement was found for use of negative examples, granularity, task, and input type. Agreement was closest on dataset-related data, followed by survey data and then model data.

• Semantic Web integration is a distinguishing axis: The paper compares itself to prior reviews and notes that it jointly analyses models and datasets together with Semantic Web resources, a combination initially proposed by Martinez-Rodriguez et al. (2020), which the subsequent surveys largely dropped.

Methodology in Plain English

The authors used a Python toolbox called SciLEx (Science Literature Exploration), which implements and extends the methodology of Kitchenham and Charters (2007) and Brereton et al. (2007). The workflow was:

1. Automated search. Starting from two keyword sets — an RE set ("Relation extraction", "Relation Classification", "Triplet Extraction", "Slot Filling", "KG-to-Text", "Text-to-data extraction") and a survey set ("Survey", "State-of-the-art", "Review", "Study") — SciLEx generated and executed 2,160 query combinations across nine digital libraries: HAL, Istex, arXiv, DBLP, Semantic Scholar, OpenAlex, Scopus, IEEE Xplore, and Springer Nature.

2. Deduplication and enrichment. Retrieved articles were deduplicated by title and imported into a Zotero library. The corpus was enriched with metadata from PaperWithCode and LinkedPaperWithCode, DOIs via CrossRef, and citation networks via OpenCitations, which supported both impact filtering and citation snowballing.

3. Filtering. Seven inclusion/exclusion criteria (publication period, coherence, language, access, publication process, referenceability, duplicates) were applied, followed by impact criteria: journal impact factor greater than 10, conference ranked A* or A per CORE 2023, at least 1 citation per year since publication for model papers, datasets used at least 2 times by one of the studied model papers, dataset papers extending an accepted dataset paper, and at least 3 citations for survey papers.

4. Annotation. The team refined PaperWithCode tags over four rounds into 32 annotation dimensions taking boolean, quantitative (expressed in powers of 10), or qualitative values, all recorded as Zotero tags. A cross-annotation study covered 10% of each subcorpus (4 datasets, 4 surveys, and 10 models), using 5 annotators who each handled 3 datasets, 2 surveys, and 6 models so that 3 annotators reviewed each paper. Agreement was measured with Fleiss Kappa, treating each possible tag set per dimension as a possible answer. Conflicts were resolved in regular meetings, guidelines were revised, and a single expert annotator then completed the full corpus, spending about 10 minutes on average per paper for more than 37 hours of total work.

Why This Matters

Impact on research. Prior RE surveys were either pre-2020, narrow in scope (grammar checking, temporal relations, annotation practices, general knowledge graph construction), or did not release their underlying annotated data. This review provides a reproducible, openly released reference corpus and a 32-dimension annotation grid, letting other researchers extend or re-cut the analysis instead of re-inferring it from prose. It also shifts the unit of analysis by jointly examining models, datasets, and their interactions rather than presenting them separately.

Real-world applications:

• Populating and maintaining knowledge graphs. Automatic RE is described as a promising mechanism for enriching knowledge bases and knowledge graphs at scale, which are otherwise labour-intensive to maintain and expand.
• Search, indexing, and retrieval. Converting unstructured text into triples of the form (subject, relation, object) enables more effective indexing, search, and retrieval.
• Search engines, recommendation systems, and scientific discovery. The paper cites these as the application areas supported by knowledge graphs such as Freebase, YAGO, DBpedia, and Wikidata.
• Question answering over knowledge graphs. RE-adjacent QA tasks can generate queries, such as SPARQL for RDF knowledge graphs, that are executed against a graph to retrieve answers.
• Bootstrapping data in low-resource settings. Open RE, which produces triples using textual predicates taken directly from text, is described as useful for bootstrapping new knowledge from large corpora, though the resulting data tends to be noisier and more redundant.

Industry relevance. Knowledge graphs are described as critical infrastructures in both academia and industry. Anyone building or curating a knowledge graph needs to know which benchmarks exist, how well they are annotated, what languages and domains they cover, and where the field's evaluation practices are weak — which is exactly what this review quantifies.

Future Directions

• Document- and paragraph-level relation extraction needs more benchmark support. Only 11 of the 64 datasets target the document level, while half are paragraph-only, even though restricting relations to single sentences limits the capture of context-long and interdependent relations.
• Multilingual coverage remains a clear gap. With approximately 80% of datasets dedicated to English, expanding resources for Arabic, Chinese, Spanish, and other languages is an open need. The paper notes an increasing awareness of multilingual information extraction but limited resources today.
• Semantic Web integration is underused. Only 28 datasets explicitly use object properties, accounting for 1/3 of the datasets proposed — leaving room for closer alignment between RE resources and ontology-based representations.
• Systematic review standards could be adopted more broadly. Only 7 of the 34 surveyed reviews follow SLR-consistent principles, and none release their annotated corpora; the authors present their own openly released grid and bibliography as a template others could extend in specific directions, since the underlying corpus has two levels: the complete set of collected papers and a refined, annotated version.
• Benchmark consolidation can be extended. The authors note that their own consolidated benchmark results, built on PaperWithCode, could be built upon further; and they flag that newer work such as Zhao et al. (2024) was published after their corpus was finalized and is more limited in paper annotation.

Target Audience

Researchers and practitioners working at the intersection of NLP and knowledge graphs who want a concrete, up-to-date picture of the main research directions for RE since the ramp-up of language models. The paper also names students and private researchers as intended readers. It is most useful to those who need benchmark inventories, dataset characteristics (granularity, language, domain, annotation strategy), or the released annotation grid and bibliography to build on — rather than to readers looking for a new extraction model.

Authors’ abstract

This article presents a systematic review of relation extraction (RE) research since the advent of Transformer-based models. Using an automated framework to collect and annotate publications, we analyze 34 surveys, 64 datasets, and 104 models published between 2019 and 2024. The review highlights methodological advances, benchmark resources, and the integration of semantic web technologies. By consolidating results across multiple dimensions, the study identifies current trends, limitations, and open challenges, offering researchers and practitioners a comprehensive reference for understanding the evolution and future directions of RE.