GSAP-ERE: Fine-Grained Scholarly Entity and Relation Extraction Focused on Machine Learning

AI summary

Overview

Research area: Natural Language Processing, specifically scholarly information extraction (named entity recognition and relation extraction) for Machine Learning research literature.

Technical level: Intermediate. Readers need familiarity with named entity recognition (NER), relation extraction (RE), fine-tuned pretrained language models, and few-shot prompting of large language models.

Scope: The paper introduces GSAP-ERE, a manually annotated full-text dataset of 10 entity types and 18 relation types for Machine Learning publications, and benchmarks fine-tuned pipeline and joint extraction models against unsupervised LLM prompting on it.

What This Paper Is About

Machine Learning research changes quickly, and papers often report methods, datasets and tasks inconsistently, which makes results hard to reproduce or compare. Existing scholarly extraction datasets tend to use coarse entity labels or cover only abstracts, so they cannot capture the fine-grained dependencies (which model was trained or evaluated on which dataset, which dataset came from which source) that reproducibility analysis requires. The authors build GSAP-ERE, a fine-grained manually annotated corpus covering the full text of 100 ML publications, and use it both to train baseline extraction models and to probe whether large language models can do the job without supervision.

Key Contributions

1. A reusable data model with 10 entity types and 18 relation labels, the relations systematically organized into seven semantic groups covering the creation, usage, property, application and referencing of scholarly entities.
2. A dataset of more than 35K fine-grained relation annotations across 100 scientific publications, with 62,619 annotated entities, connections among ML models, methods, datasets and tasks, links to references and URLs, and coverage of every sentence in the full text including null cases (6,185 sentences with no entities or relations from the data model).
3. Fine-tuned baseline models demonstrating the dataset's utility, using both pipeline and joint modeling approaches, plus reported zero-shot and few-shot results for unsupervised LLM prompting.
4. A comparative evaluation showing the gap between supervised fine-tuned models and unsupervised LLM prompting on this fine-grained task.

Main Findings

• Supervised models dominate LLM prompting. The best fine-tuned model (HGERE trained on GSAP-ERE) reaches NER 80.6% and RE 54.0%, while the best LLM result (Qwen 2.5 72b) reaches NER 44.4% and RE 10.1%. The supervised advantage ranges from 18.6% to 39.8% for NER/NER≈ and from 28.1% to 50% across all four RE settings.
• The joint approach beats the pipeline approach. HGERE outperforms PL-Marker in every setting for both NER and RE. Full comparison: HGERE 80.6 NER / 85.8 NER≈ / 54.0 RE / 59.8 RE≈ / 46.9 RE+ / 51.3 RE+≈, versus PL-Marker 72.6 / 77.7 / 41.4 / 46.2 / 36.3 / 39.9.
• Best supervised results with variance reported. The authors run their best model with 5 random seeds and report NER F1 of 80.6 ± 0.3% and RE+ F1 of 46.9 ± 0.5%.
• LLM performance is marginal across models. Qwen 2.5 72b achieves the best LLM performance in all settings (44.4 NER, 10.1 RE), but its advantage over Qwen 2.5 32b (42.0 NER, 7.2 RE) and LLaMA 3.1 72b (40.5 NER, 6.4 RE) is small, especially for RE.
• The dataset is the largest of its kind. Compared with ScienceIE, SemEval 2018 Task 7, SciERC and SciER, GSAP-ERE has the most entity types (10 versus 3, -, 6, 3), the most relation types (18 versus 2, 6, 7, 9), the most entities (62,619 versus 9,946, 7,505, 8,094, 24,518) and the most relations (35,302 versus 672, 1,583, 4,648, 12,083). It has the highest entity density in relations at 75% (46,680 of entities), compared to 18%, 42%, 77% and 70% for the others.
• Few-shot example selection matters for NER. The best NER prompting configuration is k=10 examples with the similar+diverse selection strategy, achieving 58% F1, about 5% above random selection, with performance declining sharply after 20 examples.
• RE prompting remains weak across shot counts. On the validation set with Qwen2.5-32B and similar+diverse selection, RE≈ scores go from 16.8 (0-shot) to 20.4 (1-shot), 20.5 (2-shot) and 19.9 (5-shot); RE+≈ ranges from 6.5 to 14.4. One-shot and 2-shot perform best with minimal difference.
• Interrater agreement is strong overall but uneven by relation group. The joint annotation agreement after revising entity annotations improves NER agreement from a reported 0.61 to 0.82 macro-F1 under the NER setting and from 0.69 to 0.86 under NER≈. Weighted overall relation agreement is 53.7 (RE+), 58.1 (RE), 56.9 (RE+≈) and 62.8 (RE≈), ranging from 80.1 for Data Properties down to 38.4 for Model Design.
• Supervised inference is far faster. On the 10-document test corpus, LLM inference is 182 times slower than the fine-tuned pretrained language model (4 minutes versus 12 hours and 29 minutes), while training the PLM took 2 hours and 30 minutes for a single run.
• The reported problems motivating the work. The paper cites Raff (2019) on declining reproducibility, Ferrari Dacrema et al. (2019) on poor benchmarking practices, and Pineau et al. (2021), who found only 4% of assessed ML-related publications could be reproduced when the original authors did not respond to clarification requests.

Methodology in Plain English

The authors start from GSAP-NER, an existing corpus of 100 manually annotated full-text ML publications whose 10 entity types fall into three groups: ML model related (MLModel, ModelArchitecture, MLModelGeneric, Method, Task), dataset related (Dataset, DatasetGeneric, DataSource) and miscellaneous (ReferenceLink, URL). Those publications were sampled through a popularity-diversity combined strategy from HuggingFace and arXiv and converted from PDF to plain text with GROBID.

To create relation annotations, the team used INCEpTION with a two-phase annotation-refinement strategy. Two computer-science student annotators first checked and corrected the existing entity annotations, then added relations. Ten of the 100 publications were randomly selected for double annotation by both annotators, and the remaining 90 were split between them. In the refinement phase, two PhD students and two postdoc researchers inspected the alignment of the two annotators, filtered out misalignment cases, extracted misalignment patterns per relation, and re-checked and corrected joint documents showing those patterns.

The 18 relation labels are grouped into seven semantic groups with clear purposes: Model Design (usedFor, architecture, isBasedOn), Task Binding (appliedTo, benchmarkFor), Data Usage (trainedOn, evaluatedOn), Data Provenance (sourcedFrom, transformedFrom, generatedBy), Data Properties (size, hasInstanceType), Peer Relations (coreference, isPartOf, isHyponymOf, isComparedTo) and Referencing (citation, url). Unlike SciER, the scheme retains informal mentions of named entities through the MLModelGeneric and DatasetGeneric types, which the authors argue raises entity density and reduces sparsity for some relation types.

For experiments, they fine-tuned two models using the scibert-scivocab-uncased encoder: PL-Marker (pipeline, with a span classification NER step followed by entity-pair interaction for RE) and HGERE (joint, extending PL-Marker with a hypergraph neural network, using a ternary information flow configuration and an optimized loss-weight scheme). Ten percent of the annotated publications were used for validation and 10% for testing, leaving 80 for training.

The LLM side used a two-stage prompting pipeline mirroring the supervised pipeline: extract entities from a sentence, then form subject-object candidate pairs (n(n-1) pairs for n predicted entities) and classify each relation with a separate prompt. Prompts have five structured sections: Task Introduction, Label Definitions, Step-by-Step Instructions, Few-Shot Examples and Main Input. The authors explored the number of few-shot examples and two selection strategies (random versus a dynamic retrieval-based strategy, either similar or similar+diverse). All examples came from the training set, temperature was set to zero, and few-shot sentence selection used the multi-qa-mpnet-base-cos-v1 model from Sentence-Transformers. Qwen2.5 32B was used for hyperparameter selection on the validation set; final test comparison used Qwen2.5 32B, Qwen2.5 72B and LLaMA 3.1 72b via Ollama quantized versions. Evaluation reports micro F1 under four RE settings: RE+ (correct entity labels and exact span matching), RE (correct relation labels and exact spans, no entity type restriction), RE+≈ (exact labels but partially overlapping spans) and RE≈ (exact relation label and overlapping spans with no further requirement). Experiments ran on Ubuntu 22.04.4 LTS with 2× Intel Xeon 2.1 GHz CPUs (48 cores, 96 threads), 1.4 TB RAM, 8 GPUs (4× RTX 2080 Ti with 11 GB and 4× A40 with 48 GB), and over 40 TB of multi-tier storage.

Why This Matters

Impact on research. The dataset targets a documented reproducibility problem in ML: without structured, fine-grained metadata about which models were trained or evaluated on which datasets and derived from which sources, systematic reproducibility monitoring is impractical. GSAP-ERE provides a manually verified ground truth for building extraction systems that support knowledge graph construction, research artifact tracing and computational reproducibility monitoring at scale. Its inclusion of null cases and informal mentions makes it a harder, more realistic test bed than abstract-only corpora.

Real-world applications:

• Scholarly knowledge graph construction connecting ML models, datasets, tasks and methods.
• Large-scale monitoring of computational reproducibility in AI research.
• Academic question answering over research literature.
• Tracking dataset provenance and properties to support FAIR-style data quality and bias assessment, and identifying sources of research artifacts.

Industry relevance. Teams maintaining model and dataset registries, benchmark leaderboards, or literature monitoring platforms could use such extraction to identify which resources are used for which tasks and how they relate. The runtime finding matters operationally: the fine-tuned model was 182 times faster than LLM inference on the test corpus, a meaningful difference for anyone extracting metadata across a large publication corpus.

Future Directions

• Extend beyond ML and applied ML. The authors limit the current corpus to those fields to simplify annotation without requiring domain experts, and state they will transfer the data curation lessons when expanding the domain.
• Move from sentence level to document level. The current corpus covers only sentence-level annotations; the authors argue a document-level corpus would benefit scholarly IE on long texts.
• Improve LLM prompting for fine-grained scholarly IE. The large performance gap and weak RE results (best LLM RE 10.1%) indicate substantial remaining research is needed before unsupervised prompting is viable for this task.
• Address uneven relation agreement. Agreement varies strongly by semantic group, with Model Design lowest at 38.4 (RE+) and Data Properties highest at 80.1, which the authors partly attribute to poor reporting practices in publications.

Target Audience

Researchers in natural language processing and scholarly information extraction who work on entity and relation extraction, dataset construction and annotation methodology. It is also relevant to ML reproducibility researchers and research infrastructure developers building knowledge graphs or monitoring systems over scientific literature, and to practitioners comparing fine-tuned models against LLM prompting for domain-specific extraction.

Authors’ abstract

Research in Machine Learning (ML) and AI evolves rapidly. Information Extraction (IE) from scientific publications enables to identify information about research concepts and resources on a large scale and therefore is a pathway to improve understanding and reproducibility of ML-related research. To extract and connect fine-grained information in ML-related research, e.g. method training and data usage, we introduce GSAP-ERE. It is a manually curated fine-grained dataset with 10 entity types and 18 semantically categorized relation types, containing mentions of 63K entities and 35K relations from the full text of 100 ML publications. We show that our dataset enables fine-tuned models to automatically extract information relevant for downstream tasks ranging from knowledge graph (KG) construction, to monitoring the computational reproducibility of AI research at scale. Additionally, we use our dataset as a test suite to explore prompting strategies for IE using Large Language Models (LLM). We observe that the performance of state-of-the-art LLM prompting methods is largely outperformed by our best fine-tuned baseline model (NER: 80.6%, RE: 54.0% for the fine-tuned model vs. NER: 44.4%, RE: 10.1% for the LLM). This disparity of performance between supervised models and unsupervised usage of LLMs suggests datasets like GSAP-ERE are needed to advance research in the domain of scholarly information extraction.