BioPIE: A Biomedical Protocol Information Extraction Dataset for Experiment Understanding

AI summary

Overview

Research area: Artificial Intelligence / Natural Language Processing, specifically information extraction (IE) and knowledge-graph construction for biomedical experimental protocols.

Technical level: Intermediate. The paper assumes familiarity with named entity recognition (NER), relation extraction (RE), knowledge graphs (KGs), and large language model (LLM) prompting and fine-tuning, but all methods used are established standards in the field.

Scope: This paper introduces BioPIE, a dataset of 344 in-domain and 45 out-of-domain sub-protocols annotated with 34 entity types, 21 relation types, 10,982 entities and 8,848 relations, together with benchmarks of supervised and LLM-based information extraction methods and a knowledge-graph-augmented question answering system.

What This Paper Is About

Automated understanding of biomedical experiments is hard for two reasons the authors name explicitly: HID (high information density), where a single protocol sentence packs in volumes, temperatures, timings, buffer compositions and instrument settings that must each be tied to the right operation, and MSR (multi-step reasoning), where answering a question requires combining information spread across multiple distinct steps or facts.

Existing IE datasets fall into a trade-off: general scientific datasets cover many sub-fields but lack biomedical detail, while biomedical datasets center on molecular and pharmacological entities (proteins, chemicals, drugs, diseases) at a granularity too coarse for procedural reasoning. The goal of BioPIE is to fill that gap with a procedure-centric dataset that captures experimental entities, actions and relations richly enough to support experimental understanding.

Key Contributions

1. The BioPIE dataset. An information extraction dataset for understanding complex biomedical experiments, designed to cover both the HID and MSR aspects of protocol language. It defines 34 entity types (actions, materials, laboratory instruments, biological samples, and parameters such as time, temperature and force) and 21 relation types (including is_object_of, have_parameter, use_device, contain, and next_step).

2. A systematic benchmark of IE algorithms on BioPIE. The authors evaluate supervised models (PL-Marker and HGERE, each with scibert and biobert encoders) and LLMs across zero-shot, few-shot, and LoRA settings, each combined with pipeline (separate NER and RE) and joint (ERE) frameworks, producing six LLM-based configurations.

3. A knowledge-graph-augmented QA system as downstream validation. A filter-planner-retriever-reasoner pipeline is implemented to quantitatively show that BioPIE improves question answering, particularly on question sets built to test HID and MSR.

4. A demonstrated generalization setting. The dataset is explicitly split into in-domain (ID) and out-of-domain (OOD) evaluation, where OOD protocols come from biomedical sub-fields not represented in the ID set.

Main Findings

• BioPIE is relationally dense. It contains 1,916 sentences, 10,982 entities and 8,848 relations, averaging 4.62 relations per sentence, compared to 0.68 for ACE2005, 1.76 for SciERC, and 1.33 for ChemProt. It has 34 entity types and 21 relation types versus 7/6 (ACE2005), 6/7 (SciERC) and 3/11 (ChemProt).

• Annotation agreement: Cohen's kappa is 79.20% for entity annotation and 68.26% for relation annotation, a level the authors describe as comparable to existing literature.

• Supervised baselines: Using scibert, PL-Marker achieves the best ID performance (82.55 Rel, 74.52 Rel+, 87.88 RE). HGERE shows stronger OOD robustness (74.58 NER, 70.49 Rel, 52.41 Rel+). ID-to-OOD degradation is larger for NER (about 13) than for RE (10–12), indicating unseen entities are harder to recognize than relations.

• Encoder sensitivity: Substituting scibert with biobert causes a catastrophic OOD collapse for PL-Marker (Rel drops to 23.51), while HGERE stays robust at 69.48 Rel — suggesting HGERE's OOD advantage is architectural rather than a byproduct of the pretrained encoder.

• Zero-shot LLMs are variable. GPT-5 achieves the most balanced pipeline performance on ID data (57.14 NER, 41.14 Rel+); Claude-4.5-Opus has strong NER but weaker RE; Llama-4 performs poorly across most RE metrics; Qwen-max has reasonable NER but limited RE. Across all models, pipeline extraction outperforms joint extraction.

• Few-shot helps substantially. Claude-4.5-Opus with pipeline extraction reaches 85.18 NER and 63.47 Rel+ on ID. Gains from in-context learning are generally larger on ID than OOD. Most LLMs peak with roughly 5–15 demonstrations, after which returns diminish or turn negative.

• LoRA-tuned small models are competitive. Llama-3-8B with pipeline extraction achieves 86.33 NER and 68.13 Rel+ on ID, approaching supervised performance. After fine-tuning, the pipeline–joint gap narrows considerably.

• Data scaling saturates for RE. For PL-Marker, Rel+ rises from 47.09 (0.1 of training data) to 59.79 (0.2), 69.24 (0.5), 73.92 (0.9) and 74.52 (1.0), while RE gains slow and NER continues only moderate improvement.

• QA results: The full system achieves 72.99% on the test set, 83.33% on HID questions, and 74.80% on MSR questions — the best overall across all compared systems. Comparisons include LLM only (14.74 / 16.09 / 15.45), LLM LoRA (12.44 / 11.49 / 9.76), BM25 (63.72 / 65.52 / 52.03), LaBSE (55.60 / 63.22 / 53.66), Emb-3-large (61.52 / 66.09 / 56.10), Emb-v4 (59.14 / 67.24 / 54.47), GRAG (8.83 / 10.34 / 7.32), GRAG LoRA (22.77 / 17.82 / 16.26), and ToG (69.90 / 75.29 / 69.92).

• Ablations confirm both input types matter. Removing graph-based knowledge ("Ours w/o Graph") degrades performance (55.69 / 66.67 / 60.16). Holding the KG fixed, ToG (69.90%) outperforms BM25 (63.72%), and the full reasoning planner (72.99%) outperforms the planner ablation (72.29%, with an MSR drop to 62.60%).

• Schema matters more than pipeline. Under the identical QA pipeline, the BioPIE schema (72.99% on Test) outperforms SciERC (62.40%) and ChemProt (64.70%), indicating that fine-grained, procedure-oriented knowledge representation is critical.

Methodology in Plain English

The authors first collected protocols from peer-reviewed biology research articles and used Qwen-Max to clean and normalize them into stepwise imperative sentences. From this collection they selected 344 sub-protocols covering four major domains — genetic manipulation, molecular interactions, physiological signal detection, and regenerative medicine — as the in-domain set, and a separate 45 sub-protocols as the out-of-domain set. The OOD protocols cover experiment types absent from the ID set, such as plant-based biopharmaceuticals, functional imaging, and epidemiological analysis (OOD protocols are excluded from the appendix in the truncated content provided).

Annotation used two annotators with graduate-level backgrounds in computer science and biomedical research, working in independent parallel with no access to each other's results, followed by reconciliation by a lead annotator to produce gold-standard labels. The dataset is split into training, development and ID test sets at a 7:1:2 ratio.

The annotation scheme was designed to be procedure-centric rather than concept-centric, deliberately avoiding narrowly specialized biomedical categories so that it applies consistently across cell culture, microscopy, sequencing preparation, and biomaterial fabrication. It was initially designed with reference to the EXACT2 and OBI ontologies; the authors map EXACT2's action participants to their entity types, its quantitative parameter slots to their temperature, time, volume, concentration, speed, force, mass and length types, and its relation slots (has_object, has_instrument, has_reagent, has_parameter) to their is_object_of, use_device, use_reagent and have_parameter relations. They add entity types (software, data, position, blend) and relation types (during, for_each, based_on, in_condition_of, is_goal_of) not prominently covered by those ontologies.

For benchmarking, IE is defined at the sentence level: NER identifies entity spans with types, and RE predicts a relation label for each ordered entity pair within a sentence, with a NULL label forcing LLMs to explicitly declare non-relational pairs. For fair evaluation, NULL predictions are strictly excluded from metric calculations. Metrics are span-level NER, boundary-level Rel, strict Rel+ (requiring correct entity types), and RE with gold entities (relevant only to pipeline methods). Few-shot prompting uses a sentence retriever with text-embedding-3-large to select up to 20 candidate demonstrations and pick the number maximizing Rel+ on validation; prompts are adapted from Chat IE with component label definitions added. The random seed is set to zero.

For the QA system, the pipeline has four parts: a filter scoring sentence–graph pairs with a hybrid relevance function combining semantic similarity (implemented as BM25) and entity overlap with the query; a planner that uses an LLM to generate an abstract relation sequence; a retriever that grounds those abstract paths in the knowledge graph to find entity trajectories; and a reasoner that combines textual evidence with instantiated graph descriptions to produce the answer. QA data is split into 1983 training, 159 validation and 1133 test items, with MSR and HID question sets carved out of the test set. Accuracy is measured by whether the answer appears in model output, initially via substring matching. Experiments use Llama-3-8B, HGERE as the IE method, and prompts adapted from RoG.

Why This Matters

Impact on research. BioPIE supplies the missing procedural granularity in biomedical IE: prior datasets either cover general science broadly without biomedical reagents and devices, or focus on molecular and pharmacological entities without step-level operational detail. The work shows that a procedure-centric schema, not just a larger model, improves downstream reasoning — the BioPIE schema beat SciERC and ChemProt under an identical QA pipeline.

Real-world applications:

• Laboratory automation. The dataset can act as a structured human-robot interface, bridging human-readable protocols and robotic execution scripts.
• Workflow verification. It supports local parameter consistency and constraint checking in experimental procedures.
• AI-assisted experimental planning. It provides a reusable semantic reference for modular protocol composition and conditional adaptation, informing LLM-based planners such as ReAct.
• Cross-disciplinary collaboration and communication. By formalizing protocols into structured representations, it helps analyze intricate experimental logic and facilitates intent understanding.

Industry relevance. The authors frame BioPIE as a decision-making reference for LLM-based planners that integrate AI assistants with robotic execution systems in automated laboratories. External support listed includes the National Natural Science Foundation of China (Grant No. 52475001) and Linkerbot Co., Ltd., which provided the dexterous robotic hand used in the study — indicating a robotics/automation angle to the work.

Future Directions

• Reducing normalization artifacts. The authors acknowledge a limitation: using an LLM to normalize protocol text may introduce misalignments in step references, for example shifting a reference to the product of an earlier step to a later step, which can affect fine-grained step-level grounding and temporal dependency annotation.

• Extending beyond the current scope. BioPIE is bound to foundational laboratory explorations established between 2020 and 2025, covers wet-lab workflows involving physical interactions among reagents and materials, and excludes bioinformatics (dry-lab) analyses and human subject groups. The authors state that generalizability across all peripheral biomedical disciplines cannot be guaranteed.

• Improving OOD generalization. Unseen entities proved harder to recognize than relations, and one supervised model collapsed under an encoder swap on OOD data. Closing this gap in protocol-centric IE is an open challenge the benchmark exposes.

• Exploiting the full annotation scheme. The appendix notes that recovering investigation-level information — such as dose-control-readout design, donor-to-recipient provenance, and genetic perturbation mechanisms — is possible from the fine-grained annotations, suggesting further applications beyond QA.

Target Audience

Researchers and engineers working on information extraction, scientific NLP, and biomedical text mining; developers building knowledge-graph-augmented question answering systems; and groups working on laboratory automation, protocol digitization, or LLM-based planning for robotic execution. It is also relevant to dataset builders interested in procedure-centric, cross-domain annotation schemes and to those evaluating the trade-offs between supervised models and LLMs under zero-shot, few-shot and LoRA settings.

Note: A safety warning accompanies the data. Reproducing the biomedical experiments described in BioPIE must only be carried out under direct supervision of qualified domain experts, as many procedures involve significant safety hazards. The protocols are provided strictly for reference and are not standalone or executable experimental instructions.

Authors’ abstract

Understanding biomedical experiments provides a foundation for downstream tasks, e.g., laboratory automation, and facilitates effective cross-disciplinary communication. Two challenges, High Information Density (HID) and Multi-Step Reasoning (MSR), pose unique difficulties for precise automatic experimental understanding. Extracting structured knowledge, e.g., Knowledge Graphs (KGs), is an effective approach to address the HID and MSR. However, existing biomedical datasets for structured knowledge Information Extraction (IE) are limited to a general or coarse-grained level, hindering fine-grained experimental understanding. To address this gap, we introduce Biomedical Protocol Information Extraction Dataset (BioPIE), a dataset providing procedure-centric KGs that captures entities, actions, and relations at a scale sufficient for reasoning across biomedical protocols. We evaluate both supervised and LLM-based IE methods on BioPIE to verify its effectiveness, and implement a biomedical question answering system to provide a quantitative illustration of BioPIE's effectiveness for downstream understanding tasks. The experimental results demonstrate improved understanding performance on both the HID and MSR question sets.