Exploring LLMs for Scientific Information Extraction Using The SciEx Framework

AI summary

Overview

Research area: Automated scientific information extraction from research publications using large language models (LLMs), spanning the medical and environmental sciences (virus decay, UV disinfection, and water treatment).

Technical level: Advanced. The paper assumes familiarity with retrieval-augmented generation (RAG), prompt engineering, schema-constrained extraction, multi-modal models, and information extraction metrics.

Scope (one sentence): The paper introduces SciEx, a modular, prompt-driven, RAG-based framework that parses scientific PDFs, retrieves and extracts schema-conforming data from text, tables, and figures, and aggregates results across papers, then evaluates it with Gemini-2.5-Flash and GPT-4o on three scientific datasets including a newly annotated 143-paper benchmark.

What This Paper Is About

Scientific knowledge is scattered across text, tables, and figures in long PDFs, and the same concept can appear under different names and units, which makes it hard to compile structured data at scale. Despite the general success of LLMs, existing extraction pipelines struggle with long documents, multi-modal content, inconsistent terminology, and schemas that change faster than systems can be retrained. The paper's goal is to build and evaluate a flexible framework, SciEx, that decouples PDF parsing, retrieval, extraction, and aggregation so researchers can swap models, prompts, and reasoning strategies on demand, and to measure how well current LLMs actually perform on this task.

Key Contributions

1. A prompt-driven, retrieval-augmented extraction framework that unifies text, tables, and figures from research publications into structured representations, presented as a methodology for adaptable and modular extraction rather than a fixed system.
2. A modular and composable architecture that decouples core components (PDF parsing, retrieval, extraction, and aggregation) so each can be independently replaced, upgraded, or extended, supporting rapid experimentation with different LLMs, prompting strategies, and retrieval configurations.
3. A new scientific information extraction dataset of 143 papers from the medical and environmental sciences, manually annotated by PhD students in those fields, plus three evaluation datasets spanning Virus Decay (VD), Ultraviolet (UV), and Coagulation-Flocculation-Sedimentation (CFS).
4. An empirical evaluation and error analysis showing that existing LLMs degrade when applied to new scientific domains, with a taxonomy of failure modes (parsing quality, table structure, dense figures, and cross-referencing).

Main Findings

• Modest overall extraction quality. Across all three datasets and both models, F1-scores range from 0.175 to 0.380 and precision from 0.169 to 0.333, meaning a large share of extracted records do not match ground truth.

• Recall generally exceeds precision. The authors report that recall is typically higher than precision, indicating extractions are often relevant but that a substantial number of unwanted points are also extracted.

• GPT-4o outperforms Gemini-2.5-Flash. GPT-4o achieves higher average precision (0.26 vs. 0.22), recall (0.48 vs. 0.37), and F1-score (0.29 vs. 0.27), with the largest gains on the visually dominated UV and VD datasets, reflecting stronger multi-modal comprehension. Gemini-2.5-Flash tends to omit information, retrieving correct fragments but failing to fully populate schema fields, which lowers its recall.

• Simpler datasets are easier. Both LLMs perform better on UV and VD, where most fields come directly from a single figure or caption with minimal cross-referencing. The more complex CFS dataset, which requires integrating across multiple tables and figures, shows lower precision and F1-scores due to missing rows and difficulty reconciling distributed experimental conditions.

• Per-dataset numbers (Gemini-2.5-Flash). CFS: precision 0.169, recall 0.273, F1 0.175, accuracy 0.507. UV: precision 0.199, recall 0.468, F1 0.237, accuracy 0.329. VD: precision 0.284, recall 0.382, F1 0.297, accuracy 0.556.

• Per-dataset numbers (GPT-4o). CFS: precision 0.241, recall 0.355, F1 0.248, accuracy 0.512. UV: precision 0.279, recall 0.609, F1 0.331, accuracy 0.467. VD: precision 0.333, recall 0.476, F1 0.380, accuracy 0.580.

• Field-level accuracy is moderate. The text states both models achieve moderate accuracy in the 0.5–0.6 range, suggesting that once a record is correctly localized, field-level extraction is generally reliable. The reported accuracy values in Table 1 range from 0.329 (Gemini-2.5-Flash, UV) to 0.580 (GPT-4o, VD).

• Four recurring error types. The manual error analysis identifies: (1) parsing and document quality, where older or poorly formatted PDFs have degraded visual quality, low-resolution graphs, truncated axes, or illegible scales; (2) text extraction and cross-sentence reasoning, where entities appearing close together (for example SARS-CoV-2 and Influenza A (H1N1)) cause misattribution of conditions or outcomes; (3) table extraction and structural variability, where merged cells, nested headers, and variables spread across multiple tables cause schema mismatches and missing values; and (4) figure interpretation and numeric accuracy, where implicit or inconsistent log/linear scales, dense clusters, and missing legends, titles, or captions reduce precision and recall.

• Cross-referencing failures. When figures, tables, and text reference each other, the models struggle to establish correct correspondences; the appendix gives an example where mucus-associated data is mistakenly assigned to the 3h time point instead of 2h.

• Not deployment-ready. Even with extensive prompt optimization, retrieval augmentation, and modular processing, the authors conclude the best results fall short of the precision and completeness required for production-level knowledge extraction.

Methodology in Plain English

SciEx treats each paper independently and then merges the results, conceptually like a map-reduce operation: a per-paper "map" stage extracts data using the same logic on every document, and a "reduce" stage combines the outputs.

The pipeline has four parts. First, a PDF Extractor (implemented with Docling, an open-source document conversion toolkit that performs fine-grained layout analysis and structural recognition) parses each PDF into semantically coherent text chunks, and extracts charts, diagrams, and tables as individual PNG images. A vision-language model then performs binary classification to keep scientific figures (plots, charts, diagrams) and drop non-scientific illustrations such as logos or decorative images. Each kept figure is paired with a caption, either extracted from the document or generated by the VLM, and the VLM parses axis labels, legends, and data points (bars, curves, markers) into structured JSON. Full-page images of every page are also stored. Text segments are embedded into a vector database, and figures are indexed with metadata linking them to source documents, forming a contextualized multi-modal knowledge base.

Second, a Schema Module defines what to extract. Users can either supply an explicit schema (attributes such as virus name, temperature, humidity, and measurement units, with data types such as string, float, or integer) or give a natural language instruction, in which case an LLM generates the structured schema.

Third, a Retrieval–Extraction–Verification (REV) module runs a closed loop: retrieve the top-5 most relevant chunks each round by vector-based semantic search across text, table entries, and figure-derived JSON; pass the evidence to an LLM for schema-guided extraction, constrained so each field matches the expected format and type; and attach provenance metadata (document identifiers, chunk indices, figure references) for traceability. Self-verification then flags missing, uncertain, or low-confidence fields and triggers targeted follow-up queries, repeating until there are no missing fields, a confidence threshold is reached, or a predefined number of rounds completes.

Fourth, an Aggregation and Resolution module groups records from different documents by shared entities or experimental conditions, merges fields, standardizes units using schema-defined normalization rules (for example, converting Fahrenheit to Celsius), and harmonizes categorical attributes through controlled vocabularies. An LLM-based canonicalization step maps lexical variants (for example "temp." and "temperature") to schema terms. Conflicting values are handled with cross-model ensembling and consistency voting, plus deterministic checks such as numeric range constraints and unit compatibility. The system also enforces three integrity conditions: every value must be traceable to its source context, list-type fields must contain complete and non-redundant value sets, and duplicate or contradictory entries in the same field are removed. If information is genuinely absent, the field is explicitly labeled null. The framework is model-agnostic and also supports automatic prompt optimization through tools such as DSPy.

Datasets used. The Virus Decay (VD) dataset covers how environmental factors and surrounding medium influence virus viability and infectivity (no dataset size is reported for VD in the paper). The UV dataset includes information used to calculate 224 UV disinfection rate constants for 59 viruses from 105 publications, all reporting UV disinfection of viruses in liquid suspension. The CFS dataset includes 1,624 virus LRVs across 46 viruses from 43 eligible papers, all reporting coagulation-flocculation-sedimentation reduction of viruses in impure waters, with 98 variables relating to water quality, process parameters, and virus reduction. Evaluation uses precision, recall, F1-score, and accuracy, with row-level matching: because each ground-truth paper entry has multiple fields defining a unique record, a bipartite matching algorithm maps ground-truth rows to extracted rows based on field-level similarity after first identifying the fields that distinguish rows.

Why This Matters

Impact on research. The paper provides concrete evidence that general-purpose LLM capabilities do not automatically transfer to fine-grained scientific extraction. It gives a reusable architectural blueprint (parsing, retrieval, extraction, verification, aggregation) and a domain-annotated benchmark of 143 papers that other researchers can build on, while documenting exactly where pipelines break: document quality, table structure, dense multi-panel figures, and cross-reference resolution.

Real-world applications grounded in the paper's datasets:

• Water treatment and disinfection planning: compiling UV disinfection rate constants across 59 viruses and 105 publications to support reactor design and validation.
• Drinking water safety assessment: aggregating 1,624 virus log removal values and 98 water quality and process variables from 43 CFS studies to compare treatment performance.
• Environmental persistence and public health risk modeling: assembling virus decay data on how temperature, humidity, and surrounding medium affect viability and infectivity.
• Rapid literature synthesis under changing requirements: using the schema module and on-demand extraction to re-target an existing corpus when the desired data schema changes, without re-architecting or fine-tuning a system.

Industry relevance. Any organization that must convert large scientific or technical document collections into structured databases, such as water utilities, environmental and public health agencies, regulatory bodies, pharmaceutical and materials R&D groups, and vendors building literature-review or evidence-synthesis products, can use the modular design as a reference architecture. The paper's own assessment is that current results are not yet reliable for large-scale deployment, so the near-term value is in pipeline design and benchmarking rather than turnkey automation. The work was supported in part by US National Science Foundation grants DBI-2412389, CCF-1918770, CCF-1918770, and IIS-2312794.

Future Directions

1. More robust domain adaptation and calibration to improve generalization to unseen scientific areas, since the paper shows performance degradation when LLMs are applied to new domains.
2. Better cross-modal reasoning to integrate textual, tabular, and visual information more coherently, including layout-aware parsing, hierarchical table modeling, and table-to-graph reasoning to link semantically related cells across tables.
3. Standardized datasets and evaluation protocols that more accurately reflect real-world scientific extraction tasks.
4. Improved preprocessing and figure interpretation, such as resolution enhancement, adaptive binarization, OCR refinement, scale and symbol recognition, and contextual reasoning from surrounding text to recover missing legends and captions.

Target Audience

Researchers and practitioners in natural language processing, information extraction, and scientific literature mining; domain scientists in environmental engineering, water treatment, virology, and public health who need to synthesize data across large publication corpora; and engineers building or evaluating LLM-based document-processing pipelines. Readers with a background in RAG, prompt engineering, and multi-modal models will benefit most, though the framework description and error analysis are accessible to domain experts who want to understand the practical limits of current LLM extraction tools.

Authors’ abstract

Large language models (LLMs) are increasingly touted as powerful tools for automating scientific information extraction. However, existing methods and tools often struggle with the realities of scientific literature: long-context documents, multi-modal content, and reconciling varied and inconsistent fine-grained information across multiple publications into standardized formats. These challenges are further compounded when the desired data schema or extraction ontology changes rapidly, making it difficult to re-architect or fine-tune existing systems. We present SciEx, a modular and composable framework that decouples key components including PDF parsing, multi-modal retrieval, extraction, and aggregation. This design streamlines on-demand data extraction while enabling extensibility and flexible integration of new models, prompting strategies, and reasoning mechanisms. We evaluate SciEx on datasets spanning three scientific topics for its ability to extract fine-grained information accurately and consistently. Our findings provide practical insights into both the strengths and limitations of current LLM-based pipelines.