Agentic schema-guided extraction of materials process knowledge from scientific literature

AI summary

Overview

Research area: Natural language processing for scientific information extraction, specifically materials-science process knowledge, LLM agentic workflows, and knowledge-graph population.

Technical level: Advanced. The paper assumes familiarity with large language models, schema-constrained generation, named-entity normalization, and exact-match evaluation.

Scope: The paper introduces and evaluates SciKGExtract, a schema-guided agentic framework that extracts structured experimental process records from atomic layer deposition (ALD) literature, with PubChem chemical normalization and rubric-based reflection as quality controls.

What This Paper Is About

Materials papers describe experiments in heterogeneous ways — one process may be scattered across methods, results, tables and figures; the same chemical may appear as a formula, systematic name or abbreviation; and numbers only mean something when tied to the right process, quantity kind and unit. This makes materials procedures, chemicals and measurements hard to aggregate computationally. The paper's goal is to build and test a pipeline that converts ALD literature into reusable, machine-actionable experimental records, and to measure how much chemical canonicalization and agentic verification actually help.

Key Contributions

1. SciKGExtract, a modular agentic workflow. A configuration-driven orchestration of four components — Extract & Normalize, Reflect, Feedback and KG Integration — that unifies schema-guided extraction, chemical normalization, rubric-based evaluation and feedback-driven refinement, with records prepared for knowledge-graph integration such as the Open Research Knowledge Graph.

2. A systematic comparison of refinement strategies across models. Direct extraction, self-refinement, multi-judge refinement and evaluator–critic debate are compared across GPT-4o, Llama 3.3 70B, Mistral Large, Gemma 4 26B and Ministral 3B, on 176 ALD papers (162 ZnO, 14 IGZO), plus a separate controlled full-schema sweep over GPT-4o, Mistral Large, Qwen3 30B and Qwen3 235B.

3. An expert-curated full-schema gold standard for materials extraction. Eight thermal-ALD ZnO process instances and five plasma-enhanced-ALD IGZO process instances from two representative papers, annotated against a target of 65 leaf properties and 155 QUDT-grounded quantity nodes — moving evaluation beyond entity recovery to process segmentation, nested property assignment and quantitative reconstruction.

4. Evidence that more agents do not reliably mean better extraction, plus transfer and cost analysis. The paper shows non-monotonic returns from additional judges, tests the same orchestration on the BioRED and BC5CDR biomedical benchmarks, and reports per-request provider costs to characterize quality–cost trade-offs.

Main Findings

• PubChem normalization improves exact-match F1 for every tested model in both material systems. For ZnO, Llama 3.3 70B with debate refinement reaches precision 0.780, recall 0.831 and F1 0.805. The normalization gain is largest for the smallest model: Ministral 3B rises from 0.473 to 0.673 F1 in its best configuration.

• IGZO is far harder than ZnO, even after normalization. The best IGZO normalized exact F1 is 0.344 (Llama 3.3 70B, two-judge refinement), followed by 0.334 (Ministral 3B, three judges) and 0.302 (Mistral Large, three judges). The authors attribute the gap to multicomponent supercycle structures — multiple oxide subcycles, supercycle ratios and composition-dependent properties — not just chemical naming variation.

• Agentic refinement helps on ZnO but is not monotonic. For Llama 3.3 70B on ZnO, normalized F1 rises from 0.591 (direct) to 0.617 (self-refinement) to 0.724 (two judges) to 0.805 (debate), a 0.214 absolute gain. For IGZO the same model goes 0.277 (direct) → 0.298 (self-refinement) → 0.344 (two judges), but debate-self reaches only 0.306 and debate-cross 0.331 — both below the simpler two-judge workflow.

• Full-schema extraction against expert annotations exposes a different error regime. Using the complete 65-leaf-property target, reflection improves GPT-4o on ZnO from 0.41 to 0.44 F1, Mistral Large from 0.51 to 0.53, and Qwen3 30B from 0.40 to 0.50; Qwen3 235B is essentially unchanged. On IGZO, Mistral Large improves from 0.20 to 0.32 and Qwen3 30B from 0.41 to 0.46, while GPT-4o drops from 0.48 to 0.45 and Qwen3 235B stays at 0.29.

• Over-generation, not just omission, is a failure mode. Expert IGZO annotations contain on average 61.8 populated properties per process, whereas one-shot model outputs contain approximately 75; for ZnO the expert average is 35.5 versus approximately 41. Models also differ in process segmentation — the expert annotation has eight ZnO process instances, and the tested one-shot systems recover different subsets of them.

• The same orchestration transfers to biomedical benchmarks by swapping the schema. On BioRED, Claude Sonnet 4.6 with cross-model debate reaches precision 0.3145, recall 0.2580 and exact F1 0.2834. On BC5CDR, Claude Sonnet 4.6 with two-judge refinement reaches precision 0.6985, recall 0.7542 and exact F1 0.7253.

• Quality is not proportional to cost. Average extractor request cost ranges from approximately $0.00061 for Gemma 4 26B and $0.00139 for Llama 3.3 70B to $0.017 for GPT-4o and $0.020 for Claude Sonnet 4.6; Mistral Large averages approximately $0.00424 and Ministral 3B approximately $0.00373. Llama 3.3 70B produces the best normalized ZnO result while remaining among the lowest-cost extractors.

• Evaluation design matters. The authors argue exact precision, recall and F1 should be primary metrics for outputs intended for structured reuse, and that embedding-based similarity is risky for numerical values, chemical identifiers and process assignments.

Methodology in Plain English

The researchers treat extraction as a controlled pipeline rather than a single model call. A semantic schema for ALD experiments — 65 leaf properties plus 155 QUDT-grounded quantity nodes — defines exactly what the output must look like, and is supplied to the model in JSON form alongside typed data models used to parse and validate generated records.

An extraction agent reads the paper (PDFs converted to Markdown with Docling) and produces typed process records. Selected chemical values are then resolved to PubChem identifiers through a sequence that tries deterministic API and local synonym lookups first, with an optional language-model reformulation step for unresolved forms; the original surface string is preserved for provenance rather than replaced.

A reflection stage then judges each extracted record against the source article and the schema on correctness and completeness rubrics — using one judge, several independent judges, or an evaluator–critic pair. Ratings and rationales are converted by a feedback agent into revision instructions that go back to the extractor, forming an extract → reflect → feedback → refine loop. Only records passing the configured validation, normalization and reflection steps are prepared for graph integration.

Evaluation draws on 176 AtomicLimits-linked papers for the core three-property task (deposited material, precursor, co-reactant), a separate expert-annotated full-schema experiment on two papers, transfer tests on BioRED and BC5CDR, and logged provider cost and token usage per request. The authors note that model APIs, versions and pricing change rapidly, so cost figures characterize the executed configurations rather than permanent rankings.

Why This Matters

Materials knowledge bases only work if records from different papers can be compared, and that requires correct entity identity, correct process assignment and correct quantitative attachment — not text that merely looks plausible. This paper shows that chemical canonicalization and agentic verification address distinct error sources and are complementary, and that scaling up the number of agents is not a substitute for evaluating whether feedback targets errors that matter.

Real-world applications:

• Populating FAIR, machine-actionable materials infrastructures such as NOMAD, FAIRmat, the U.S. Materials Genome Initiative and NFDI-MatWerk with reusable experimental records.
• Building and curating knowledge graphs of thin-film deposition processes, where process identity, supercycle ordering and composition-dependent measurements determine downstream property prediction.
• Semiconductor and display manufacturing research, since ZnO and IGZO ALD are established thin-film systems relevant to transparent electronics and device stacks.
• Reusable text-mining pipelines for other scientific domains — demonstrated by schema swaps that carry the same orchestration to biomedical relation extraction on BioRED and BC5CDR.

Industry relevance: Companies and labs that maintain internal materials databases face the same bottleneck of converting heterogeneous internal literature into structured, queryable form. The finding that a relatively inexpensive extractor combined with canonicalization and selective verification can outperform uniformly calling the most expensive model has direct cost implications for anyone running extraction at scale.

Future Directions

• Expand the deep-schema gold standard. The current expert annotation covers two representative papers and thirteen text-derived process instances; broader annotation across more materials and deposition chemistries would support stronger variance estimates and property-level error analysis.

• Make extraction multimodal. Inspection of figures revealed an additional ZnO process that the text-only evaluation excludes, which the authors cite as evidence that complete materials extraction is inherently multimodal and requires integrating tables, plots and schematics.

• Design reflection that targets correctable, metric-aligned errors. The non-monotonic results — debate underperforming two-judge refinement on IGZO, and one model losing accuracy after reflection — raise the open question of when an evaluator's feedback actually aligns with exact structured correctness, process identity and property assignment.

• Improve handling of multicomponent processes. The 0.344 best IGZO F1 identifies supercycle reconstruction, process segmentation and dense numerical assignment as the unresolved technical problems, rather than entity recognition.

• Establish the right diagnostic role for embedding-based similarity. The authors hold that embedding proximity does not establish scientific equivalence for quantities, identifiers or process assignments, leaving open how such measures should be implemented and interpreted as secondary diagnostics.

Target Audience

Researchers and engineers working on scientific information extraction, materials informatics and scholarly knowledge graphs will gain the most, particularly those building LLM-based pipelines that must produce schema-conformant records rather than free text. The paper is also relevant to database and infrastructure teams at materials data repositories who need to decide where in their workflow identifier linking and quality control should live. Because the transfer experiments use biomedical benchmarks and the cost analysis is provider-neutral, NLP practitioners evaluating agentic refinement strategies, LLM-as-a-judge evaluation, or quality–cost trade-offs in structured generation will find the results applicable beyond materials science.

Authors’ abstract

Materials literature contains detailed experimental knowledge, but procedures, chemical entities and measurements remain difficult to aggregate because they are reported in heterogeneous forms and depend on process-specific context. We present SciKGExtract, a schema-guided framework that combines large-language-model extraction with chemical normalization and agent-based evaluation and refinement before knowledge-graph integration. We evaluate the framework on 176 atomic-layer-deposition papers describing zinc oxide (ZnO) and indium--gallium--zinc oxide (IGZO), together with an expert-annotated full-schema subset. PubChem normalization improves exact-match extraction F1 for every tested model. For ZnO, the best F1 increases from 0.591 for direct normalized extraction to 0.805 with agentic refinement, whereas the best IGZO result is 0.344, revealing the greater difficulty of multicomponent supercycle processes. Evaluation against a deeply nested schema containing 65 experimental properties and 155 quantitative measurement nodes further exposes errors in process segmentation and numerical assignment. These results show that chemical canonicalization and targeted agentic verification provide complementary controls for converting complex materials literature into reusable, machine-actionable experimental knowledge.