Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis

AI summary

Overview

Research area: Natural Language Processing — scientific information extraction, LLM evaluation, and automated evidence synthesis for systematic reviews and meta-analyses.

Technical level: Intermediate. The paper assumes familiarity with schema-constrained extraction, tuple-level precision/recall/F1, and standard LLM prompting regimes, but it is written as a diagnostic study rather than a modeling paper.

Scope: The paper proposes and applies a structural, diagnostic evaluation framework to test whether two state-of-the-art LLMs can produce schema-conformant, numerically grounded study records from full-text scientific articles across five domains.

What This Paper Is About

Systematic reviews and meta-analyses require converting narrative articles into structured, numerically grounded study records, where populations, variables, statistical methods, and effect sizes must all be bound together correctly and attributed to the right document. The authors argue it is unclear whether current LLMs can meet these structural requirements, since success hinges on preserving roles and effect-size attribution rather than on recognizing isolated entities. Their goal is to build a framework that turns evidence extraction into a graded series of schema-constrained queries, so that the exact point where LLM performance collapses can be identified.

Key Contributions

1. A structural and diagnostic perspective on LLM-based scientific evidence extraction, framing extraction as a progression of increasingly constrained, schema-guided tasks rather than as an end-to-end pipeline problem.
2. A tiered query framework that operationalizes this perspective through controlled variations in tuple arity and relational binding complexity, covering both direct extraction and derived statistical reasoning.
3. A manually curated, multi-domain, schema-aligned benchmark spanning five scientific domains (civil engineering, medical and health science, agricultural science, earth and environmental sciences, and social science), designed to systematically investigate LLM performance under increasing structural demands.
4. A comprehensive empirical analysis identifying where along this structured progression modern LLMs begin to fail, revealing specific weaknesses in relational binding, numerical grounding, and document-level attribution.

Main Findings

• Performance is moderate at the single-property level and collapses with binding. Macro-averaged over all tasks under the Per-Paper regime, object-centric single-atom extraction (O1) reaches F1 = 0.62 for Qwen3-VL and 0.50 for GPT-5.2, dropping to O2 = 0.30 and 0.20 respectively once multiple properties must be jointly bound. Method-centric tasks score M1 = 0.45 for both models and M2 = 0.22.

• Full meta-analytic association tuples are essentially never recovered. The highest-arity method-centric task (i, IV, DV, A, C, E) — variable pair, statistical method, conditions, and effect size — attains an F1 of 0.00 under both input regimes and both models.

• Long-context input makes things worse, not better. Per-Paper input outperforms Global input for both models on macro-average: Qwen3-VL falls from 0.35 to 0.18, and GPT-5.2 from 0.28 to 0.24. The Global regime hurts L2 tasks more than L1 tasks, with the largest drops in method-centric extraction: for Qwen3-VL, M2 decreases from 0.22 to 0.05 and MC from 0.28 to 0.00, while O1 only goes from 0.62 to 0.57 and O2 from 0.30 to 0.27.

• Role-aware extraction is harder than role-agnostic extraction. At L1, role-agnostic variable extraction (i, V) achieves F1 = 0.44, versus 0.37 for (i, IV) and 0.42 for (i, DV). At L2, (i, V, S, U) reaches 0.25, whereas (i, IV, S, U) and (i, DV, S, U) reach 0.15 and 0.16. Adding method binding to variable pairs reduces F1 from 0.26 (i, IV, DV) to 0.21 (i, IV, DV, A).

• Not all entity types are equally hard. Within object-centric tasks, geolocation and sample size yield the highest F1 (0.72 and 0.70), while study population or unit-of-analysis extraction is substantially lower (0.35).

• Object-centric tasks beat method-centric tasks at matched arity. At L1, macro-averaged F1 is 0.59 for O1 versus 0.38 for M1; at L2, O2 = 0.24 versus M2 = 0.17.

• Degradation is recall-dominant. Precision and recall both fall as structural complexity increases, but recall more frequently reaches near-zero first, particularly for method-centric L2 and derived tasks — a pattern consistent across models and domains.

• Domain-level results shift with the input regime. Civil Engineering achieves the highest overall performance under both regimes (Per-Paper 0.47/0.43; Global 0.24/0.30 for Qwen3-VL/GPT-5.2). Medical & Health Science ranks second under Per-Paper input (0.42/0.36). Social Science yields the lowest overall F1, particularly under Global input (0.08/0.18). Earth & Environmental Science shows a large model gap under Global input (Qwen3-VL 0.13, GPT-5.2 0.24), and domain-level variance increases substantially under the Global regime.

• Failures are structural, not recognition-based. The authors report four recurring error patterns accounting for the declines in method-centric and derived tasks: role reversals, cross-analysis binding drift, instance compression in dense result sections, and numeric misattribution. Downstream aggregation amplifies even minor upstream errors, rendering corpus-level statistics unreliable.

Methodology in Plain English

The authors start from a semantic study schema — a defined list of fields such as study population, geolocation, sample size, statistical method, independent variable, dependent variable, scale and unit, conditions, and effect size — with every extracted value tagged by the ID of its source document. This schema is a simplified, generalized version of a richly nested Open Research Knowledge Graph (ORKG) template for early drought stress indicators in plants, stripped of domain-specific detail so it can apply across association-based empirical research.

They then design a tiered suite of queries. L1 queries ask for one field at a time (e.g., "Extract the study geolocation as a country name"). L2 queries ask for several fields bound into one record (e.g., "Extract each independent variable with its Scale and Unit," or the full (i, IV, DV, A, C, E) association tuple). A third family, derived statistical queries (C), operates on the extraction output — counting documents with sample size above 100, computing the mean or median of sample size across documents, counting methods or variables per document, or listing independent–dependent variable pairs with effect size above 0.7. Queries split into object-centric (study setting and subjects) and method-centric (statistical evidence structures) families.

Five domain corpora were assembled from meta-analysis review articles (plus one journal query for agricultural science), containing 9 to 11 papers each. Gold annotations were created manually by a single expert annotator, with no inter-annotator agreement procedure, on the grounds that the task is factual extraction rather than interpretive labeling. PDFs were converted to markdown with MinerU 2.5, preserving tables as HTML and formulas as LaTeX, and concatenated into roughly 20k-token inputs per domain with no manual editing or truncation.

Two models were tested under identical decoding settings (temperature = 0.1): GPT-5.2 (version 2025-12-11) and the open-source vision-language model Qwen3-VL (Qwen3-VL-235B-A22B-Instruct-FP8), served via vLLM on 4× H100 GPUs. No external tools, retrieval, or code execution were allowed. Two input regimes were compared: Global (all documents in a domain concatenated into one input) and Per-Paper (each query applied to one document at a time, followed by a separate aggregation step). A total of 2,976 LLM calls were executed.

Scoring uses tuple-level precision, recall, and F1. A prediction counts as correct only if all required fields match a gold tuple and it is attributed to the correct paper — enforced via citation markers. Matching uses character-level sequence similarity first, with candidates below 0.95 similarity passed to an LLM judge for confirmation. Counting and aggregation queries require exact numerical equality with the value computed from gold tuples.

Why This Matters

Impact on research. The paper reframes the question from "can LLMs find entities?" to "can LLMs preserve structure under aggregation?" Its central claim is that current LLMs lack the structural fidelity, relational binding, and numerical grounding required for automated meta-analysis, and that the failure mode is systematic and predictable rather than a matter of entity recognition. Because the code, gold-standard JSON annotations, query suite, experiment scripts, and article DOIs are released (PDFs are not redistributed due to copyright), the diagnostic framework can be reused as a stress test for future systems. The paper positions this as complementary to existing schema-aware and neural–symbolic approaches.

Real-world applications:

• Systematic review production in evidence synthesis organizations (e.g., Cochrane-style workflows), where data abstraction is described as highly labor-intensive and a major bottleneck to reproducible, machine-actionable meta-analytic datasets.
• Biomedical and health evidence pipelines, where semi-automated evidence extraction has already been explored and where the medical corpus here had the largest sample sizes (total 1,391,776 across 9 papers, with a maximum of 1,323,052).
• Scholarly knowledge graph and digital library infrastructure, such as ORKG, Semantic Scholar's Literature Graph, and the Computer Science KG, which depend on schema-conformant, document-attributed study records.
• Environmental, agricultural, and social science meta-analyses, where the presented corpora cover carbon storage vs. biodiversity, environmental stress vs. plant responses, and compassion vs. well-being and burnout.

Industry relevance. The findings argue against relying on long-context ingestion as a shortcut to cross-document evidence integration, and they quantify how per-document decomposition with a separate aggregation step outperforms a single global context. For teams building extraction products for publishers, research-integrity services, or clinical evidence platforms, the result is a concrete warning that corpus-level statistics computed from LLM-extracted evidence can be unreliable even when upstream field-level accuracy looks acceptable.

Future Directions

• Building systems that survive structural demands rather than entity recognition. The paper concludes that its analysis provides "a foundation for designing future LLM-based systems that better align with the needs of evidence synthesis and digital library infrastructure," but it does not itself propose or test a new extraction model or pipeline.
• Improving relational binding and numeric grounding. The identified failure modes — role reversals, cross-analysis binding drift, instance compression in dense result sections, and numeric misattribution — define specific targets for schema-aware, neural–symbolic, or constrained-decoding approaches that the related work section surveys.
• Closing the gap at the highest arity. Since the full (i, IV, DV, A, C, E) tuple scores 0.00, the question of what architectural or prompting change could make complete meta-analytic association tuples extractable at all remains open.
• Understanding domain and regime variance. Domain-level performance shifted markedly between Per-Paper and Global input, with Earth & Environmental Science showing a large model gap under Global input; the truncated content does not report an explanation for these domain-specific differences, and the paper's conclusions and limitations sections are not included in the available text.

Target Audience

Researchers and practitioners in NLP for scientific text, scholarly knowledge graph construction, and evidence synthesis who need to know the limits of LLM-based data extraction. It is also relevant to systematic review methodologists and digital library engineers evaluating whether LLM pipelines can be trusted for structured study records, and to developers building schema-constrained extraction systems who want a concrete diagnostic suite and a released benchmark to test against. Readers should be comfortable with precision/recall/F1 terminology and with the notion of schema slots and tuple arity; no deep machine learning background is required.

Authors’ abstract

Systematic reviews and meta-analyses rely on converting narrative articles into structured, numerically grounded study records. Despite rapid advances in large language models (LLMs), it remains unclear whether they can meet the structural requirements of this process, which hinge on preserving roles, methods, and effect-size attribution across documents rather than on recognizing isolated entities. We propose a structural, diagnostic framework that evaluates LLM-based evidence extraction as a progression of schema-constrained queries with increasing relational and numerical complexity, enabling precise identification of failure points beyond atom-level extraction. Using a manually curated corpus spanning five scientific domains, together with a unified query suite and evaluation protocol, we evaluate two state-of-the-art LLMs under both per-document and long-context, multi-document input regimes. Across domains and models, performance remains moderate for single-property queries but degrades sharply once tasks require stable binding between variables, roles, statistical methods, and effect sizes. Full meta-analytic association tuples are extracted with near-zero reliability, and long-context inputs further exacerbate these failures. Downstream aggregation amplifies even minor upstream errors, rendering corpus-level statistics unreliable. Our analysis shows that these limitations stem not from entity recognition errors, but from systematic structural breakdowns, including role reversals, cross-analysis binding drift, instance compression in dense result sections, and numeric misattribution, indicating that current LLMs lack the structural fidelity, relational binding, and numerical grounding required for automated meta-analysis. The code and data are publicly available at GitHub (https://github.com/zhiyintan/LLM-Meta-Analysis).