Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI

AI summary

Overview

Research area: Generative AI for information extraction and automated evaluation of structured outputs, applied to Health Technology Assessment (HTA) documents.

Technical level: Intermediate. The paper assumes familiarity with JSON schemas, precision/recall/F1 metrics, and large language model APIs, but the prose is written for a mixed audience of domain researchers and research engineers.

Scope: The paper presents and evaluates a single-schema pipeline that uses a generative AI model to extract 14 hierarchical, variable-cardinality attributes from full HTA documents in one zero-shot call, and then automatically scores those extractions against a human gold standard using path-based matching plus generative AI semantic comparison.

What This Paper Is About

HTA organisations publish long, highly specialised documents whose content and structure vary from document to document, organisation to organisation, and over time. Extracting the same set of attributes from all of them for comparative research is currently done manually by domain experts because the data is nested, variable in length, and requires deep domain knowledge to interpret. This paper asks whether a single JSON schema, acting as a shared information model, can guide a generative AI model through both the extraction and the subsequent evaluation of that complex data end-to-end.

Key Contributions

1. A schema-as-information-model framework that encodes domain knowledge (attribute names, descriptions, types, patterns, enum value sets, and nesting structure) in one JSON Schema that orchestrates extraction, output formatting, and evaluation in a single unified pipeline.

2. Single-call, zero-shot extraction of hierarchical data — the entire nested output, including attributes of variable cardinality and attributes with sub-attributes, is produced from a 20–40 page document in one API call with no in-context examples.

3. A path-based semantic matching algorithm for evaluation, which aligns nested, variable-cardinality attributes between extracted results and the gold standard by tracing paths to leaf-level elements, using designated identifier (anchor) attributes such as indication_name, inn, and brand_name, and resolving matches by ranking (exact > semantic > useful) with greedy tie-breaking.

4. A four-category comparison rubric — exact match, semantic match, useful match, and non-match — applied by a generative AI model to classify non-identical values, with exact/semantic/useful all counted as true positives and non-match counted as a false positive.

Main Findings

• Extraction performance: 12 of the 14 attributes were extracted with all metrics (precision, recall, accuracy, F1) above 90%, using the generative AI model Claude Opus 3 on documents from the UK health technology assessment organisation NICE.

• Two weaker attributes: outcome_rea reached an F1 score of 0.88 and clinical_restrictions reached 0.83. outcome_rea is described as difficult even for a human, because relative effectiveness discussion is long, nuanced and ambiguous — when documents said things like "This drug seems to be clinically effective, but the evidence presented is insufficient", the AI usually classified the outcome as positive while the human expert classified it as negative. clinical_restrictions performs poorly because it is reported in the same "Recommendations" section as indication_name, making the two hard to separate.

• Speed and cost: Claude Opus 3 took roughly 0.5 minute to extract attributes from one HTA document, against roughly 15 minutes for a domain expert — an approximately 30-fold increase in time efficiency. API cost was about 0.5 euro per document, and both time and cost scale linearly with the number of documents.

• Evaluation behaviour: Out of 796 comparisons against the gold standard, 634 were resolved by the deterministic exact string check. For the remainder, Claude Opus 4.6 classified 20 as exact matches, 19 as semantic matches, 84 as useful matches, and 39 as non-matches. Humans and the AI agreed on all 39 non-matches; where they differed on the other categories, both still classified the pair as one of the three positive match types, so the metrics were unaffected.

• Consistency: Across two runs with the same model, parameters, prompt, and data (using run one as the gold standard for run two), all metrics for all attributes exceeded 96%.

• Comparison with traditional methods: A rule-based method extracted hta_id, assessment_date, internal_identifier, and indication_name at 90% accuracy, and assessment_type, inn, and brand_name at 70% accuracy, and could not extract the remaining attributes with significant accuracy. Among machine learning approaches (Gradient Boosting, Support Vector Machine, Logistic Regression, Random Forest, Naive Bayes, BERT), the best outcome_rea results were Gradient Boosting and Random Forest at F1 0.76 each; the best outcome_cea and final_recommendation results were Gradient Boosting at F1 0.98 and 0.93 respectively. The generative AI method outperformed both.

• Generalisability across models: Comparable F1 scores were obtained with Claude Opus 3, Claude Opus 4.6, Gemini 3.1 Pro, and the open-weights model GPT OSS 120B (accessed through the AI-Hub provided by SURF), with Claude Opus 4.6 used for automated evaluation in all cases.

• Transferability across organisations and languages: The pipeline built for NICE, used without modification, performed reasonably well on several attributes for 25 documents each from Zorginstituut Nederland (ZIN, the Dutch HTA organisation) and Haute Autorité de Santé (HAS, the French HTA organisation). HAS does not report internal_identifier or outcome_cea, and for HAS's managed_entry_agreement, either both the AI-extracted and gold standard values were null, or the AI did not classify the requirement of future assessments as an outcome-based managed entry agreement the way the human expert did.

• Where the pipeline works best: Because the prompt was developed on NICE documents with Claude Opus 3, that combination gives the best performance. The authors suggest the apparently poorer performance of the more advanced Claude Opus 4.6 in some cases stems from it extracting attributes at a higher degree of granularity than the gold standard or than Claude Opus 3.

Methodology in Plain English

Domain experts identified 14 research-relevant attributes and described their hierarchical structure — which attributes are lists of unknown length, and which contain further sub-attributes. Research engineers translated that into a JSON Schema whose fields carry descriptions, allowed types, format patterns, and fixed value sets.

For extraction, the researchers converted PDF documents to text and sent a single prompt containing three parts: the full document text, instructions setting the AI's role and output requirements, and the schema. The model returns one nested JSON object per document. No examples were given (zero-shot), and the temperature parameter was set to zero to make outputs as deterministic and reproducible as possible. Claude Opus 3 was chosen because it produced the most accurate and consistent results among the GPT 3.5, GPT 4, and Claude models tried.

For evaluation, both the extracted output and the human gold standard follow the same schema. Because the two JSON objects may contain different numbers of list elements, and the elements may appear in different orders, direct comparison is not straightforward. The algorithm instead traces the path down to each leaf-level element in one object and looks for the matching path in the other, comparing values at each nesting level. For lists of dictionaries, it uses an identifier attribute to decide which element corresponds to which — indication_name for indications, inn (or brand_name when no INN is present) for technologies. When several candidate pairs match, the algorithm picks the best one-to-one pairing by rank, and uses a greedy search to break ties.

At each leaf-level comparison, the pipeline first checks for an exact string match. If that fails, a generative AI model (Claude Opus 4.6) judges whether the two values are an exact match, a semantic match (same meaning in different words), a useful match (not identical but still useful for comparative HTA research), or a non-match. Presence and absence combinations are classified as true positive, true negative, false positive, or false negative according to a fixed table.

Why This Matters

The work matters because it shows that a schema designed by domain experts can act as a single control point for an entire extraction-and-evaluation pipeline, without fine-tuning, labelled training data, or deep machine learning expertise — resources that domain researchers typically lack. It also pushes beyond previous schema-guided extraction work, which the paper notes was either limited to one level of nesting with exact-string-match evaluation, or used flat schemas that could not represent attribute lists of unknown length.

Real-world applications:

• Comparative HTA research: building an Open Science database of standardised attributes across multiple national HTA bodies to compare assessment practices and recommendations.
• Regulatory and reimbursement intelligence: monitoring how drugs, devices, or therapies are assessed and reimbursed across countries and over time.
• Private or sensitive document processing: using locally hosted open-weights models such as GPT OSS 120B, whose performance the paper reports as comparable to proprietary models, to keep data in-house.
• Reuse of the evaluation algorithm: the authors state it can be applied to any outputs that follow a prescribed schema regardless of how they were produced, provided a matching gold standard exists.

Industry relevance centres on the cost and labour economics: roughly 0.5 euro and about 0.5 minute per document against roughly 15 minutes of expert time, with linear scaling. The paper also flags a practical risk for any organisation building on commercial APIs — the availability, latency, and cost of proprietary models can change, and providers routinely deprecate older models, which threatens reproducibility and the long-term sustainability of such pipelines. The authors also stress that the evaluation step requires a gold standard, so it can only be used on labelled development and test datasets, not in production on unseen data.

Future Directions

• Stronger gold standards: the current gold standard was created by a single domain expert, and manual extraction is itself subject to subjectivity and error. The authors call for gold standards built by multiple experts on the same documents, plus inter-rater reliability analysis comparing human-human agreement with human-AI agreement.

• Prompt tuning per organisation and model: refining prompts for a specific HTA organisation and a specific generative AI model may improve performance, since the current prompt was optimised for NICE documents and Claude Opus 3. The granularity issue with Claude Opus 4.6 suggests prompts may also need to state the desired level of detail explicitly.

• Moving away from proprietary models: the paper raises local deployment of state-of-the-art open-weights models as a way to eliminate dependence on commercial providers for reproducibility and long-term sustainability.

• Applying the framework in new domains: the authors report using the framework for a scoping review of young people's geographies (human geography) and for extracting sociodemographic attributes from the Oxford Dictionary of National Biography (historical sociology), with promising preliminary results.

Target Audience

This paper is most useful to HTA researchers and analysts who need standardised data pulled from large document sets, to research engineers and data scientists building schema-guided LLM pipelines, and to anyone designing automated evaluation of nested structured outputs against a gold standard. It is also relevant to research software teams in universities and public institutions weighing proprietary versus open-weights models, and to reviewers and methodologists interested in how prompt engineering, domain expertise, and human verification interact in applied generative AI work.

Authors’ abstract

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90\% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.