PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction

AI summary

Overview

• Research area: Natural Language Processing, specifically structured information extraction with large language models (LLMs), JSON schema design, and reflection-based guardrails for LLM agent systems.
• Technical level: Advanced. The paper assumes familiarity with JSON Schema, constrained/grammar-guided decoding, LLM agents, tool/API calling, and prompt-based self-correction loops.
• Scope in one sentence: The paper introduces PARSE, a two-part system (ARCHITECT and SCOPE) that automatically rewrites JSON schemas so LLMs can extract structured data more reliably, then validates and repairs extractions with static and LLM-based guardrails, evaluated on three datasets.

What This Paper Is About

Structured extraction systems typically hand an LLM a JSON schema written by human developers and force the model to conform to it, using constraint decoding, reinforcement learning, or retry loops. The authors argue this is a mismatch: such schemas were written as contracts for humans and static code, not as instructions for LLM agents, so they contain ambiguous field descriptions, missing validation rules, and structures that confuse the model — causing hallucinations and schema non-adherence. The goal of PARSE is to treat the schema itself as an optimizable artifact, refining it for LLM consumption while keeping output compatible with the original schema through generated transformation code, and pairing it with an extraction stage that reflects on and corrects its own errors.

Key Contributions

1. ARCHITECT (Automated Refinement and Conversion Handler for Information Transformation and EnhanCemenT): an automated, iterative schema optimization framework that analyzes extraction performance on synthetic test data, then adds entity descriptions, validation rules, and structural clarity to produce a schema optimized for LLM consumption.
2. RELAY (Reverse Engineering Layer for Automated Yoking): a sub-module of ARCHITECT that automatically generates Python transformation code to map outputs from the optimized schema back to the original user schema, preserving backward compatibility with downstream systems.
3. SCOPE (Schema Compliant Organized Pattern Extractor): a reflection-based extraction framework combining static guardrails with LLM-based ones, performing multi-stage validation and generating structured reflections to guide correction.
4. An integrated two-phase design — a one-time Build Phase (ARCHITECT + RELAY) and a repeated Extract Phase (SCOPE) — evaluated on Retail-Conv, Schema-Guided Dialogue (SGD), and Structured Web Data Extraction (SWDE) across five LLM variants.

Main Findings

• Large accuracy gains on web-structured data: PARSE achieves up to 64.7% improvement in extraction accuracy on SWDE over the state-of-the-art baseline.
• Consistent gains across datasets and models: combined framework improvements reach 10% across SGD, SWDE, and internal retail conversation data, and hold across Claude 3.5/3.7 Sonnet, Claude 3.5 Sonnet V2, Claude 3.5 Haiku, Llama 4-Maverick, and DeepSeek-R1-671B.
• Fast error recovery: SCOPE reduces extraction errors by 92% within the first retry, compared with a baseline that simply re-prompts on failure.
• Most schema gains come early: ARCHITECT improves validation accuracy over iterations but most gains are achieved within the first 5-6 iterations, after which further optimization risks overfitting.
• Schemas transfer across models: schemas optimized with one model (Table 2 reports Claude 3.5 Sonnet V2 in ARCHITECT) retain benefits when used with another (Llama), suggesting ARCHITECT finds model-agnostic improvements rather than model-specific tweaks.
• Schema edits follow a stable pattern: across datasets the change types are structural reorganization 55%, entity description enhancement 34%, pattern rule additions 3%, and validation rule addition 0.08%.
• Latency is a real cost, partly offset: SCOPE increases latency by an average of 10.16 (the paper reports this figure without a unit, while Table 3 reports latency figures in what appear to be seconds), but ARCHITECT-optimized schemas reduce that latency penalty by an average of 4.05s because fewer reflection rounds are needed. Reported SWDE latency examples include Claude 3.7 Sonnet at 9.08 with the baseline agent on the original schema, 25.58 for SCOPE on the original schema, 8.80 for the baseline agent on the ARCHITECT schema, and 19.06 for SCOPE on the ARCHITECT schema.
• A worked HTML example explains the SWDE gains: in the reported case, the expected value was "2010 Subaru Legacy"; the baseline extracted "2010 Subaru Legacy 2.5 i 4dr Sedan", while PARSE's optimized schema (with a description specifying full model name including year, make and model, plus a regex pattern) and SCOPE's grounding check returned "2010 Subaru Legacy".
• Guardrails catch semantic confusions: SCOPE's grounding verification caught cases where the baseline confused similar entities such as "old price" versus "new price".
• A metric caveat: the paper does not report precision/recall breakdowns or cost-per-extraction figures; performance is measured as strict field-level accuracy (all required fields correct and properly formatted) plus end-to-end extraction time.

Methodology in Plain English

The system runs in two phases.

Build Phase (ARCHITECT + RELAY). Starting from the user's original JSON schema and a natural-language task description, ARCHITECT generates an improved schema. It then creates synthetic test cases designed specifically to break the extraction system — examples targeting contextual ambiguity, structural challenges, semantic traps, linguistic complexity, and error conditions — seeded with a small dataset of real examples so the synthetic data stays realistic. An extraction agent (which can be SCOPE or any other extraction agent) runs over these synthetic cases, the errors are analyzed, and a refinement step edits the schema: rewriting field descriptions, flattening or reorganizing structure, adding rules such as regex patterns, length limits, enums, and required fields. This loop repeats until accuracy crosses a threshold or a maximum iteration count is reached. In parallel, RELAY writes and tests Python code that converts optimized-schema output back into the original schema format, so existing downstream systems keep working.

Extract Phase (SCOPE). Input text and the optimized schema go to an extraction agent with static guardrails baked into the schema. The output passes through three validation stages in order: (1) missing attribute check, verifying all required fields are present; (2) grounding verification, confirming extracted values actually appear in the source text; (3) rule compliance check, verifying patterns, length limits, enum values, date formats, and the like. Each stage returns pass/fail plus error details. On failure, SCOPE produces a structured reflection that tells the extraction agent what went wrong, and the agent retries. Date grounding is handled through LLM-based prompting using declared allowed date formats and delimiters.

Evaluation compares a best-practice prompting baseline with constraint decoding on the original schemas against SCOPE with and without ARCHITECT-optimized schemas, on Retail-Conv (an internal set of 6 retail conversation schemas with 40 samples each, 240 total), SGD (20,000 annotated task-oriented conversations across 20 domains), and SWDE (1,600 test samples across 8 verticals, 200 per vertical), using strict field-level accuracy and end-to-end latency.

Why This Matters

Impact on research. The paper reframes structured extraction as a co-optimization problem between schema design and extraction mechanism, rather than treating schemas as immutable and only pushing on decoding or training. It connects schema-level natural-language contract design to agent reliability, and provides evidence that schema optimization generalizes across models.

Practical applications:

• Tool and API calling by LLM agents: extracting precise parameters from natural-language requests so agents invoke functions with valid arguments.
• Conversational/assistant systems: pulling slot values such as dates, locations, and transaction amounts out of multi-turn dialogue (the SGD and Retail-Conv settings).
• Web data extraction: parsing semi-structured HTML for entities such as vehicle model, price, engine type, and fuel economy (the SWDE setting).
• Backward-compatible schema migration: using RELAY-generated transformation code to adopt optimized schemas without breaking existing consumers.

Industry relevance. The work targets production LLM agent systems where schema non-adherence and hallucinated fields cause failures, and where accuracy improvements must be weighed against latency. The authors are affiliated with RBS Tech Sciences, Amazon, and one evaluation dataset is internal retail conversation data, indicating the authors frame this as directly relevant to deployed commercial assistants and agentic pipelines.

Future Directions

• Controlling the cost of iterative refinement: the authors note that each refinement iteration requires synthetic data generation, extraction evaluation, and failure analysis, which can be expensive for schemas with many attributes, and that optimizing too long can cause overfitting.
• Reducing dependence on seed data: ARCHITECT's quality depends on having representative seed examples, which is hard for new domains or rapidly evolving requirements; a method that needs little or no seed data is an open problem.
• Handling continuously evolving schemas: the current framework assumes schemas are relatively static and can be optimized offline, which does not fit environments where schemas change frequently.
• Multi-modal schema optimization: extending the same "schema as optimizable interface" principle to extraction from combined textual and visual content, which would require cross-modal grounding in the validation framework.
• Application to other extraction domains: the discussion suggests applying PARSE's schema optimization and validation layer to named entity recognition and multi-agent orchestration, where parameter hallucination and formatting errors are known failure modes.

Target Audience

Researchers and engineers working on LLM-based information extraction, structured output generation, and agent tool-calling pipelines; practitioners who maintain JSON schemas consumed by LLMs and need both higher extraction accuracy and backward compatibility; and NLP researchers interested in self-correction, guardrails, and treating prompts/schemas as optimizable artifacts rather than fixed inputs.

Authors’ abstract

Structured information extraction from unstructured text is critical for emerging Software 3.0 systems where LLM agents autonomously interact with APIs and tools. Recent approaches apply large language models directly to extraction tasks using existing JSON schemas, often with constraint decoding or reinforcement learning approaches to ensure syntactic validity, but treat JSON schemas as static contracts designed for human developers, leading to suboptimal extraction performance, frequent hallucinations, and unreliable agent behavior when schemas contain ambiguous or incomplete specifications. We recognize that JSON schemas themselves are a form of natural language understanding contract that encodes rules, relationships, and expectations about data structure contracts that LLMs should be able to both interpret and systematically improve. Consequently, we develop PARSE (Parameter Automated Refinement and Schema Extraction), a novel system with two synergistic components: ARCHITECT, which autonomously optimizes JSON schemas for LLM consumption while maintaining backward compatibility through RELAY (an integrated code generation system), and SCOPE, which implements reflection-based extraction with combined static and LLM-based guardrails. We evaluate PARSE qualitatively and quantitatively on three datasets including Schema-Guided Dialogue (SGD), Structured Web Data Extraction (SWDE), and internal retail conversation data, and find that it achieves up to 64.7% improvement in extraction accuracy on SWDE with combined framework improvements reaching 10% across models, while reducing extraction errors by 92% within the first retry and and maintaining practical latency.