Improving Information Extraction with Learned Queries

AI summary

Overview

Research area: Natural Language Processing — information extraction (IE) and question-based argument extraction, with a focus on clinical and biomedical text.

Technical level: Intermediate. The paper assumes familiarity with LLMs, prompting, fine-tuning with LoRA, and standard IE metrics (precision, recall, F1), but the core argument is conceptual rather than mathematically heavy.

Scope: The paper argues that the questions used to elicit information from a document, not just the model answering them, are a primary bottleneck in LLM-based information extraction, and introduces a framework (List of Questions, or LoQ) for learning those questions.

What This Paper Is About

When an LLM fails to extract the right facts from a document, the usual fix is to improve the extraction model — scale it up or refine its reasoning. This paper shows that a different part of the pipeline matters at least as much: the queries used to elicit the information. The authors introduce LoQ, a framework for generating document- and role-specific question sets, plus FeedQ, a feedback-driven loop that iteratively refines those questions against actual extraction outcomes, and then distills the resulting questions into lightweight, fine-tuned question generators.

Key Contributions

1. Empirical evidence that question design is a major bottleneck in LLM-based information extraction. Across four clinical benchmarks (CaseReportBench, PHEE, DiscourseEE, MACCROBAT) and five LLMs, improving questions raises F1 by 18.6 points on average — more than the gain from using larger extraction models.

2. FeedQ, a feedback-driven iterative refinement method for optimizing extraction questions without annotated question supervision. It generates, evaluates, and refines role-specific questions conditioned on the document, roles, and ground-truth arguments, using a leakage check to prevent ground-truth arguments from being copied into the questions.

3. A dataset of 12,820 (document, role, question-set) triples, produced by running FeedQ over the training splits of the four benchmarks, released to support treating question design as a first-class research problem.

4. A fine-tuning recipe that transfers learned questioning behavior to small models. Fine-tuned Qwen3-4B and Qwen3-8B (via LoRA) match or outperform expert-derived question baselines and substantially exceed much larger untuned question generators, using only the document and role at inference time — no ground-truth access.

Main Findings

• Questions are necessary at all. The No-Question setting (prompting the prediction model with document and role only, no question) degrades performance across all datasets relative to question-based approaches, establishing that questions carry weight in the pipeline.

• Naive document-conditioned questions actively hurt. Contextual-Q — dynamic questions generated per document–role pair without ground-truth access — falls below the No-Question baseline on every dataset. It generates 5.27 questions per document–role pair on average, raising recall but sharply lowering precision (on DiscourseEE, recall rises from 65.7 to 76.3 while precision drops from 37.4 to 28.8). Overall it falls 15.6 F1 below Knowledge-Q.

• Optimized questions deliver large, consistent gains. Optimized-FeedQ outperforms the strongest baseline (Knowledge-Q) by 18.6 F1 averaged across four datasets and all prediction models. Gains hold on every dataset–model combination, ranging from 11.0 (PHEE) to 32.7 (MACCROBAT).

• Much of the gain comes from grounding, and refinement adds more. Zero-shot FeedQ already exceeds Knowledge-Q by 13.2 F1 without iterative refinement. FeedQ optimization adds a further 5.4 F1, with the largest gains on DiscourseEE (+4.9) and MACCROBAT (+10.6), where arguments are most often implicit or distributed.

• Reasoning does not fix a poorly targeted question. CoT-Q shows no benefit over Knowledge-Q (61.1 vs. 61.7).

• Iterative refinement is not redundant. A single FeedQ pass suffices for the majority of roles in PHEE (87%) and MACCROBAT (75%), but nearly half of DiscourseEE roles (47%) and a quarter of CaseReportBench roles require two or more iterations to reach their best-scoring question set.

• One question per role is often not enough. CaseReportBench assigns only 41% of roles a single optimized question; the remaining 59% receive two or more, with 27% requiring four or more.

• Larger question generators are better but not necessarily worth the cost. Zero-shot question generation quality averaged over prediction models was 76.1 F1 for GPT-5.2, 74.9 for GPT-OSS-120B, and 72.1 for Gemini-3.1, versus roughly 54 for the Qwen3 models (Qwen3-4B 54.8; Qwen3-8B 53.4). GPT-OSS-120B was selected because it was over 40× cheaper per token than GPT-5.2 at the time of writing, and its open weights permit local deployment.

• Fine-tuned small models win on the held-out test set. After tuning, both Qwen3-4B and Qwen3-8B reach 62.7 mean F1, up from roughly 53 without fine-tuning, and outperform all non-fine-tuned strong baselines (GPT-OSS-120B at 45.7, GPT-5.2 at 54.9, Gemini-3.1 at 60.1). They also beat Knowledge-Q on average (62.7 vs. 61.3), outperforming it by 4.5–4.7 F1 on PHEE and 4.3–5.6 F1 on MACCROBAT.

• Where human questions still win, data volume explains it. Knowledge-Q retains a marginal edge on CaseReportBench (under 1 F1) and a larger gap on DiscourseEE (3–4 F1). PHEE and MACCROBAT each contribute 5,000 training samples, while CaseReportBench provides only 620 and DiscourseEE 2,200.

• Out-of-domain transfer is mixed. On 500-sample subsets of DocEE, GENEVA, and MUC4, fine-tuning helps DocEE (Qwen3-4B: 46.53 → 55.28 F1; Qwen3-8B: 49.20 → 56.65) and MUC4 (Qwen3-8B: 58.56 → 59.90), but hurts GENEVA (Qwen3-4B: 59.13 → 52.82), which annotates multiple coreferent arguments per role and is better served by the broader question sets of the base model.

Methodology in Plain English

The framework has three phases.

Phase 1 — Question optimization (FeedQ). Given a document, a role, and the ground-truth arguments for that role, a question generator produces an initial set of questions. A leakage-check module (implemented with gpt-oss-120b) flags and rewrites any question that directly or indirectly gives away the ground-truth answer. The cleaned questions go to a prediction model, which extracts candidate arguments. Those predictions are compared against the ground truth, yielding matched, missed, and over-generated arguments plus precision, recall, and F1. That structured feedback is handed to a refiner model, which adds questions targeting missed arguments, removes or rewrites questions causing over-generation, and preserves questions that worked. The loop repeats until an F1 target is hit, no improvement occurs for a patience window, or a maximum number of iterations is reached — and the best-scoring question set is returned, not necessarily the final one.

Phase 2 — Question generation (fine-tuning). The optimized question sets become supervision data. Qwen3-4B and Qwen3-8B are fine-tuned with LoRA to map a document and role to a question set, learning to produce effective questions without seeing ground-truth arguments. Three training mixes were tried: a balanced mix capping datasets at comparable counts (about 6K total), an all-available mix (about 12K total), and an in-domain mix using only the target dataset's triples.

Phase 3 — Prediction. At test time, the fine-tuned question generator produces questions for unseen document–role pairs, and an off-the-shelf prediction model extracts arguments using those questions. Every question generator was paired with every prediction model for a full comparison. Evaluation on the test set uses the hierarchical scheme of Sharif et al. (2025), combining exact match, relaxed match, and LLM-as-judge, since exact-match and relaxed-match metrics are known to penalize semantically correct LLM outputs with surface-form differences.

Why This Matters

Impact on research. The paper reframes question design as a separable, learnable component of the extraction pipeline rather than a fixed artifact written once from annotation guidelines. It provides both the mechanism (FeedQ) and the material (12,820 optimized questions) for others to build on, and it shows that a 4B-parameter tuned model can match or exceed expert-written question baselines — undercutting the assumption that better extraction requires bigger models.

Real-world applications (drawn from the paper's benchmarks and framing):

• Clinical case report mining — turning unstructured rare-disease narratives into structured fields such as laboratory findings, diagnosis, treatment, and outcomes.
• Pharmacovigilance — surfacing adverse drug reaction details (drug, dosage, route, frequency, duration, disorder, patient demographics) from drug-safety texts.
• Online health forum analysis — extracting information-seeking behavior and experiences from informal posts, where 51.2% of arguments in the DiscourseEE dataset are implicit.
• Biomedical literature curation — extracting dense event structures from PubMed case report snippets, where the MACCROBAT dataset averages 3.21 events per sentence.

Industry relevance. The cost argument is explicit: GPT-OSS-120B was over 40× cheaper per token than GPT-5.2 while scoring close to it as a question generator, and its open weights allow local deployment on private data. More importantly, the paper demonstrates that distilling optimized questioning behavior into a 4B or 8B model removes the need to run a large model at inference time — a deployment-relevant result for clinical and other privacy-sensitive settings. The LoQ framing also generalizes beyond clinical text, as the out-of-domain experiments on DocEE, GENEVA, and MUC4 indicate.

Future Directions

• Vary or ensemble the optimizer. FeedQ used GPT-OSS-120B as the sole model for question generation, refinement, leakage checking, and prediction during optimization. Whether different optimizers, or ensembling questions from multiple models, yields further gains is left open.

• Extend beyond clinical and biomedical text. All four primary benchmarks are clinical, where arguments are distributed across narrative sections and expressed in domain-specific terms — properties that make targeted questioning particularly impactful. Applying LoQ to news, law, or finance would test broader applicability.

• Understand when broad question sets beat targeted ones. Fine-tuning helped DocEE and MUC4 but hurt GENEVA, where multiple coreferent arguments per role are annotated. This suggests a boundary condition on the method that is not yet characterized.

• Address the low-supervision regime. Fine-tuned models lagged Knowledge-Q on CaseReportBench (620 training samples) and DiscourseEE (2,200), while beating it on PHEE and MACCROBAT (5,000 each). How to close that gap without more annotation is unresolved.

Target Audience

Researchers and practitioners in information extraction and clinical NLP who work with LLMs; engineers building extraction pipelines who need to decide whether to spend budget on bigger models or better prompts and questions; and clinical informatics teams extracting structured facts from case reports, pharmacovigilance records, or patient-generated text. Readers interested in prompt optimization and feedback-driven methods will also find the FeedQ design relevant, since it optimizes a document- and role-conditioned set of questions under a multi-tier extraction objective with a leakage constraint, rather than a single text prompt.

Authors’ abstract

When information extraction fails, a natural instinct is to improve the model doing it: for example, by scaling it up or refining its reasoning. In this paper, we show that another part of the pipeline matters at least as much: the queries used to elicit this information. Across four clinical benchmarks and five LLMs, improving the question design alone raises performance by 18.6 F1-score points, i.e. more than using larger extraction models. To make such question design learnable, we introduce List of Questions (LoQ), which generates document-specific question sets, and FeedQ, a feedback-driven optimization method that iteratively refines questions against extraction outcomes. The resulting optimized questions can be used to train lightweight generators: with fine-tuning, 4B-parameter models match or outperform expert-derived baselines and substantially exceed the performance of much larger untuned models. We release a dataset of 12,820 optimized questions to support a broader shift in information extraction research toward treating question design as a first-class problem.