RECITE: Revising Eligibility Criteria Incorporating Textual Evidence

AI summary

Overview

Research area: Clinical trial design informatics — specifically, automated revision of clinical trial eligibility criteria using large language models (LLMs), retrieval-augmented generation (RAG), and agentic literature search.

Technical level: Intermediate. The paper assumes familiarity with LLM evaluation concepts (ordinal scoring, Cohen's kappa, retrieval-augmented generation, context windows) but explains the clinical motivation from first principles.

Scope: RECITE is a framework plus benchmark that pairs original trial eligibility criteria with documentary evidence and the revision actually implemented, then evaluates fifteen LLM configurations on generating evidence-grounded criteria revisions, and applies the pipeline to published literature to surface trial-design recommendations.

What This Paper Is About

Clinical trials routinely fail to enroll enough participants, and eligibility criteria that are stricter than medically necessary are a recognized cause. A large body of published guidance already says how specific criteria should be loosened, but no system connects those recommendations to the thousands of registered trials at scale. RECITE addresses this by building a benchmark from protocol amendment documents (where the "before" criteria, the rationale document, and the "after" criteria naturally co-occur), benchmarking LLMs at producing those revisions, and then deploying the best-performing approach against real published papers to match their recommendations to active trials.

Key Contributions

1. Task formulation. The authors define "evidence-guided eligibility criteria revision" as a formal mapping from original criteria plus an evidence document to revised criteria plus a provenance mapping linking each change to specific passages — designed to work across protocol amendments, papers, and guidelines.

2. Benchmark construction. A pipeline over 567,815 registered ClinicalTrials.gov trials yields 11,913 curated instances across 5,735 unique trials, each containing original criteria, grounding evidence from a protocol amendment PDF, and the target revised criteria; a 3,116-instance stratified sample is used for evaluation.

3. An agentic literature discovery system. A Python-controlled orchestrator alternates between LLM query generation, PubMed and Semantic Scholar API search, relevance scoring (1–5), and query expansion, then generates ClinicalTrials.gov API queries and scores candidate paper–trial pairs for match quality and applicability.

4. Real-world application. The pipeline produced 44 paper–trial matches with extracted eligibility-change directives and directional accrual signals, drawn from ASCO-Friends broadening, organ-function liberalization, hepatocellular carcinoma criteria expansion, and ECOG performance-status adjustments.

Main Findings

• Judge reliability: The GPT-4o LLM-as-judge agreed with ten domain experts at quadratic-weighted Cohen's κ_qw = 0.694 ± 0.096 ("substantial"), which falls inside the inter-human range of [0.531, 0.823]; on the binary acceptability scale agreement was only κ = 0.292 ± 0.286 ("fair"), with roughly triple the standard deviation.

• Top model performance: On the full 3,116-instance sample, GPT-4o reached an ordinal score of 3.0 ± 0.8 (normalized 0.745, 75.3% of ratings ≥ 3), and the best open-weight model, Gemma 2 9B, reached 3.0 ± 0.9 (normalized 0.742, 70.1% ≥ 3). Qwen 2.5-72B also scored 3.0 ± 0.8 (normalized 0.754).

• Long context is not capability: Qwen 2.5-7B-128K retained 99.4% of evidence text but scored only 0.2 ± 0.6 on the ordinal scale, versus 2.8 ± 0.9 for the standard 32K-window Qwen 2.5-7B.

• Scaling helps within families: Qwen 2.5 went 0.7 → 2.5 → 2.8 across 0.5B → 3B → 7B; Gemma 2 went 1.9 → 3.0 → 2.8 across 2B → 9B → 27B.

• Hard cases remain hard: On the substantial-change subset (sim ≤ 0.95, n = 1,351, 43.4% of the benchmark), every model degraded — GPT-4o fell from 75.3% to 68.4% of ratings ≥ 3.

• RAG gives modest, variable gains: Under truncated-to-context-window conditions, 128K-window models already retain 99.4% of evidence, so RAG gains mainly benefit small-context models on long documents.

• Document length barely matters in the single-document setting: Judge performance showed negligible dependence on evidence length (Spearman ρ = −0.04), and RAG versus no-RAG curves were highly correlated (Pearson R > 0.96), differing in level rather than trend.

• Fine-tuning gave small returns: QLoRA on 720 stratified samples moved Qwen 2.5-3B from 2.5 to 2.6 ordinal and left Qwen 2.5-0.5B essentially unchanged (0.7 → 0.5).

• Literature mining works at small scale: The agent found 44 paper–trial matches, with source-paper accrual signals ranging from 25% to 143% estimated gain; the abstract states these span five publications, while Section 6.1 describes six source publications and 41 unique trials, and two of the papers (ECOG GU and ECOG all) were reclassified as reporting scoring practices rather than protocol-level eligibility changes.

Methodology in Plain English

Building the benchmark from amendments. The authors started with every trial on ClinicalTrials.gov. They first used registry metadata to keep only trials whose "Eligibility" module was ever flagged as changed, cutting 567,815 trials to 59,996. They then compared consecutive versions of the criteria text and threw out pairs that were identical apart from whitespace or formatting (similarity above 0.80, or fewer than 5% of lines changed), leaving 85,083 confirmed changes across 49,369 trials. Roughly 3,500 of those trials (about 7%) had a downloadable protocol amendment PDF, which supplies the "why" document. Text was pulled from those PDFs and joined to the criteria changes, using a date-based match with a 7-day tolerance window. The result is a set of triplets: criteria before, the amendment text, and criteria after.

Asking models to do the revision. Each model is given the original criteria plus the evidence document and asked to produce a revised version. Two delivery modes are tested: pasting the whole document into the context window, or chunking and embedding it (BAAI/bge-small-en-v1.5 via LlamaIndex) and retrieving the top 10 passages. Because the average evidence document is about 134,000 characters (up to 3.1 million), the authors report how much evidence survives truncation: 23.5% for 8K-window models, 81.1% for 32K-window models, and 99.4% for 128K-window models.

Scoring without a human for every item. A GPT-4o judge rates each generated revision on a 0–4 scale, from no overlap (0) through partial overlap (2) to fully equivalent (4). To check whether the judge is trustworthy, ten domain experts rated 160 held-out instances, giving 200 paired ratings; the ordinal scale held up better than a yes/no acceptability judgment, so the ordinal score is the headline metric.

Matching papers to trials. For the real-world application, an LLM proposes search queries, PubMed and Semantic Scholar return papers, and each paper is scored 1–5 for relevance, extraction confidence, and accrual-impact ease. Papers scoring highly feed back into the query queue. From papers scoring at least 4 for relevance, the system extracts concrete change directives and impact quotes, generates ClinicalTrials.gov search queries, and scores each paper–trial pair for match quality and applicability; pairs scoring at least 4 on both seed further searches. Extracted directives are fuzzy-matched against the source text with a sliding-window SequenceMatcher check to catch hallucination.

Why This Matters

Research impact. The paper reframes a clinical knowledge problem as an infrastructure problem: the recommendations for broadening eligibility criteria already exist across oncology, cardiology, neurology, psychiatry, rheumatology, and nephrology, but nobody can manually connect them to tens of thousands of registered trials. RECITE supplies both a benchmark for measuring progress and a validated automated judge for scoring revisions. The reported motivation is concrete: 76% of protocols undergo at least one amendment and 53% modify eligibility criteria, each Phase III amendment costs a median of $535,000, and insufficient accrual accounts for 36.7% of stoppages among 28,561 discontinued trials.

Real-world applications:

• Protocol drafting support — surfacing evidence-backed broadening options for criteria like brain-metastasis exclusions, prior-malignancy exclusions, and creatinine clearance thresholds, with citations to the source passage.
• Amendment triage — pairing trials whose criteria are unusually restrictive with published directives applicable to them, so sponsors can prioritize amendments.
• Systematic review of exclusion practices — quantifying how many trials still exclude populations that consensus guidance says should be included (the paper cites CKD exclusion rising from 66% to 79% over twenty years).
• Trial-design literature surveillance — the agentic crawler continuously finds new papers and matches them to active trials, keeping design guidance current.

Industry relevance. Sponsors, contract research organizations, and trial sites bear the cost of failed accrual and repeated amendments; a system that proposes defensible, citation-backed broadening options maps directly onto feasibility assessment and protocol design workflows. The paper's use of a PHI-safe, HIPAA-compliant Azure deployment (UCSF Versa) for commercial models signals a path toward regulated-environment deployment.

Future Directions

• Extend beyond abstracts. Directive extraction currently operates on paper abstracts, so recommendations buried in methods or results sections may be missed; full-text extraction is named as future work.
• Verify accrual estimates against real enrollment outcomes. The 25–143% figures are inherited from source publications and treated as directional only; the authors explicitly decline to claim numeric gains for matched trials, and note that mechanically scaling source percentages would assume patient-population homogeneity those papers do not establish.
• Move past single-document evaluation. The paper argues retrieval-based methods will become necessary when extending from one evidence document to multi-document evidence collections such as full guideline sets.
• Test other therapeutic areas and document types. The real-world application is confined to oncology because that is where eligibility liberalization research is most mature; extension to other areas is described as architecturally straightforward but empirically unvalidated, and systematic evaluation on clinical guidelines and institutional reports remains open.

Target Audience

Clinical trial informaticists and computational researchers working on LLM applications in medicine; trial sponsors, CROs, and site investigators interested in evidence-based protocol design; and benchmark builders who need a worked example of validating an LLM-as-judge protocol against domain experts before trusting it as a primary metric. Readers without a clinical trials background can follow the framing, but the model-comparison and reliability sections assume comfort with evaluation statistics.

Authors’ abstract

Overly restrictive eligibility criteria contribute to widespread enrollment failure in clinical trials, yet published recommendations for how specific criteria should be broadened remain disconnected from trial design at scale. We introduce RECITE (Revising Eligibility Criteria Incorporating Textual Evidence), a framework for automated discovery, extraction, and implementation of evidence-based eligibility criteria revisions. RECITE uses protocol amendment documents from ClinicalTrials.gov to construct a dataset of 11,913 curated instances across 5,735 trials, each pairing original criteria with documentary evidence and target revisions. We benchmark fifteen LLM configurations under truncated-to-context-window and retrieval-augmented generation conditions, on a 3,116-instance sample under an LLM-as-judge protocol validated against ten domain experts (judge vs. human quadratic-weighted Cohen's $κ_{qw} = 0.694 \pm 0.096$). The best open-weight model, Gemma 2 9B, achieves $κ_{qw}$-aligned ordinal score $3.0 \pm 0.9$ on a 0--4 scale (normalized 0.742) and the best closed-weight model, GPT-4o, reaches $3.0 \pm 0.8$ (normalized 0.745). Using an agentic literature discovery system, we then identified 44 paper--trial matches across five publications providing trial-design and EC-modification evidence. Our findings suggest that existing clinical knowledge can be operationalized into actionable trial design recommendations.