ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts

AI summary

Overview

• Research area: Artificial intelligence for healthcare data science, specifically large language model (LLM) agents that automate Real-World Studies (RWS) over electronic health records and insurance claims.
• Technical level: Advanced. The paper assumes familiarity with LLM agents, intermediate representations, benchmark evaluation, and clinical epidemiology concepts such as index dates, censoring, confounding, and analysis grain.
• Scope in one sentence: ProCARE proposes a "contract-valid" pipeline in which a heterogeneous data profile, a typed RWS intermediate representation, deterministic violation checks, and evidence certificates constrain every step from a natural-language study goal to an auditable report, and it is evaluated on a 36-task hepatocellular carcinoma (HCC) benchmark plus an 18-task public-data transfer suite.

What This Paper Is About

Automating real-world studies with LLM agents is appealing but brittle, because valid analyses depend on dataset-specific clinical semantics (index dates, follow-up windows, patient- vs. event-level units, outcome boundaries) that are usually implicit in observational data. The authors identify three recurring failure modes, query-to-specification, specification-to-code, and artifact-to-claim, which they trace to missing contracts between the data, the study specification, the executable workflow, and the final report. ProCARE's goal is to make those contracts explicit, machine-checkable, and auditable so that generated code runs correctly and reported claims stay within the evidence actually produced.

Key Contributions

1. A heterogeneous RWS profile and a typed RWS-IR. The profile Φ is organized into four blocks by level (dataset/task vs. field/instance) and abstraction (structural vs. knowledge-driven): Φ^H,S, Φ^H,K, Φ^L,S, Φ^L,K. The intermediate representation Z = ⟨D, G, J, K, T, V, M, A, Π_q⟩ encodes study design, analysis grain, join graph, entity keys, temporal anchors, variable-role bindings, method specification, required artifacts, and claim policy.
2. Evidence certificates that restrict report claims. Each admitted claim q carries a certificate e_q = ⟨o_q, c_q, z_q, φ_q, a_q⟩ binding it to a supporting artifact, executed code region, IR field or method, profile entry, and optional assumption. A claim is admitted only if a trace condition and a policy condition both pass.
3. Violation-guided execution with localized repair. Deterministic schema, method, and artifact gates return a violation set U_t recording the failed contract clause, the offending code or artifact, the nearest valid profile entries, and the permitted repair action; the next edit is conditioned on these localized failures.
4. An end-to-end RWS evaluation over 36 HCC tasks and 18 Public-RWS transfer tasks, measuring executability, code quality, artifact completeness, and claim traceability, including ablations, expert review, backbone sensitivity, and cost analysis.

Main Findings

• End-to-end benchmark results (36 HCC tasks). ProCARE scored GA 0.86, PF 0.84, XE 0.92, CQ 0.88, VC 0.72, CF 0.74, RS 0.80, SL 0.79, VRR 0.81, CTC 0.90, UCR 0.06, and CLV 0.03. The strongest code-agent baselines were lower across the board: Claude Code at XE 0.86, CQ 0.80, VRR 0.70, CTC 0.66, UCR 0.16, CLV 0.12; Codex at XE 0.89, CQ 0.82, VRR 0.73, CTC 0.69, UCR 0.14, CLV 0.10. Gemini-3-Pro direct scored XE 0.13 and PF 0.51, and several baselines (TableGPT2, ToolLLM, ChatExcel-Max, TableGPT-Agent) produced no valid artifact bundle on at least some metrics or scored XE 0.00.
• Scaffolded fairness controls. Adding evaluator-facing scaffolds (profile summary, bundle templates, validator feedback, and a repair budget) lifted generic baselines; ReAct + scaffold reached XE 0.74 and CTC 0.70, and a G3P + claim-trace schema variant reached CTC 0.74 but only VRR 0.35. The ProCARE reference remained highest on VRR (0.81) and CTC (0.90) with the lowest UCR (0.06) and CLV (0.03), suggesting the advantage is not explained by schema exposure or manifest formatting alone.
• Which contract mechanisms matter (ablation). Removing the profile caused the largest damage: PF 0.49, XE 0.18, VRR 0.35, CTC 0.58, UCR 0.28, SL 0.42, with the dominant failure listed as field hallucination and invalid joins. Removing RWS-IR dropped XE to 0.34 with PF reported as N/A. Removing repair left VRR at 0.43 (unresolved runtime and artifact violations). Removing certificates left CTC as N/A and raised UCR to 0.26. Removing DAG ordering raised UCR to 0.11 and lowered SL to 0.64.
• Claim traceability rises with prompt specificity. Across 12 instances per prompt level, mean claim counts were 5.8 (Goal), 6.4 (Goal+Approach), and 6.9 (Goal+Method); CTC rose from 0.87 to 0.90 to 0.94, while UCR fell from 0.09 to 0.05 to 0.03 and CLV from 0.05 to 0.03 to 0.02. Overall (all 36 instances): 6.4 claims, CTC 0.90, UCR 0.06, CLV 0.03. The dominant downgrade reason shifted from unsupported causal/comparative wording to proxy endpoints beyond evidence level to sparse endpoint or subgroup evidence.
• Automated diagnostics agree with blinded experts. Across the 36 primary HCC instances, Spearman correlations were ρ = 0.82 for RWS-IR validity ((GA+PF)/2), 0.74 for code quality, 0.75 for visualization, and 0.71 for reporting, all with p < 0.001. Expert means on a 5-point scale were 4.38 ± 0.48 (RWS-IR, three clinical plus two statistics/data reviewers), 4.42 ± 0.21 (code, two statistics/data reviewers), 4.50 ± 0.53 (visualization), and 4.13 ± 0.64 (reporting).
• Cross-dataset transfer (Public-RWS-18). ProCARE beat both Claude Code and Scaffolded G3P on every reported transfer metric. On UCI Diabetes it scored XE 0.64, VRR 0.72, CTC 0.87, UCR 0.07, CLV 0.04; on NHANES XE 0.58, VRR 0.66, CTC 0.84, UCR 0.09, CLV 0.06; on TCGA-LIHC XE 0.61, VRR 0.69, CTC 0.86, UCR 0.08, CLV 0.05. NHANES remained the hardest because of multi-table survey

Authors’ abstract

Real-world studies (RWS) analyze routinely collected healthcare data, such as electronic health records and insurance claims, to estimate treatment effects, monitor safety, and identify risk factors outside randomized trials. Automating RWS with large language model agents is appealing but brittle: valid analyses depend on dataset-specific clinical semantics, including index dates, follow-up windows, patient/event-level units, and outcome boundaries that are often implicit in observational data. This leads to schema mismatches, temporal errors, inconsistent analytical granularity, and unsupported clinical claims. We present ProCARE, a profile-grounded framework that reformulates RWS automation as constrained compilation. ProCARE converts heterogeneous data profiles into Evidence Contracts specifying available variables, temporal attributes, distributions, and evidentiary limits, and compiles study objectives into RWS-IR, a strongly typed intermediate representation for variables, time relations, analysis units, and statistical tasks. Deterministic validation and localized repair then check and revise generated workflows across schema mapping, cohort construction, variable derivation, analysis, and interpretation. On 36 EHR-based benchmark tasks, ProCARE outperforms strong LLM-agent baselines in code executability, semantic consistency, and conclusion traceability, with automated violation diagnoses showing moderate-to-substantial agreement with blinded expert review.