Human-AI Co-design for Clinical Prediction Models

AI summary

Overview

Research area: Clinical artificial intelligence, specifically human-in-the-loop machine learning for clinical prediction models (CPMs), with connections to large language models (LLMs), electronic health records (EHR) and concept bottleneck models.

Technical level: Intermediate. The underlying statistics (linear, lasso-penalized and ridge-penalized logistic regression, AUC evaluation) are standard, and the paper is written to be readable by clinicians, but it assumes familiarity with LLM prompting and model validation concepts.

Scope: The paper introduces HACHI, a framework in which an AI agent and a clinical AI team co-design fully interpretable clinical prediction models by iteratively exploring concepts extracted from unstructured clinical notes, evaluated on two real-world prediction tasks (traumatic brain injury and acute kidney injury).

What This Paper Is About

Building a useful, fully interpretable clinical prediction model normally requires a long, laborious collaboration between clinicians, data scientists and informaticists, because many small decisions (which features and patients to include, how clinical categories are defined) determine whether the model works. This is hard enough with tabular EHR data, but clinical notes contain an essentially infinite number of possible concepts, so manually exploring them is infeasible, and models learned by an AI agent working alone may be clinically simplistic or implausible. The paper's goal is to show how a clinical AI team and an AI agent can collaborate iteratively so that the AI agent does the heavy lifting (analyzing notes, proposing concepts, fitting models) while humans steer the process with clinical judgment.

Key Contributions

1. The HACHI framework: an iterative human-in-the-loop co-design framework (Human+Agent Co-design for Healthcare Instruments) with an outer loop in which the clinical AI team gives guidance and an inner loop in which an AI agent searches for k concepts, formally defined as yes/no questions, that maximize a CPM's predictive accuracy.
2. A fully interpretable, AI-guided CPM learning procedure: a greedy hill-climbing procedure based on prior work that initializes a CPM by brainstorming concepts from keyphrases extracted from clinical notes, proposes candidate concepts, evaluates them by LLM annotation, and repeats steps 2 and 3 until convergence.
3. Two real-world case studies demonstrating improvement and new clinical insight: the framework was applied to pediatric traumatic brain injury (TBI) and postoperative acute kidney injury (AKI), where it outperformed comparator approaches, surfaced clinically relevant concepts absent from commonly used CPMs, and improved generalizability across clinical sites and time periods.
4. A characterization of the human role: the paper documents specific interventions by the clinical AI team, such as directing the AI agent to explore entire categories of concepts it had not considered, adjusting concept granularity, changing the objective function, and identifying data bias and leakage, plus an open-source, PHI-compliant review interface and released code at http://github.com/jjfenglab/HACHI.

Main Findings

• TBI performance and iteration over four rounds: Round 1 used default prompts and reached an AUC of 0.92. Round 2 fell to 0.90, Round 3 maintained 0.9, and Round 4 reached an overall AUC of 0.91. Error bars in figures show standard errors.
• Data leakage was discovered in TBI Round 1: the model learned "whether a note mentions Glasgow Coma Scale (GCS)," a spurious note-writing-style cue, and a "brain bleed" concept that is typically known only after a CT scan. Investigation revealed that a significant proportion of patients were transferred from another ED with an existing TBI diagnosis, so those patients were removed, leaving 304 remaining cases.
• Sign constraints aligned the model with clinical intuition in TBI: in Round 3, greedy concept selection was modified so that selection required the sign of the estimated coefficient to match the LLM's clinical prior, after the team noticed that a normal GCS had a positive coefficient.
• Site-level unfairness in TBI was addressed with sample weights: Round 3's campus-specific AUCs were 0.93 at Oakland and 0.71 at Mission Bay, with a 3:1 ratio of Oakland to Mission Bay patients. After reweighting the two campuses equally in Round 4, overall AUC was 0.91 and campus-specific AUCs were 0.93 and 0.80.
• TBI comparators were weaker: extracting PECARN attributes from the same clinical notes gave an AUC of 0.75, and simulating one-time expert brainstorming (features originally considered for PECARN plus features brainstormed by OpenEvidence) gave an AUC of 0.88, both substantially worse than HACHI. Specificity comparisons holding sensitivity constant are reported in Table A2 of the Appendix, where the HACHI model again outperformed comparators.
• A surprising retained concept: the final TBI model includes "head trauma," which made sense because in about 5% of cases there is a suspicion of unwitnessed head trauma with no reliable patient history (e.g., abandoned child, unclear mechanism of injury, no chief complaint on file).
• AKI performance improved across three rounds: AUC was 0.60 in Round 1, 0.70 in Round 2, and 0.73 in Round 3 on the internal validation set (Period 1).
• Temporal generalizability in AKI improved: on a temporally disjoint test dataset (Period 2, April–December 2024), AUCs were 0.61 in Round 1 and 0.77 and 0.76 in Rounds 2 and 3 respectively.
• AKI comparators were weaker: the unweighted and weighted Kheterpal models achieved AUCs of 0.65–0.66 using LLM extractions from preoperative clinical notes and 0.64 using tabular data from the EHR; a single-round brainstorming simulation using 30 concepts from OpenEvidence achieved 0.70. On the temporally disjoint test dataset, the Kheterpal models and the single-round brainstorming approach had similarly low AUCs of 0.67.
• Concept precision mattered: in AKI Round 3, requiring precise definitions with examples (e.g., "Does the note mention the patient having good exercise tolerance, e.g., having at least 5 METS of functional capacity or ability to climb two flights of stairs?") increased performance, and the concepts gained construct validity.
• Novel predictors were surfaced: the final AKI model included minimally invasive surgery, tachycardia and sleep apnea, which have not been commonly included in interpretable AKI risk scores; the final TBI model is a simpler form of PECARN using only a subset of its features plus the novel feature of "whether the patient has a normal gait."
• Human feedback was efficient: teams in both case studies spent approximately 1–2 hours reviewing results and providing feedback per round, with 3–4 rounds sufficient to reach a satisfactory model.

Methodology in Plain English

The team treats model building as a conversation between humans and an AI agent rather than a one-time automated pipeline. The AI agent is an LLM managed entirely through text prompts. It reads clinical notes and extracts keyphrases; it uses those keyphrases plus its own world knowledge to propose candidate concept questions ("Does this note mention the patient having...?"); it annotates each note with yes/no answers to those questions; and it fits a statistical model (lasso-penalized logistic regression) to see which concepts predict the outcome best. The agent runs a greedy hill-climbing search: it visits each of the k concept slots in turn and swaps in a better-performing candidate if one is found, running 10 iterations of the propose-and-evaluate loop. Because the process is stochastic, it is run with multiple seeds in parallel so differences between learned models reveal uncertainty and stability.

The humans sit in an outer loop. After each round, they review the learned factors, the patient- and note-level LLM annotations, which patients were misclassified, and performance on a held-out test set through a PHI-compliant, single-page, locally hosted web interface, and they meet to discuss improvements. Their main lever is editing the free-text prompts (readable by anyone, even those with limited AI expertise), though they can also change which observations are included, add statistical tools, or change sample weighting. Feedback is then translated into prompt and code updates for the next round, and this repeats until the model satisfies the team's goals. The paper reports that a per-round validation AUC did not improve significantly with more than 10 iterations in ablation studies.

For TBI, the team used a retrospective case-control dataset of 400 cases and 400 controls from two academic pediatric EDs within Benioff Children's Hospital (Oakland and Mission Bay campuses) between March 1, 2014 and December 31, 2024, using ED Triage Notes, ED Provider Notes and Nursing Notes, restricted to the History & Physical section of provider notes and to notes timestamped before the encounter's CT scan where a CT scan was performed, and chose a 5-concept CPM. For AKI, the dataset was sampled uniformly at random from all General Surgery patients between January 2016 and March 2024, with 800 cases and 800 controls, using preoperative anesthesia notes and defining the outcome as AKI within 7 days following surgery per KDIGO criteria (stage 1 or higher), with a 10-concept CPM chosen because the Kheterpal model used 11 clinical features.

Why This Matters

The work reframes how interpretable clinical prediction models can be built: not by fully automating model discovery nor by relying solely on slow manual collaboration, but by combining an AI agent that can search an essentially infinite space of note-derived concepts with human oversight that catches leakage, bias and clinical implausibility. It also argues that HACHI has minimal requirements (clinical notes, a PHI-compliant LLM, and guidance from a clinical AI team) and that deployment does not require LLMs at the bedside, since clinicians can annotate the yes/no questions during an encounter.

Real-world applications:

• Deciding which children with head trauma need CT imaging, where the goal is to reduce unnecessary ionizing radiation while not missing traumatic brain injury.
• Identifying patients at high risk of acute kidney injury after general surgery, enabling preventive measures such as optimizing hemodynamics, avoiding nephrotoxic medications and closer monitoring.
• Local model development for specific institutions and patient populations, using local notes rather than a single global model.
• Concept-level clinical documentation and review, since every learned concept is a reviewable yes/no question with an explicit coefficient.

Industry relevance: the framework is relevant to health systems and EHR vendors building local, auditable decision support; to developers of clinical AI products who need transparent, iteratively refinable models rather than black boxes; and more broadly to any domain with large amounts of unstructured text where interpretable predictive instruments are needed, which the authors suggest might be called HACHII ("Human+Agent Co-design for Highly Interpretable Instruments").

Future Directions

• Prospective validation: the models were developed and evaluated retrospectively, so prospective validation in real-time clinical workflows is needed to confirm they perform as expected.
• Extending to multimodal and temporal data: incorporating imaging, laboratory results or time series data into the framework, as prior concept bottleneck model work has done.
• Reducing human burden: exploring other frameworks for incorporating human feedback into co-design that maintain quality while reducing the effort required from clinical experts.
• Broader and multi-institutional testing: the studies were conducted at a single academic medical center with specific documentation practices and patient populations, and it remains to be determined whether a different clinical AI team would achieve similar performance and learn similar concepts; fairness, bias and equity also require continued attention, as does variability in LLM concept-extraction accuracy across models.

Target Audience

This paper is most valuable to clinical AI teams (clinicians, clinical data analysts and data scientists) who build or maintain clinical prediction models; to researchers working on human-in-the-loop machine learning, LLM-based concept bottleneck models and iterative concept refinement; and to health system leaders and industry practitioners evaluating how to make interpretable, auditable predictive tools practical to develop and deploy. Readers primarily interested in LLM prompt engineering, EHR data science, or AI-assisted clinical decision support will also find it useful.

Authors’ abstract

Developing safe, effective, and practically useful clinical prediction models (CPMs) traditionally requires iterative collaboration between clinical experts, data scientists, and informaticists. This process refines the often small but critical details of the model building process, such as which features/patients to include and how clinical categories should be defined. However, this traditional collaboration process is extremely time- and resource-intensive, resulting in only a small fraction of CPMs reaching clinical practice. This challenge intensifies when teams attempt to incorporate unstructured clinical notes, which can contain an enormous number of concepts. To address this challenge, we introduce HACHI, an iterative human-in-the-loop framework that uses AI agents to accelerate the development of fully interpretable CPMs by enabling the exploration of concepts in clinical notes. HACHI alternates between (i) an AI agent rapidly exploring and evaluating candidate concepts in clinical notes and (ii) clinical and domain experts providing feedback to improve the CPM learning process. HACHI defines concepts as simple yes-no questions that are used in linear models, allowing the clinical AI team to transparently review, refine, and validate the CPM learned in each round. In two real-world prediction tasks (acute kidney injury and traumatic brain injury), HACHI outperforms existing approaches, surfaces new clinically relevant concepts not included in commonly-used CPMs, and improves model generalizability across clinical sites and time periods. Furthermore, HACHI reveals the critical role of the clinical AI team, such as directing the AI agent to explore concepts that it had not previously considered, adjusting the granularity of concepts it considers, changing the objective function to better align with the clinical objectives, and identifying issues of data bias and leakage.