Timely Clinical Diagnosis through Active Test Selection

AI summary

Overview

Research area: Machine learning for clinical decision support, specifically combining Bayesian Experimental Design (BED) with large language models (LLMs) to choose which diagnostic tests to order next.

Technical level: Intermediate. The ideas are intuitive (reduce uncertainty before ordering a test), but the paper uses Bayesian belief updating, Kullback–Leibler (KL) divergence, and expected-information-gain criteria.

Scope: The paper proposes and evaluates a stepwise diagnostic framework in which an LLM simulates plausible test outcomes and a Bayesian criterion selects the test expected to reduce diagnostic uncertainty the most, tested on three real-world single-disease datasets plus a custom OSCE-style multi-condition dataset.

What This Paper Is About

Clinical diagnosis is sequential: clinicians gather information step by step, ordering tests that they hope will sharpen the diagnosis, while trying to avoid delays, over-testing, and unnecessary cost. Most machine learning models for diagnosis instead assume a static, fully observed dataset and classify in one shot, which does not match how clinicians actually reason under uncertainty and resource limits.

This paper asks whether an LLM can act as a flexible simulator of "what a test might show" for a given patient, and whether pairing that simulator with Bayesian Experimental Design can make test selection more accurate, more personalized, and less wasteful. The authors frame the framework as a clinician-in-the-loop decision-support tool rather than an autonomous diagnostician.

Key Contributions

1. The authors motivate and formalize a transparent, stepwise diagnostic framework that mirrors clinical reasoning, with explicit belief updates at each stage.
2. They propose a probabilistic approach to timely diagnosis that uses Bayesian Experimental Design with LLMs to adaptively select tests based on their expected diagnostic utility.
3. They show that shifting reasoning from the LLM's input space to the natural language output (solution) space can improve clinical decision-making.
4. They validate the framework on real-world datasets, reporting improvements in test selection, diagnostic accuracy, interpretability, and resource use relative to several baselines, including a full-information LLM classifier.

Main Findings

• LLMs approximate test-outcome distributions reasonably well. Given a patient's known information, both GPT-4o and GPT-4o-mini generated ten plausible values per missing laboratory feature. Comparing these to empirical distributions with normalized Wasserstein and Energy distances, GPT-4o produced the lowest average distances on every dataset: diabetes (0.110 ± 0.038 Wasserstein, 0.256 ± 0.079 Energy), hepatitis (0.130 ± 0.157, 0.265 ± 0.191), and kidney (0.082 ± 0.055, 0.203 ± 0.102). GPT-4o-mini diverged slightly more, consistent with its smaller capacity.

• Bayesian test selection beat the baselines on the harder diseases. Under a three-test-per-patient constraint, the framework outperformed the LLM classifier using all features, random selection, a globally best fixed three-feature subset, and implicit LLM-based selection on hepatitis C and diabetes, using the same underlying models and risk-prediction prompts.

• Fewer features sometimes meant better accuracy. On hepatitis and diabetes, the proposed method surpassed even the full-information baseline, which the authors attribute to targeted selection reducing the influence of irrelevant or misleading features.

• CKD was too easy to discriminate between methods. Chronic kidney disease, diagnosed via well-established biomarkers such as serum creatinine or glomerular filtration rate, was classified with near-perfect accuracy by both models even under the feature constraint, so the authors focused later analysis on hepatitis and diabetes.

• Implicit LLM selection lacked personalization. Compared against the globally optimal features identified without any patient data, implicit selection was strongly biased toward those global features with low variability across seeds, most dramatically in diabetes, where both models almost exclusively chose globally optimal features despite the proposed method achieving higher predictive accuracy.

• A KL-based stopping rule cut testing substantially. A test is acquired only if its expected KL divergence from the current belief to a target posterior exceeds a threshold, with target posterior defined as θ ± γδ where θ = 0.5 and δ = |p_prior − θ|. The authors report evaluating γ ∈ {0.3, 0.5, 0.7}; average tests selected ranged from 1.04 ± 0.24 (GPT-4o, kidney, γ = 0.7) to 2.67 ± 0.60 (GPT-4o, hepatitis, γ = 0.3). They report that the criterion reduces overall diagnostic burden by nearly 50 percent while providing comparable or superior accuracy to baseline feature selection. (One passage in the text refers to a "conservative threshold (γ = 0.6)" while the reported grid is {0.3, 0.5, 0.7}.)

• Clinicians found the recommendations reasonable. In a clinician-in-the-loop evaluation with three experienced clinicians and two senior medical students, covering 450 diagnostic test decisions (10 simulated diagnostic traces across three datasets, each reviewed by five evaluators with three decisions per trace), experts judged the test selections and resulting risk adjustments clinically reasonable in 94.5 ± 1.4 percent of cases. Participants said they would be reluctant to trust a purely black-box LLM without transparent reasoning, and several remarked that the Bayesian decision framework closely reflects their own reasoning when guidelines are absent.

• The approach transferred to multi-condition cases. On a custom OSCE-style dataset built from 114 representative AgentClinic cases with corresponding synthetic negatives created by adjusting laboratory values toward physiological ranges, the framework continued to outperform baselines, with the authors reporting superior calibration and diagnostic accuracy.

Methodology in Plain English

The framework treats diagnosis as a loop of belief updating and test choice, with the clinician in the driver's seat.

At each stage, the system starts from a prior belief: the current probability that the patient has the target disease, given everything known so far (demographics, prior results, other findings). It then considers each candidate test that has not yet been done.

For every candidate test, the LLM acts as a surrogate simulator. Instead of requiring a mechanistic model of human physiology, the LLM is prompted with the clinical context, the known information formatted as a vignette, and a task instruction, and it generates multiple plausible results the test might produce for this patient. Each hypothetical result is turned into a hypothetical posterior probability of disease, using the same LLM to update the belief.

The system then measures how much each test would move the belief, using the expected Kullback–Leibler divergence between the posterior and the prior (modeled as Bernoulli distributions over the disease label). This captures epistemic uncertainty, the kind reducible by new information, rather than inherent randomness.

To reflect resource limits, the information gain is divided by a cost term (with cost modeled on a logarithmic scale), giving a utility score = expected information gain / cost. The test with the highest utility is selected. The clinician reviews the suggestion, can override it, performs the test, and the real result is added to the knowledge base before the loop repeats.

Stopping is built into the same criterion. If no remaining test is expected to shift the belief enough relative to a target posterior near the decision boundary, testing stops and the current belief is used. To keep the LLM's simulated outcomes realistic and diverse, the prompts discourage population-average answers, sampling temperature is increased, and the model is asked to sample outcomes under both disease presence and absence.

Baselines for comparison were an LLM classifier with all features available, random selection of three features, a globally best fixed three-feature subset chosen before seeing any patient, and implicit LLM selection of three features without Bayesian modeling.

Why This Matters

Impact on research. The paper sits at the intersection of two active lines of work: using LLMs as general-purpose medical reasoners, and Bayesian Experimental Design for information-efficient decision-making. Its central claim is that LLMs are better used as generative simulators inside a principled probabilistic loop than as direct classifiers or implicit test pickers. It also reports a negative result with practical weight: implicit LLM selection tends to fall back on globally popular tests rather than personalizing, which is a concrete diagnostic of where LLM reasoning breaks down.

Real-world applications:

• Resource-limited clinics. The WHO projects a shortage of more than 12 million qualified health professionals by 2035. A tool that recommends the single most informative next test, and stops when further testing would not change the picture, targets exactly this constraint.
• Reducing unnecessary testing. The paper cites estimates that 40 to 60 percent of diagnostic tests are unnecessary, and that around 15 percent of clinician-ordered genetic tests go unperformed due to financial barriers. Fewer, better-targeted tests speak directly to both waste and access.
• Non-trivial diagnostic cases. Clinicians in the study noted that decision support is most valuable where genuine uncertainty exists and no clear pathway is defined, such as distinguishing hepatitis C from other causes of subtle liver biomarker changes, or synthesizing indirect indicators for diabetes.
• Clinical training and assessment. The authors position the OSCE-style evaluation as evidence that sequential Bayesian reasoning could be integrated into assessment frameworks such as Objective Structured Clinical Examinations.

Industry relevance. For developers of clinical decision-support software, the paper outlines an architecture that is model-agnostic: performance depends on the quality of the surrogate model rather than on a specific LLM. That means the same pipeline can be run on smaller or larger models, which the authors demonstrate with GPT-4o-mini and GPT-4o. The emphasis on intermediate outputs, clinician override, and restricting recommendations to clinically approved tests for the suspected condition speaks to the transparency and safety requirements that regulators and hospital systems impose on diagnostic tools.

Future Directions

• Scale and heterogeneity. The authors note their evaluation is limited to approximately 1,000 patients combined across all datasets, with a limited number of covariates per condition, and call for larger, more heterogeneous datasets with richer feature spaces and overlapping comorbidities.
• Beyond binary diagnosis. The framework is currently limited to binary classification and each task is a binary decision. Extending to multi-label datasets and co-morbidities is flagged as necessary for broader clinical applicability.
• Unstructured data. Only categorical and numerical features are handled; free-text outputs such as imaging or pathology reports are not yet incorporated, though structured representations could be added.
• Cost functions and computational expense. Uniform test costs were used because task-specific costs would require expert clinical input, so explicitly expert-defined cost functions remain open. The method also requires more LLM queries than simpler heuristics, which the authors acknowledge may be challenging in resource-constrained environments, and it depends on strong, high-capacity LLMs to produce physiologically coherent distributions.

Target Audience

This paper is most useful to machine learning researchers working on sequential decision-making, Bayesian Experimental Design, or LLM-based clinical reasoning; to clinical informatics and health-system teams evaluating decision-support tools; and to clinicians interested in how uncertainty-aware test selection could complement or challenge existing guidelines. Readers without a background in probability will still follow the intuition, but the KL-divergence formulation and stopping rule are aimed at readers comfortable with Bayesian inference.

Authors’ abstract

There is growing interest in using machine learning (ML) to support clinical diagnosis, but most approaches rely on static, fully observed datasets and fail to reflect the sequential, resource-aware reasoning clinicians use in practice. Diagnosis remains complex and error prone, especially in high-pressure or resource-limited settings, underscoring the need for frameworks that help clinicians make timely and cost-effective decisions. We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design), a diagnostic framework that integrates Bayesian Experimental Design (BED) with large language models (LLMs) to better emulate real-world diagnostic reasoning. At each step, ACTMED selects the test expected to yield the greatest reduction in diagnostic uncertainty for a given patient. LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data. Clinicians can remain in the loop; reviewing test suggestions, interpreting intermediate outputs, and applying clinical judgment throughout. We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use. This represents a step toward transparent, adaptive, and clinician-aligned diagnostic systems that generalize across settings with reduced reliance on domain-specific data.