Trajectory-Derived Confidence for Reliable, Resource-Aware Clinical Text-to-SQL Agents

AI summary

Overview

Research area: Natural language processing, specifically reliability and uncertainty monitoring for LLM-based agents that translate clinical questions into SQL over electronic health record databases.

Technical level: Intermediate. The paper combines a formal decision-theoretic framing (Chow's rule) with a practical classifier-based monitoring layer, and reports standard benchmark metrics without requiring deep mathematical background.

Scope: The paper introduces and evaluates Sentinel, a confidence layer built from two small text classifiers that read an unmodified ReAct agent's own reasoning transcript and decide when to refuse, stop, or withhold at three points in the trajectory.

What This Paper Is About

LLM agents that answer clinical questions by writing SQL over patient databases can reason over many steps but cannot tell whether their own reasoning or outputs should be trusted. A wrong lab value or cohort size is delivered in the same fluent, confident register as a correct one, and an answer to a question the database cannot support is fabricated rather than refused. The paper's goal is a reliability layer that wraps an existing agent without modifying it, deciding whether to abstain at three points (before execution, mid-trajectory, and at delivery) in a way that reflects how costly a delivered mistake is, while also saving the computation that doomed trajectories would have consumed.

Key Contributions

1. Stakes-aware reliability. The paper formulates reliability for clinical text-to-SQL agents as a stakes-dependent decision problem in which the cost of an incorrect answer can exceed the cost of abstaining, using the EHRSQL Reliability Score RS(λ) at λ ∈ {1, 4, 9}.

2. Trajectory-derived monitoring. Sentinel scores the agent's evolving trajectory rather than only its final output or self-reported confidence, operating before execution, during reasoning, and at delivery, across five different agent models.

3. Resource-aware early stopping. The same confidence estimates are shown to cut computation by stopping trajectories unlikely to produce a useful answer, reducing reasoning steps and latency while preserving reliability at several operating points.

Main Findings

• Reliability gains at low stakes, per agent. On the held-out pool, GPT-4o's RS(1) rises from +0.08 for the vanilla agent to +0.24 for the full system, and GPT-4o-mini's rises from −0.02 to +0.21, both marked as gains over always-abstaining whose 95% bootstrap interval excludes zero. Llama-3.1-8B goes from −0.23 to +0.06, Qwen2.5-Coder-7B from −0.46 to +0.10, and Phi-4-mini from −0.46 to +0.04, with the last two significant over always abstaining.

• The always-abstain reference point. A system that refuses every question scores the unanswerable fraction of the pool, reported as +0.03 at every stakes level, because abstention is never penalized. Gains over this baseline are significant at λ = 1 for four of the five agents and indistinguishable from zero elsewhere.

• Higher stakes trigger near-total abstention. At λ = 4 and λ = 9 the system delivers little or nothing and matches always abstaining within noise, while the unmonitored GPT-4o scores −3.02 at λ = 9. The authors read this as a gating result: at those stakes no tested agent produces enough reliable answers to deploy, and the layer reaches that verdict from the trajectories alone.

• Chow's rule does most of the work. Of the two checks, the delivery-time rule accounts for nearly all of the gain; the answerability gate on its own moves RS(λ) by +0.01 to +0.14, since unanswerable questions are only 3% of the constructed pool (versus 20–33% in EHRSQL's own evaluation splits, where the gate's share grows). As a single binary withhold classifier, the layer separates wrong deliveries from right ones at 0.76 AUC on held-out committed answers.

• Accuracy rises as coverage falls. GPT-4o delivers 55% of questions at 69% accuracy under the full system, against 84% coverage at 54% accuracy for the vanilla agent, and is wrong on 46% of the answers it delivers without the layer.

• Resource savings. At λ = 1, where the system still answers over half of the questions, early exit eliminates 13% of GPT-4o's agent steps with no loss in reliability, and 13–28% across agents at a cost of at most 0.01 RS(1). At λ = 4 and λ = 9 the saving rises to 42% for GPT-4o (up to 70% on other agents), but there the system delivers almost nothing. Table 2 reports 16% / 47% / 47% of generated tokens saved and 29 min / 1.5 h / 1.5 h of wall-clock time saved at λ = 1/4/9, at a measured 1.9 seconds per agent step, with the monitor's two ModernBERT-base passes per step not subtracted.

• The signal lives in the trajectory, not the question. Read on the question alone, the correctness head scores 0.705 AUROC for GPT-4o, reaching 0.762 at delivery; on Qwen2.5-Coder-7B a single step of execution evidence lifts it from 0.742 to 0.898. Training-free telemetry does not suffice — mean token entropy reaches only 0.565 for correctness on GPT-4o-mini.

• Calibration makes the thresholds meaningful. The raw correctness head is overconfident, but isotonic maps fit on the calibration pool hold on held-out data, with held-out ECE of 0.03–0.07 versus 0.25 raw. For GPT-4o the calibrated delivery estimate exceeds 0.9 on only three held-out questions, which is why the system abstains almost entirely at λ = 9.

• Monitorability tracks how legibly an agent fails, not how capable it is. The two agents whose trajectories fill with execution errors and fast fabrications are easiest to read (Qwen2.5-Coder-7B at 0.920 and Phi-4-mini at 0.921), while GPT-4o, whose rare failures resemble its successes, and Llama-3.1-8B, which meanders through the longest trajectories, are hardest (0.762 and 0.707). Phi-4-mini fabricates answers to half of its unanswerable questions, and the layer's verdict is near-total refusal; Qwen2.5-Coder-7B answers 84% of questions at 22% accuracy, and the layer still delivers 21% of questions at 68% accuracy at λ = 1.

• Wasted steps on unanswerable questions. A vanilla GPT-4o agent spends 2.5 reasoning steps per unanswerable question on average; when it fabricates an answer it takes 3.35 steps, 58% more than the 2.12 steps it takes on average when it correctly refuses.

• Reported limitations. The primary correctness label is an LLM judge, and a two-rater audit found it agreeing with the raters on 69 of 93 cases (74%, 95% CI 64–82%; Cohen's κ = 0.86 for inter-rater agreement), with 1,294 of 5,119 answered questions (25.3%) flipped from strict-wrong to semantically correct. Evaluation is limited to a single benchmark and database; EHRSQL's unanswerable questions come from roughly 16 short templates (median 17 characters) versus roughly 100 full-question answerable templates, and a question-only baseline already reaches 0.955 AUROC on answerability. No competing confidence method was run and only one training seed was used; the heads are trained per agent, so a new agent requires a new collection run.

Methodology in Plain English

The researchers leave the agent untouched and wrap it in a separate observer. A frozen ReAct agent processes each question as a growing transcript: its reasoning text, any SQL it writes, and the execution result observed. Two ModernBERT-base text classifiers read a plain-text rendering of that transcript at every prefix, including the very first state containing only the question.

The answerability head estimates whether the question can be answered from the database at all, and its reading at time zero powers a gate that can refuse a question before the agent runs a single step. The correctness head estimates whether the trajectory will end in a delivered correct answer; it is trained on answered answerable questions and then recalibrated with isotonic regression so its scores behave like honest probabilities.

Three decisions follow. First, an exit gate fires the moment the answerability estimate drops below a threshold set as the largest value that wrongly exits at most 0.5% of answerable questions on a calibration pool. Second, in resource-aware mode, the trajectory is stopped mid-run when the calibrated correctness estimate falls below an empirically chosen stop threshold. Third, at delivery, the committed answer is checked against Chow's rule: deliver only if the calibrated probability of being correct meets λ/(1+λ), which gives thresholds of 0.5, 0.8, and 0.9 for λ = 1, 4, and 9. Because abstention is never penalized under the Reliability Score, both stopping and withholding can only convert a penalized output into one worth at least zero.

The setup uses EHRSQL on MIMIC-III. From the official train and validation splits the authors build three pools with no question template shared between them: a train pool of 6,179 questions (231 unanswerable), a calibration pool of 2,160 (71), and a held-out pool of 2,101 (60) evaluated exactly once, leaving the official test split untouched. Five agent models run under an identical harness: GPT-4o, GPT-4o-mini, Qwen2.5-Coder-7B, Llama-3.1-8B, and Phi-4-mini. All reported training-pool estimates are out-of-fold, using 5-fold cross-validation grouped by question template. Correctness labels come primarily from a semantic scheme in which GPT-4o-mini judges whether the agent's executed result is equivalent to the gold query's executed result; exact matches are auto-correct. A pre-registered rule selects per agent between the correctness head alone and a gradient-boosted hybrid of the head with ten trajectory features, preferring the simpler option when scores are within 0.005.

Why This Matters

Impact on research. The paper reframes agent reliability as a selective-prediction problem applied at two checkpoints of a trajectory rather than to a single prediction, and shows that externally grounded trajectory signals — executed SQL, error messages, empty result sets, failed schema references — carry information that verbalized confidence does not. It also argues that an agent's stated chain-of-thought is not a dependable indicator of its behavior, and that reliability can be added on top of an agent rather than trained into it.

Real-world applications.

• Clinical decision support, where a fabricated lab value or medication count delivered with false confidence can directly harm a patient, and where declining to answer is often the cheaper error.
• Reducing cost and latency of agentic systems, since every agent step is a full LLM call over a growing context and each avoided step is also one fewer live SQL query against a production database.
• Auditing or gating already-deployed agents without retraining them, using a monitor trained on that agent's own past trajectories.
• Triage of borderline questions, where a system that abstains on low-confidence items routes them to a human reviewer instead of presenting them as facts.

Industry relevance. The results speak directly to the operational economics of agentic systems: at λ = 1 the layer saves 13% of steps with no measured reliability cost, and the paper notes the monitor's own compute is not subtracted from those savings. The finding that the loudest, most error-strewn agents are the easiest to monitor, while capable agents whose rare failures look like their successes are hardest, is a practical caution for anyone monitoring frontier models in production.

Future Directions

1. Let the confidence signal act, not just abstain. At training time the monitor could serve as a cheap verifier that labels preference pairs or supplies rewards for DPO- or GRPO-style fine-tuning, though the authors note an agent trained against a fixed monitor may learn to look confident rather than be correct, which needs testing.

2. Replace stopping with resampling. At inference time, a budget policy could resample a low-confidence trajectory and rerank candidates rather than halting it.

3. Test transfer to a new database such as eICU, since a monitor that transfers across schemas would be far more useful than one rebuilt per database. The authors also flag that a monitor frozen at deployment faces an agent whose behavior may drift from the training distribution.

4. Strengthen and broaden evaluation. The paper is limited to one benchmark and database, uses a single training seed, runs no competing confidence method, and relies on an LLM judge whose errors on 26% of its hardest cases propagate to every reported number. It also warns that as agents improve and their failures get quieter, the signal this approach reads may fade.

Target Audience

This paper is most useful to researchers and engineers working on LLM agents in high-stakes domains, particularly clinical and healthcare NLP, and to practitioners who need to add abstention, monitoring, or cost control to an agent they cannot retrain. It also suits readers interested in selective prediction, uncertainty calibration, and the reliability economics of agentic systems. Readers looking for a new agent architecture or a new benchmark will not find one here; the contribution is a monitoring layer, its decision-theoretic justification, and an empirical audit of when it works and when it may not.

Authors’ abstract

LLM agents for clinical text-to-SQL applications reason autonomously over multiple steps but cannot assess whether their own reasoning or outputs can be trusted. In high leverage applications such as healthcare, this presents a critical risk where system mistakes can be costly. These reliability failures are also resource failures: an incorrect reasoning trajectory spends computation budget on outputs that must be discarded. We introduce Sentinel, a trajectory-derived, classifier-based confidence layer that analyzes an agent's reasoning, code and database outputs to decide at three points whether to stop: refusing unanswerable questions before the agent runs, halting doomed trajectories mid-run, and withholding untrustworthy answers at delivery. Here, utilizing Chow's rule, we optimize decisions under the EHRSQL shared task's Reliability Score, which penalizes incorrect answers given a utility weighting, and find on the benchmark EHRSQL that Sentinel raises this score from +0.08 to +0.24 when mistakes have a low utility weighting, well above the +0.03 earned by refusing every question, with delivered-answer accuracy rising from 54% to 69% as coverage falls from 84% to 55%. At higher stakes the agents we test rarely answer reliably enough to deliver, and Sentinel detects this on its own, abstaining to that same +0.03 where the unmonitored agent scores -3.02. The same stopping decisions cut computation: at low stakes, where the system still answers, Sentinel eliminates 13-28% of agent steps at little or no reliability cost.