ALLMTS: Automated LLM-based Medical Triage System

AI summary

Overview

Research area: Clinical artificial intelligence — specifically large language model (LLM) systems for telephone and telehealth triage, combining retrieval, schema-constrained extraction, and executable clinical guidelines.

Technical level: Advanced. The paper assumes familiarity with LLM pipelines, retrieval-augmented generation, structured/schema-constrained output, cohort-level error metrics (under-triage, over-triage, weighted F1), and inter-rater agreement statistics such as Cohen's and Fleiss' kappa.

Scope (one sentence): The paper describes and evaluates ALLMTS, a pipeline in which an LLM executes a commercial telehealth triage guideline corpus rather than predicting a disposition directly, benchmarked against raw-LLM and web-search baselines on 566 expert-validated synthetic scenarios plus a released 50-scenario open benchmark.

What This Paper Is About

Telephone triage is a safety-critical decision: from a short intake note, a nurse must decide whether a caller needs an ambulance, an emergency department, an appointment, or self-care at home. Queried directly, LLMs are unreliable — the paper reports that they over-escalate routine cases and cannot justify recommendations against an auditable clinical standard. The goal of ALLMTS is to make the LLM execute the same commercially maintained triage guidelines nurses use, so that the disposition follows deterministically from a specific guideline question and answer instead of being generated by the model.

Key Contributions

1. A guideline-executing triage architecture. ALLMTS operationalises a corpus of 314 adult telehealth triage guidelines as an LLM pipeline that combines metadata pre-filtering and dense retrieval for guideline selection, runtime schema-locked evaluation of Triage Assessment Questions (TAQs), a deterministic first-positive-TAQ disposition rule, and guideline-to-guideline SMAG (See More Appropriate Guideline) re-routing with a default depth limit of 5. The LLM performs only language understanding; every clinical decision remains traceable to a guideline question.

2. An expert-validated evaluation and an open benchmark. Evaluation uses 566 scenarios spanning 14 disposition labels, authored by the clinical content team at LaGrange Medical Software (Schmitt-Thompson Clinical Content) through a 14-step process and validated by a panel of five expert telehealth triage nurses. A curated 50-scenario subset is publicly released with reference dispositions as STCC Adult After-Hours Telehealth Triage 2026 – 50 Clinical Scenarios.

3. A clinically grounded comparison. The system is benchmarked against a raw-LLM baseline and a web-search baseline, plus a guideline-context ablation, across four label granularities (fine-grained disposition, Acuity, Urgency, Urgency Group), reporting accuracy, support-weighted F1, under-triage and over-triage rates, and run-to-run stability.

4. Reproduction of prior LLM triage methods. Two prior evaluations (Levine et al. and Gaber et al.) are re-run verbatim on the same backbone model and projected into a shared label space, isolating method effects from model effects.

Main Findings

• Full dataset accuracy: On the 566-scenario set, ALLMTS reaches 0.95 disposition accuracy (weighted F1 0.96) against 0.47 (F1 0.48) for the raw LLM and 0.50 (F1 0.51) for web search. Across the projected views it reaches up to 0.96, versus a 0.70 best baseline view ceiling.

• Over-triage reduction: On the full set, the raw LLM over-triages 0.44 of cases and web search 0.39, while ALLMTS over-triages 0.02. On the 50-scenario subset, baseline over-triage rates reach up to 0.50, whereas ALLMTS's only error is a single under-triage with no over-triage.

• Human comparison: On the released 50-scenario subset, ALLMTS reaches 0.98 accuracy, on par with the 0.944 correct disposition rate recorded by the five-nurse expert panel on the same scenarios.

• Guideline context does most of the work: The ablation that pastes the full selected guideline into the prompt without TAQ traversal reaches 0.90 ad-hoc disposition accuracy on the full set, versus 0.95 for ALLMTS and 0.47/0.50 for the baselines. Explicit TAQ evaluation and SMAG routing add a further 0.02 to 0.05 in every like-for-like cell; the gap widens where the ablation predicts a coarse label directly (ad-hoc Urgency 0.89 and Urgency Group 0.86, versus ALLMTS's projected 0.95 and 0.96).

• Errors are systematic, not noise: Over 10 repeated runs on the Acuity view of the 50 scenarios, ALLMTS averages 0.966 ± 0.018 accuracy with Cohen's kappa against the reference of 0.955, versus 0.658 ± 0.024 and 0.541 for the raw LLM and 0.706 ± 0.031 and 0.605 for web search. Self-agreement is high for every method (Fleiss' kappa ≥ 0.84), so the baselines converge on the same over-triaged label even when wrong. 47 of 50 scenarios are correct on every ALLMTS run and none is wrong on all ten, against 28 and 12 for the raw LLM.

• Web search is the least stable method: It has the lowest self-agreement (Fleiss' kappa 0.844) and a flip rate of 0.26, and it is only unevenly better than the raw LLM (ahead on full-set disposition and Acuity, behind on Urgency, and behind on both disposition and Urgency at n = 50). It stays at least 0.45 below ALLMTS on disposition.

• Prior methods over-triage: On the 50-scenario set with gpt-5.2, Levine's method scores 0.70 accuracy and Gaber's 0.58, with over-triage rates of 0.20 and 0.24; their most common failure is escalating an urgent case to an emergent one.

• External vignette performance drops: On in-scope subsets of two external sets, ALLMTS reaches only 0.41 accuracy on Levine et al. (43 of 48 vignettes) and 0.58 on Semigran et al. (37 of 45), far below its 0.95–0.98 on its own data; clinical experts judged those vignettes too thin to reach a disposition without a clarifying question.

• A small residual failure mode: ALLMTS can return "No positive TAQ found," which occurred in 9 of 566 cases (1.6%) and is counted as under-triage and as discordant.

Methodology in Plain English

The system never asks the model "what disposition does this patient need?" Instead, it hands the model a job it can do reliably — reading a note and answering yes/no questions — and computes the disposition itself with a fixed rule.

The pipeline works in stages. First, two extraction steps read the free-text intake note: one pulls out age and biological sex, taking only explicitly stated values and leaving fields null when unstated (if neither is recoverable, the guideline pre-filter is bypassed entirely); the other pulls out the reason for visit as a primary reason plus a list of secondary ones, each with a type (Exposure, Injury, Pain, or Symptom) and optional onset. Second, dense retrieval over the 314 guidelines narrows the corpus to the top-k candidates (k = 50) by cosine similarity, because the full corpus cannot fit in context; a selection model then reads each candidate's title, definition, category, gender, type, system, and anatomy alongside the scenario and returns exactly one title, or none if nothing is clinically relevant. Third, every TAQ of every disposition level is evaluated concurrently, each disposition with its own answer schema so the model can only answer the questions as written; answers are boolean under a closed-world assumption, so anything the note does not mention counts as false. Compound questions are split and folded with standard boolean logic, and exception clauses force an answer to false. Fourth, the checklists are merged from highest to lowest acuity and the first positive TAQ fixes the disposition — the same traversal rule a nurse follows. If that TAQ links to an alternate guideline, the pipeline switches and re-evaluates, up to depth 5, with a visited set preventing cycles. Every prediction returns the disposition, the selected guideline, the full answered checklist, and retrieval and SMAG metadata.

The evaluation data was built by first having a large language model draft scenarios under structured prompts, then having a Nurse Editor review and refine each one, a second Nurse Editor review independently, disagreements resolved by revision or discard, and survivors validated by a five-nurse expert panel and approved by the Senior Medical Editor. Scenarios are a median of 81 words long, follow a chief-complaint/history/review-of-systems/past-medical-history format, and are heavily imbalanced: five dispositions carry 12–15% of volume each, while six rare ones cover at most seven scenarios each, two occurring exactly once — which is why F1 is weighted by support. All methods used gpt-5.2 through the Azure OpenAI Responses API at low reasoning effort; retrieval used Azure-hosted text-embedding-3-large (d = 3072) with the title representation and L2-normalised embeddings.

Why This Matters

Impact on research. The paper makes a design argument that is testable and reproducible: for safety-critical clinical tasks, the model should be a language-understanding component inside an auditable procedure, not the clinical authority. It supplies an open 50-scenario benchmark, a clinically decisive error taxonomy (under-triage and over-triage rather than accuracy alone), and a stability analysis showing that single-run comparisons can conflate miscalibration with sampling noise. The finding that handing the model the guideline verbatim already recovers most of the gain (0.90 versus 0.95) sharpens where the remaining value of structured execution actually lies.

Real-world applications:

• After-hours telehealth and nurse triage lines, where intake notes are already written and a disposition must be assigned consistently under call volume.
• Clinical documentation and decision-support tooling that must produce an auditable justification tied to a named guideline question rather than free-form reasoning.
• Emergency department demand management, since the baselines' dominant failure is over-triage (0.44 for the raw LLM on the full set), which the paper frames as flooding already crowded emergency departments.
• Quality assurance and audit workflows, where the returned checklist, retrieval metadata, and SMAG chain allow a reviewer to trace why a case was escalated or de-escalated.

Industry relevance. The work was performed by an organisation acting as a software development partner to LaGrange Medical Software under a nondisclosure agreement, and the guidelines remain the exclusive intellectual property of LMS. That framing matters commercially: the approach lets a guideline vendor keep its curated content as the source of clinical authority while an LLM supplies the language layer. The paper is explicit that this creates circularity in evaluation and limits reproducibility, and that generic web search is not a substitute for a maintained corpus.

Future Directions

• An interactive variant. The closed-world assumption means an omitted finding and an explicitly negative one are indistinguishable. The authors state that a variant with an explicit unknown state, triggering clarification questions to the caller or deferral to a triage nurse, is in development, with its evaluation left to future work.

• Evaluation beyond guideline-aligned scenarios. Accuracy fell to 0.41 on the in-scope Levine et al. vignettes and 0.58 on Semigran et al., and experts judged those vignettes too thin to judge without clarification questions — the authors flag the need for further evaluation outside guideline-aligned scenarios.

• Addressing the circularity of evaluation. The system was built around the STCC guidelines and evaluated on a dataset built for the same guidelines; the paper acknowledges this directly and notes the limited reproducibility that proprietary content imposes.

• Reducing residual failure modes. The "No positive TAQ found" outcome occurred in 9 of 566 cases (1.6%) and counts automatically as discordant, and the paper reports no per-stage ablation because removing any single stage leaves traversal undefined rather than merely degraded. The choices that can vary while keeping traversal intact — backbone model, reasoning effort, and retriever configuration — are the documented degrees of freedom for future work.

Target Audience

Researchers and engineers building clinical decision-support or triage systems on LLMs; clinical informatics and telehealth operations teams evaluating whether LLM automation is safe enough to deploy; guideline publishers and health IT vendors interested in keeping curated content as the auditable source of clinical authority; and evaluation-focused ML researchers who need a benchmark with clinically meaningful error categories rather than accuracy alone. Readers without a background in LLM pipelines or clinical triage metrics will find the architecture description accessible but the results tables dense.

Authors’ abstract

Telephone triage is a safety critical task: from a brief intake note, a nurse must decide whether a caller needs an ambulance, an emergency department, an appointment, or self care at home. Queried directly, large language models (LLMs) are unreliable: they over-escalate routine cases and cannot justify their recommendations against an auditable clinical standard. We present the Automated LLM-based Medical Triage System (ALLMTS), which grounds every disposition in the same commercially maintained telehealth guidelines nurses use. Rather than asking an LLM to triage, ALLMTS makes it execute the guideline: it selects the appropriate guideline and evaluates its triage assessment questions. The disposition follows deterministically from the first positive answer, using the same traversal rule a nurse follows. We evaluate ALLMTS on 566 synthetic, expert-validated scenarios spanning 14 dispositions and designed to test specific guideline questions. ALLMTS achieves 0.95 accuracy, compared with 0.47 for a raw-LLM baseline. On a curated 50-scenario subset released as an open benchmark, ALLMTS reaches 0.98 accuracy, on par with the 0.944 correct disposition rate that a five nurse expert panel recorded on those same scenarios. Critically, on that subset the baselines have over-triage rates of up to 0.50 (0.44 on the full dataset), whereas ALLMTS reduces the over-triage rate to at most 0.02 while maintaining low under-triage. These results support coupling LLM language understanding with executable clinical guidelines for triage from intake notes.