Multi-agent Self-triage System with Medical Flowcharts

AI summary

Overview

Research area: AI for healthcare, clinical decision support, multi-agent LLM systems, and conversational self-triage.
Technical level: Intermediate.
Scope: This paper presents a proof-of-concept multi-agent LLM self-triage system guided by 100 clinically validated American Medical Association flowcharts and evaluates its retrieval and navigation performance on synthetic conversations.

What This Paper Is About

Online health resources and large language models are increasingly used for initial medical decision-making, but they can be inaccurate, opaque, and vulnerable to unverified information. The paper builds a conversational self-triage system that combines free-text patient interaction with structured, clinically validated flowcharts to produce transparent and auditable recommendations. The goal is to test whether this approach can make AI-assisted self-triage more accurate, reliable, and generalizable.

Key Contributions

1. Introduces a multi-agent self-triage architecture with three specialized agents: a retrieval agent, a decision agent, and a chat agent.
2. Integrates 100 clinically validated flowcharts from the American Medical Association as a structured, auditable framework for LLM-guided patient decision support.
3. Evaluates the system at scale using synthetic datasets of simulated conversations across varied conversational styles and medical conditions.
4. Demonstrates high retrieval and navigation accuracy, supporting the feasibility of transparent, protocol-grounded AI self-triage.

Main Findings

• High flowchart retrieval accuracy: The system achieved 95.29% top-3 accuracy in retrieving the most relevant flowchart across 2,000 cases.
• Strong flowchart navigation accuracy: It achieved 99.10% accuracy in navigating flowcharts across 37,200 simulated conversations with varied styles and conditions.
• Effective multi-agent division of labor: The retrieval agent identifies the relevant flowchart, the decision agent interprets patient responses, and the chat agent delivers personalized, patient-friendly recommendations.
• Combines flexibility with rigor: Free-text conversation is paired with standardized clinical protocols, preserving natural interaction while making decisions more transparent and auditable.
• Proof-of-concept evidence: Evaluation used synthetic conversations, so the results support feasibility but do not yet establish real-world clinical safety or effectiveness.

Methodology in Plain English

The researchers selected 100 clinically validated medical flowcharts from the American Medical Association. They built a multi-agent system in which one agent finds the most relevant flowchart for a patient’s described symptoms, another agent follows the flowchart logic and interprets the patient’s answers, and a third agent turns the result into clear, patient-friendly guidance. To test the system, they generated large synthetic datasets of simulated patient conversations, including different conversational styles and medical conditions. They then measured whether the system retrieved the correct flowchart in its top three choices and whether it navigated the flowchart correctly.

Why This Matters

Research impact: The work offers a pattern for grounding LLM-based healthcare tools in auditable clinical protocols, which may reduce reliance on unverified model knowledge and improve transparency in medical AI.

Real-world applications:

• Patient-facing self-triage before contacting a clinician or emergency service.
• Telehealth intake and symptom routing to the appropriate care level.
• Nurse triage lines and front-door clinical decision support.
• Standardized protocol navigation in clinical workflows, documentation, and care coordination.

Industry relevance: The approach is relevant to digital health companies, telehealth platforms, EHR vendors, providers, and insurers interested in safer AI triage, better patient decision-making, and more efficient healthcare resource utilization. It also speaks to regulators and AI governance teams focused on transparency, auditability, and clinical validation.

Future Directions

• Validate the system with real patient conversations and clinical outcomes, rather than only synthetic data.
• Test generalizability across more flowcharts, medical specialties, languages, populations, and rare or ambiguous cases.
• Evaluate safety, bias, failure modes, escalation behavior, and integration with human clinicians.
• Conduct prospective clinical trials and address regulatory, privacy, and deployment requirements before real-world use.

Target Audience

AI and ML researchers working on LLM agents, clinical informatics specialists, digital health product teams, physicians and nurses involved in triage, healthcare regulators and policymakers, and students interested in trustworthy AI for healthcare.

Authors’ abstract

Online health resources and large language models (LLMs) are increasingly used as a first point of contact for medical decision-making, yet their reliability in healthcare remains limited by low accuracy, lack of transparency, and susceptibility to unverified information. We introduce a proof-of-concept conversational self-triage system that guides LLMs with 100 clinically validated flowcharts from the American Medical Association, providing a structured and auditable framework for patient decision support. The system leverages a multi-agent framework consisting of a retrieval agent, a decision agent, and a chat agent to identify the most relevant flowchart, interpret patient responses, and deliver personalized, patient-friendly recommendations, respectively. Performance was evaluated at scale using synthetic datasets of simulated conversations. The system achieved 95.29% top-3 accuracy in flowchart retrieval (N=2,000) and 99.10% accuracy in flowchart navigation across varied conversational styles and conditions (N=37,200). By combining the flexibility of free-text interaction with the rigor of standardized clinical protocols, this approach demonstrates the feasibility of transparent, accurate, and generalizable AI-assisted self-triage, with potential to support informed patient decision-making while improving healthcare resource utilization.