Human-in-the-Loop Interactive Report Generation for Chronic Disease Adherence

AI summary

Overview

Research area: Human-Computer Interaction (cs.HC) applied to clinical AI, specifically clinician-in-the-loop interfaces for chronic disease adherence reporting.

Technical level: Intermediate. The paper is readable without deep machine learning background, but it assumes familiarity with human-in-the-loop system design, evaluation study methodology (Likert scales, t-tests), and basic clinical workflow concepts.

Scope in one sentence: The paper designs and pilot-tests a single-page clinical report editor in which an LLM drafts structured adherence reports from routine patient data while physicians retain full approval authority through recognition-based review.

What This Paper Is About

Clinicians treating chronic diseases need to send patients regular, personalized adherence feedback to prevent avoidable hospitalizations, but documentation already consumes substantial clinical time and manual authoring does not scale. Fully automated AI generation scales well but risks errors that undermine trust in patient-facing text, and existing human-in-the-loop tools often shift the burden from drafting to multi-screen review, making approval rather than generation the bottleneck. This paper asks whether deliberately constraining AI to data organization, combined with an interface built for fast local verification, can deliver both efficiency and accountability.

Key Contributions

1. Bounded AI preparation with recognition-based review. The AI is constrained to data organization—drafting structured text into a fixed template (what happened, why it matters, what to do next) and explicitly marking data gaps—while never approving or sending content. A single-page editor pairs each text section with time-aligned charts so physicians verify locally through recognition rather than recall.

2. Attention management via visual urgency flags. The system automatically labels cases as urgent, attention, or stable by analyzing vital trends, adherence gaps inferred from dialogue, and a rule-based verifier, with urgent cases highlighted in red at the queue level to reduce cognitive triage across multiple patients.

3. Conservative safety through fail-safe rules. If disease-specific critical monitoring tasks are missed (for example, daily blood pressure checks for hypertensive patients or glucose monitoring for diabetics), the case automatically escalates to urgent regardless of other indicators, preventing algorithmic optimism from hiding time-sensitive risks.

4. Empirical identification of the "accountability paradox." A pilot with 3 physicians reviewing 24 cases showed AI drafts reached near-baseline clinical quality, yet review time did not improve, because non-delegable professional responsibility compels complete verification regardless of AI accuracy.

Main Findings

• AI drafts reached clinician-level quality. Across all 24 cases, the overall mean score across the 12 quality dimensions was 4.86/10 (SD = 0.52), against a baseline of 5.0 representing physicians' established manual authoring practice. Adherence accuracy was the strongest dimension (Q8: 5.25) and intervention recommendations the weakest (Q2: 4.42).

• Editing effort was minimal. Physicians modified an estimated 8.3% of content on average (95% CI: 3.3–13.3%). One case raised a minor concern that was corrected during review (1/24), and no safety-critical issues were identified (0/24).

• Quality did not translate into time savings. The pooled mean for perceived time effort saved (Q10) was 4.79 (SD = 0.83), not significantly different from the baseline of 5.0 (t(23) = -1.23, p = 0.233). Physician 1 reported exact parity (mean = 5.00), Physician 2 rated below baseline (mean = 4.00, t(7) = -3.74, p = 0.007), and Physician 3 trended above baseline without significance (mean = 5.38, t(7) = 1.43, p = 0.197).

• The accountability paradox. Even when AI drafts are accurate, physicians must still perform complete verification because clinical responsibility cannot be delegated. The paper argues interaction design improvements alone cannot overcome accountability-driven verification requirements.

• Interface features supported verification, not time reduction. Chart–text pairing supported section-level verification, reflected in strong ratings for information-location efficiency (Q11: 5.08, SD = 0.65). Urgency flags assisted triage and the single-page layout reduced context switching, but overall review duration remained unchanged.

• Reviewer-level variation. Overall means differed by physician (Physician 1: 5.31; Physician 2: 4.28; Physician 3: 5.00), which the authors attribute to individual tolerance for AI-generated text rather than systematic model errors.

• High internal consistency. Cronbach's alpha = 0.89 across the 12 evaluation dimensions.

• AI's benefit may lie in attention management. The authors conclude from the urgency-flagging results that AI's value may be in triaging and prioritizing attention rather than reducing per-task time.

Methodology in Plain English

The researchers built a system around a deliberate division of labor: the model prepares, the physician decides, and the patient receives only a physician-approved note. Patient data—medication lists and refills, basic device trends, and a brief dialogue—flow into a unified memory and are processed in parallel into sectioned drafts and urgency flags. Drafting used Qwen3-8B with fixed decoding parameters (temperature = 0.7, max_tokens = 1200) for reproducibility.

The physician works in a single HTML page rather than a multi-screen workflow. Sections act as anchors for quick jumps, sentences are edited in place, and controls sit next to the items they affect—a checkbox to confirm medications, a menu to set the follow-up interval, and a button to approve and export. Charts sit beside the sentences they explain. A typical session follows one short pass: load the draft, read each section summary, check the essentials, make focused inline edits if needed, and approve the final note.

Urgency assessment combines an LLM-generated initial estimate with a simple rule-based verifier that can escalate or adjust the final label; when the LLM is unavailable, rule-based heuristics provide fallback classification. A separate fail-safe rule escalates missed disease-specific critical monitoring tasks to urgent.

For evaluation, 3 physicians reviewed 24 generated chronic disease adherence reports skewed toward higher-risk scenarios (14 urgent, 8 attention, 2 stable). After each case they completed a structured questionnaire rating 12 quality dimensions on a 1–10 Likert scale, organized into core medical judgment (Q1–5), data and factual accuracy (Q6–8), and workflow integration (Q9–12). The baseline of 5 was set at the midpoint to acknowledge that manual authoring already achieves good clinical quality, with the note that manual reports typically require 15+ minutes per report. Physicians also recorded editing scope (unmodified, less than 10%, 10–30%, more than 30%) and flagged safety concerns.

Why This Matters

Impact on research. The paper reframes the clinical AI problem away from generation quality and toward interface design and verification mechanisms. The accountability paradox challenges the assumption that better models or better interfaces will reduce physician workload in high-stakes settings, and it argues that future work must target accountability-preserving selective verification rather than accuracy alone.

Real-world applications:

• Chronic disease management programs in primary care and community health centers that need to send personalized adherence reports to patients at scale without losing clinical oversight.
• Clinical documentation and summarization tools that must integrate into existing charting and editor code with minimal friction—fewer screens, clicks, and handoffs.
• Patient triage and panel management, where automated urgency labeling could surface high-priority patients from vital trends and adherence patterns.
• Safety-critical monitoring workflows, where fail-safe escalation rules can guarantee that missed disease-specific tasks (for example, blood pressure or glucose monitoring) are never buried beneath algorithmic optimism.

Industry relevance. The findings speak directly to electronic health record vendors and clinical AI product teams. The paper points out that medical malpractice insurance often excludes AI-related incidents while covering human errors, and that physicians retain full liability when using AI tools. That liability structure reinforces mandatory complete verification regardless of AI accuracy, meaning the commercial case for clinical AI may rest on attention management and prioritization rather than headline time savings.

Future Directions

• Confidence-based selective verification. Designing mechanisms that let physicians verify only the portions of a draft that warrant scrutiny, rather than the whole document from scratch.
• Graduated trust based on measured accuracy. Formalizing how trust in AI output could increase as demonstrated accuracy accumulates, while preserving the clinician as author of record.
• Accountability-preserving shortcuts. Implementing section-level approval rather than whole-document approval, so oversight is retained without requiring exhaustive line-by-line review.
• Evolving legal and insurance frameworks. The authors argue that technical mechanisms alone are insufficient and that systemic solutions recognizing AI-assisted practice in malpractice coverage and liability rules are needed.

Target Audience

This paper benefits HCI researchers working on clinical decision support and human-AI collaboration, clinical informatics teams designing documentation and patient-communication tools, health system leaders evaluating whether AI can reduce documentation burden, and policy or governance researchers interested in how liability structures shape the adoption of AI in high-stakes domains. Readers looking for a technical machine learning contribution will not find one here; the value is in interaction design patterns, an evaluation of those patterns in a small pilot, and a conceptual finding about accountability.

Authors’ abstract

Chronic disease management requires regular adherence feedback to prevent avoidable hospitalizations, yet clinicians lack time to produce personalized patient communications. Manual authoring preserves clinical accuracy but does not scale; AI generation scales but can undermine trust in patient-facing contexts. We present a clinician-in-the-loop interface that constrains AI to data organization and preserves physician oversight through recognition-based review. A single-page editor pairs AI-generated section drafts with time-aligned visualizations, enabling inline editing with visual evidence for each claim. This division of labor (AI organizes, clinician decides) targets both efficiency and accountability. In a pilot with three physicians reviewing 24 cases, AI successfully generated clinically personalized drafts matching physicians' manual authoring practice (overall mean 4.86/10 vs. 5.0/10 baseline), requiring minimal physician editing (mean 8.3\% content modification) with zero safety-critical issues, demonstrating effective automation of content generation. However, review time remained comparable to manual practice, revealing an accountability paradox: in high-stakes clinical contexts, professional responsibility requires complete verification regardless of AI accuracy. We contribute three interaction patterns for clinical AI collaboration: bounded generation with recognition-based review via chart-text pairing, automated urgency flagging that analyzes vital trends and adherence patterns with fail-safe escalation for missed critical monitoring tasks, and progressive disclosure controls that reduce cognitive load while maintaining oversight. These patterns indicate that clinical AI efficiency requires not only accurate models, but also mechanisms for selective verification that preserve accountability.