Clinician-in-the-Loop Smart Home System to Detect Urinary Tract Infection Flare-Ups via Uncertainty-Aware Decision Support

AI summary

Overview

• Research area: Applied machine learning for health monitoring — specifically uncertainty quantification (conformal prediction) applied to ambient smart home sensor data for detecting health flare-ups in older adults.
• Technical level: Intermediate. The core ideas are accessible, but the paper assumes familiarity with binary classification metrics, calibration, and conformal prediction terminology.
• Scope (one sentence): The paper proposes a clinician-in-the-loop smart home system that extracts behavioral markers from ambient sensors and uses a Conformal-Calibrated Interval method to produce uncertainty-aware UTI predictions with the option to abstain, evaluated on 117 labeled days from eight smart homes and a survey of 42 nurses.

What This Paper Is About

Older adults with chronic conditions often develop urinary tract infections (UTIs) that go unnoticed until they become severe, partly because symptoms in this population are atypical and can include delirium, confusion, dizziness, drowsiness, and falls rather than the classic painful urination. Smart home sensors can monitor daily behavior, but standard machine learning systems output only a bare "UTI / no UTI" label with no indication of how confident the model is, which makes it hard for a nurse to decide whether to escalate care. This paper builds a system that predicts UTI flare-ups from sensor-derived behavior markers and attaches statistically grounded uncertainty intervals to each prediction, abstaining ("I don't know") when confidence is low.

Key Contributions

1. A clinician-in-the-loop (CIL) smart home framework that extracts clinically relevant behavioral markers from ambient sensor data to predict UTI flare-ups, closing a sensing–detecting–identifying–acting loop that includes clinicians rather than stopping at detection.
2. The Conformal-Calibrated Interval (CCI) method, a new approach to predictive uncertainty quantification for binary classification that calibrates a pre-trained probabilistic classifier (Logistic Regression or Neural Network) via a nonconformity score and a held-out calibration set, providing statistically valid coverage guarantees and an abstention option.
3. An empirical demonstration on 117 labeled days from eight smart homes showing CCI produces compact intervals, achieves higher recall, and reduces abstention rates compared to a Naive interval baseline and non-interval baselines.
4. Validation through a survey of 42 nurses, indicating that CCI outputs improve interpretability and support clinical decision-making for managing UTI flare-ups.

Main Findings

• Dataset composition: The final dataset contains 117 labeled daily data points from eight participants, each with at least one documented UTI event. It includes 56 UTI days and 61 non-UTI days, with a mean age of 83.8 years (SD = 15.8), five females and three males, and an average of 15.8 years of education (SD = 3.3). Per-participant sample counts range from 9 to 38 days (P1 = 38, P2 = 13, P3 = 15, P4 = 10, P5 = 9, P6 = 13, P7 = 9, P8 = 10).
• CCI outperforms all baselines on classification metrics: Over 20 independent runs, CCI achieved accuracy 0.72 ± 0.16, precision 0.74 ± 0.17, recall 0.78 ± 0.17, and F1 0.75 ± 0.14. The base ML model achieved accuracy 0.69 ± 0.15, precision 0.68 ± 0.15, recall 0.77 ± 0.15, and F1 0.72 ± 0.12. Random Guess achieved accuracy 0.49 ± 0.14, precision 0.48 ± 0.29, recall 0.24 ± 0.14, and F1 0.32 ± 0.19.
• CCI abstains far less than the Naive interval method: CCI abstention proportion was 0.22 ± 0.14 versus 0.73 ± 0.12 for Naive intervals.
• CCI produces much tighter prediction intervals: CCI interval width was 0.20 ± 0.05 versus 0.60 ± 0.06 for Naive intervals, as reported in Table 2.
• The Naive interval method was clinically impractical: It reached only F1 0.57 ± 0.36 with an abstention rate of 0.73, meaning it avoided making predictions most of the time, and its wide intervals indicated low confidence even when it did predict.
• A note on reported figures: The narrative "Visual Analysis of Interval Quality" section describes CCI intervals as having a "mean width of 0.22" and Naive intervals "0.73," which correspond to the abstention proportions in Table 2 rather than the table's interval-width values (0.20 for CCI, 0.60 for Naive). Similarly, the results narrative cites Logistic Regression recall as 0.68, while Table 2 reports recall 0.77 and precision 0.68 for the base ML model.
• Feature selection: From 17 extracted behavioral features, 5 were selected based on SHAP feature importance: Nocturnal Bathroom Visits, Nocturnal Non-Bathroom Movement, Percentage of Nocturnal Bathroom Visits, Health Event (last 3 days), and Daily Movement Entropy.
• Nurse survey results: Nurses gave mixed feedback on the base model prediction plots and negative responses to Naive interval plots, citing lack of clarity and limited clinical value, while the CCI plots were consistently rated as clear, trustworthy, and useful.
• Qualitative nurse feedback: Respondents said the graphs could help decide whether ordering a urinalysis would be indicated, asked to see the factors behind the algorithm's conclusion (addressed via SHAP explanations), requested that a confident indicator move far left or right rather than sitting in the middle, said a smaller confidence range would make them more inclined to use it as a diagnostic tool, and asked for outcomes data before routine adoption.

Methodology in Plain English

Data collection. Ambient sensors were installed in the homes of older adults managing multiple chronic conditions — passive infrared motion detectors, magnetic door detectors, and ambient light and temperature sensors (following the CASAS smart home design). Continuous data were collected for one year in each home. Registered nurses conducted weekly telehealth visits to obtain ground truth, documenting symptoms, interview summaries, and vital signs and annotating specific days as UTI flare-ups. The dataset was balanced by including all UTI-positive days plus adjacent UTI-negative days (days immediately before or after a flare-up without a UTI). The study was approved by the Washington State University Institutional Review Board.

Feature engineering. Raw sensor event streams were converted into 17 daily behavioral features, of which five were used in the main results. Examples include daily bathroom visit counts (a visit defined by sensor ON events followed by at least 5 minutes of inactivity), average bathroom visit duration, nocturnal bathroom visits (9pm–7am), bed-to-bathroom transit times, daily movement entropy (Shannon entropy of sensor ON events across the home), nocturnal awakenings, and rolling 3-day summaries of bathroom behavior.

Models. UTI detection was framed as binary classification over each day. Logistic Regression, a Neural Network, and a Random Guess baseline were compared. Hyperparameters were tuned with GridSearchCV using 3-fold cross-validation and F1 scoring. Data were split with 10% held out for testing, 40% of the remainder used as the conformal calibration set, and the rest for training. The error rate was set to α = 0.1, and all results were averaged over 20 independent runs. Logistic Regression was chosen as the primary model because it consistently outperformed the Neural Network across metrics.

Uncertainty quantification — the two approaches.

• Naive intervals use a Random Forest's tree-level probability spread: the prediction is the average probability across trees, and the interval is the mean ± the standard deviation of tree predictions, clipped to [0, 1]. These have no coverage guarantee and can become wide and unreliable when tree predictions are skewed.
• Conformal-Calibrated Intervals (CCI) transform binary labels into interval centers in the unit interval (0.25 for class 0, 0.75 for class 1), define an uncertainty scaling that grows as the predicted probability approaches the 0.5 decision threshold and shrinks near 0 or 1, and compute nonconformity scores as the squared error between transformed label and predicted probability, divided by that scaling. A quantile of these scores on the calibration set determines the interval width for new predictions. Under exchangeability of calibration and test data, Theorem 1 guarantees the interval contains the transformed label with probability at least 1 − α.

Turning intervals into decisions. Each prediction is mapped to one of three outcomes via an interval-based rule: "UTI" if the interval's lower bound is at least 0.5 or the right-tail probability is at least 1 − α; "No UTI" if the upper bound is below 0.5 or the left-tail probability is at least 1 − α; otherwise the system abstains with "I don't know" and flags the case for nurse review. Classification metrics are reported only on non-abstained predictions to keep comparisons fair, and abstention proportion and interval width are reported separately.

Why This Matters

Impact on research. The paper is distinctive in adapting conformal prediction to binary health-event classification on smart home sensor data and in measuring abstention and interval width as first-class evaluation criteria alongside accuracy, precision, recall, and F1. It argues that point-prediction ML gives clinicians limited utility, and it pairs the technical method with direct end-user evaluation from practicing nurses — closing the loop between statistical guarantee and bedside usefulness.

Real-world applications.

• Home health nursing triage: flagging days when a patient's bathroom and nighttime movement patterns suggest a developing UTI, so a nurse can order a confirmatory urinalysis sooner.
• Remote monitoring of older adults with multiple chronic conditions, where in-person assessment is infrequent and self-reports are inconsistent or unclear.
• Reducing avoidable emergency department visits and hospitalizations by catching infections before they become severe.
• Escalation and treatment planning inside clinical workflows, including integration with electronic health records (EHR).

Industry relevance. The approach targets the cost pressures cited in the paper — chronic conditions account for 90% of what the US spends on healthcare yearly, and in 2023, 78.8% of older adults reported multiple chronic conditions — while the ratio of healthcare professionals to patients keeps falling. A system that produces calibrated confidence intervals and explicit abstentions could slot into remote patient monitoring products, aging-in-place platforms, and payer or home-health agency programs that need defensible, statistically grounded alerts rather than raw ML scores.

Future Directions

1. Deployment pipeline build-out. The paper lays out a path involving integration with existing smart home infrastructure such as CASAS, automated real-time pipelines that compute behavioral markers and prediction intervals daily, and a secure clinician-facing dashboard showing daily risk scores, uncertainty intervals, and abstention flags.
2. Clinical workflow integration. Aligning the system with clinical workflows and the EHR, plus collaboration between the smart home team and healthcare providers to establish intervention thresholds, escalation procedures, and response protocols for flagged cases.
3. Pilot studies and retraining. Multi-month pilot studies to gather clinician feedback and refine model parameters and interval settings, periodic retraining to adapt to changes in residents' routines, and pilot studies with home health and palliative care agencies.
4. Economic validation and generalization. A cost-benefit analysis of reductions in hospitalization and emergency care rates, and — as an open question the paper implies but does not resolve — whether the same uncertainty-aware CIL framework extends to other chronic condition flare-ups beyond UTI and to larger participant populations, since the current evaluation covers eight homes and 117 labeled days.

Target Audience

Researchers and practitioners working at the intersection of machine learning and health monitoring — especially those interested in conformal prediction, uncertainty quantification, and human-in-the-loop decision support. It is also relevant to smart home and ambient assisted living engineers, home health and nursing informatics professionals, and clinical informatics teams evaluating whether sensor-based alerts are trustworthy enough to act on. Readers without a machine learning background can follow the problem framing, dataset description, results table, and nurse feedback sections, but the conformal prediction derivation requires some statistical familiarity.

Authors’ abstract

Urinary tract infection (UTI) flare-ups pose a significant health risk for older adults with chronic conditions. These infections often go unnoticed until they become severe, making early detection through innovative smart home technologies crucial. Traditional machine learning (ML) approaches relying on simple binary classification for UTI detection offer limited utility to nurses and practitioners as they lack insight into prediction uncertainty, hindering informed clinical decision-making. This paper presents a clinician-in-the-loop (CIL) smart home system that leverages ambient sensor data to extract meaningful behavioral markers, train robust predictive ML models, and calibrate them to enable uncertainty-aware decision support. The system incorporates a statistically valid uncertainty quantification method called Conformal-Calibrated Interval (CCI), which quantifies uncertainty and abstains from making predictions ("I don't know") when the ML model's confidence is low. Evaluated on real-world data from eight smart homes, our method outperforms baseline methods in recall and other classification metrics while maintaining the lowest abstention proportion and interval width. A survey of 42 nurses confirms that our system's outputs are valuable for guiding clinical decision-making, underscoring their practical utility in improving informed decisions and effectively managing UTIs and other condition flare-ups in older adults.