TRACER: A Multimodal Harness for Patient Care in Hospitals

AI summary

Overview

Research area: Multimodal machine learning and long-form video understanding, applied to critical care (ICU) patient monitoring; sits at the intersection of computer vision, clinical informatics, and LLM agent design.

Technical level: Advanced. The paper assumes familiarity with vision-language models, agentic tool-use loops, long-context limits, and benchmark construction, though the harness idea itself is explainable without that background.

Scope: TRACER is a training-free system that turns continuous ICU bedside video plus electronic health records into a queryable minute-by-minute and hour-by-hour behavioral record, which an LLM agent then uses to answer retrospective clinical questions and to compare behavior across a 280-patient cohort.

What This Paper Is About

Understanding how an ICU patient's behavior changes over a multi-day stay requires connecting brief, moment-level observations (a hand gesture, a clinician leaning in) to patterns that unfold over days — and doing so while drawing on clinical records that give those observations meaning. Existing approaches either cannot cover a full stay without losing detail, or require training models on protected patient video, which makes them impossible to share with other hospitals. TRACER's goal is to build a hierarchical, shareable representation of a stay that an LLM agent can navigate with tools, so that both fine-grained visual evidence and stay-long trends stay accessible without any model training on patient data.

Key Contributions

1. A training-free multimodal harness. TRACER (Trajectory Reasoning Across Clinical Event Records) separates temporal abstraction — building minute-level action records and hourly behavioral reports continuously as the stay is recorded — from evidence acquisition, where an agent reasons over those compact records and pulls raw video or EHR on demand. All problem-specific structure (taxonomy, timeline, tools, prompts) lives in components that can be publicly released.

2. A minute-level clinical action taxonomy and perception pipeline. Fifteen atomic actions in five groups (patient–clinician interaction: contact, talking, object exchange, restraint; plus head, torso, arm, and leg movements), detected zero-shot by a VLM from 60 frames sampled at 1 fps per minute, with the two camera views merged by logical OR.

3. Two new benchmarks. A perception benchmark of 397 manually annotated one-minute clips from 26 patient stays covering all 15 actions (double-annotated, κ = 0.81, 92.7% agreement on a 20-clip audit), and TRAJECTORY-144, a clinician-informed set of 144 five-way multiple-choice questions grounded in 264 hours of video and EHR from two ICU stays, tagged by required evidence modality (video, EHR, cross-modal) and by temporal span.

4. A cohort-level behavioral analysis. TRACER's perception layer was run across 280 patients spanning approximately 60,000 camera-hours, linking detected action frequencies to delirium status.

Main Findings

• Action detection is useful but noisy. Gemini 3.5 Flash reached 84.8% accuracy and 68.6 micro-F1 across the 15 atomic actions. Gemini 3.1 Flash Lite reached 82.3% accuracy and 62.2 micro-F1 at less than one-tenth the cost, which is why it was used as the perception VLM for the 280-patient cohort run. For context, the paper's majority-class baseline scores 77.7% accuracy but only 45.2 F1, so the F1 gap is the more informative comparison.

• TRACER beats long-context and caption-based baselines on TRAJECTORY-144. With Gemini 3.1 Flash Lite as backbone, TRACER reaches 61.0% average accuracy versus 49.3% for the strongest baseline (the same model given 475 raw frames plus EHR). The gain is +11.7 points (95% CI [4.4, 18.9], paired bootstrap over questions). Baselines use roughly three times more tokens than TRACER during question answering.

• EHR questions are where the hierarchy pays off most. TRACER with Gemini 3.1 Flash Lite scores 96.2% on EHR-related questions, while baselines with EHR access top out at 83.8%.

• Agent model choice matters less than perception. With perception fixed to Gemini 3.1 Flash Lite, Gemini 3.5 Flash as agent reaches 70.7%, Gemma-4 31B reaches 67.9%, Qwen3.5 9B reaches 61.6%, gpt-oss-20b reaches 57.4%, and Gemini 3.1 Flash Lite as its own agent reaches 61.0%. The authors conclude that the open-weight versus proprietary gap lies primarily in perception, not agentic reasoning — an important result for on-premise deployment.

• Each tool family serves a different question type. In the ablation (averaged over Gemini 3.5 Flash, Gemini 3.1 Flash Lite, and Qwen3.5 9B), EHR tools drive EHR-question accuracy (96.4% alone), video tools help most on questions under 5 minutes, reports help on 5–30 minute questions, and timeline tools help over long horizons. Adding toolsets cumulatively gives +3.9, +17.4, and +21.6 average points; the full harness exceeds the best single toolset by 16 points.

• Delirium patients show less purposeful activity. Across 280 patients (n = 53 with at least one positive CAM-ICU assessment, n = 227 without), restricted to the 13 actions with sufficient positive samples, seven actions are significantly less frequent in the delirium group. The largest reductions are in goal-directed actions: using non-medical objects (0.41×), leg movements associated with daily living (0.46×), object exchange with a clinician (0.48×), and gesticulation (0.60×). The fraction of observed minutes containing any of these four is lower in the delirium group (median 0.053 vs. 0.146 per patient, Mann–Whitney p = 1.3 × 10⁻⁸). Clinician–patient contact is the only action more frequent in the delirium group (1.20×). The paper reports p < 0.0001 for the overall cohort comparison and notes all but head turning survive Holm correction on per-patient rates.

Methodology in Plain English

The design rests on one observation: you cannot feed days of video to a model, and you cannot train on patient video you are not allowed to share. So TRACER pre-digests the stay into something small enough to reason over, and keeps the raw data behind tools.

Each minute of video from each of two room cameras is sampled into 60 frames and passed to an off-the-shelf vision-language model with a fixed prompt. The model labels which of 15 clinically defined actions occurred and writes a short caption. Two camera views are combined into one label vector per patient-minute.

When an hour completes, a language model takes those 60 minute-records plus the medication events charted in the EHR during that hour and writes a structured six-section report: movement summary with minute citations, patient–clinician interaction summary, significant moments, medication context, activity–medication dynamics, and a short narrative. Hour-level was chosen because it matches how often ICU flowsheet entries are typically charted.

At question time, an agent model gets a fixed system prompt and eleven tools in four families: report (read hourly reports), timeline (query per-minute action labels, count minutes per action, find first occurrences), video (author a custom prompt for a VLM to re-inspect a chosen segment at 5 fps for Gemini or 2 fps for open-weight models), and EHR (pull predefined record categories). The agent loops — choose tool, read result, repeat — until it answers or hits a 25-call budget, at which point a forced-choice prompt makes it answer from what it has. Nothing in the pipeline is trained.

Evaluation runs at three levels: perception on the 397 clips, longitudinal reasoning on TRAJECTORY-144 (averaged over 5 runs, each with the gold answer in a different position), and cohort analysis over all 280 patients' minute-level records.

Why This Matters

Impact on research. The paper makes a structural argument that matters beyond the ICU: in high-stakes domains, the shareable artifact should be the harness — taxonomy, temporal hierarchy, tools, prompts — rather than model weights fitted to protected data. That reframes reproducibility for clinical vision work, where datasets routinely cannot leave the institution. It also extends long-video understanding from the hour-scale benchmarks toward week-scale stays, and does so by fusing a second, asynchronous modality (the EHR) into the temporal hierarchy, which prior long-video systems do not do.

Real-world applications:

• Retrospective chart review and quality improvement: asking "how did this patient's mobility change after the sedative was started?" and clicking through to the cited minute of video.
• Delirium surveillance support: flagging patients whose purposeful activity profile resembles the hypoactive pattern, which the paper notes is the most prevalent ICU subtype and the one most often missed by intermittent screening.
• On-premise deployment in hospitals that cannot send video to a third-party API — supported by the finding that open-weight agents match proprietary ones once perception is fixed.
• Portable instantiation of the same hierarchy over other temporally indexed video-plus-record collections, since no part of the harness is fitted to this data.

Industry relevance. The cost finding (Gemini 3.1 Flash Lite at 82.3% versus Gemini 3.5 Flash at 84.8% for under one-tenth the cost) and the explicit 25-tool-call budget make the trade-offs legible for product teams. The interface described — a bedside video viewer with an action timeline underneath and a chat panel that links cited timestamps to the corresponding minute — is a concrete design pattern for clinical review tools, and the authors explicitly position TRACER for retrospective querying rather than continuous alerting, citing existing alarm fatigue in ICUs.

Future Directions

• Broaden the reasoning benchmark. TRAJECTORY-144 is explicitly "deep rather than broad": 144 questions over two stays at a single center. Extending to more stays, more patients, and multiple hospitals is the obvious next test of whether the harness generalizes.
• Improve the perception layer. The authors locate the open-weight gap in perception, so better zero-shot or openly available action detectors would lift the whole system — and would also reduce the per-action detection error that currently propagates into the cohort-level delirium frequency ratios.
• Validate the delirium signal prospectively. The paper is careful that its cohort associations are hypothesis-generating from retrospective data and do not by themselves establish predictive capability; a prospective study would be needed to see whether the activity profile anticipates delirium rather than merely correlating with a documented positive screen.
• Address the governance questions. Consent, staff surveillance, and fairness are named as prerequisites for deployment but not examined in this work, and hourly reports are model-generated and may contain inaccuracies or hallucinations.

Target Audience

Researchers working on long-form video understanding, multimodal agents, and clinical machine learning will get the most from this paper, particularly those wrestling with data that cannot be shared. It is also relevant to hospital informatics and critical-care teams evaluating ambient sensing, and to engineers building tool-augmented agent systems who want a concrete case study in how temporal abstraction and tool design interact. Readers without a background in VLMs or agent architectures will need to work through the harness terminology, but the core argument — build a hierarchy, expose tools at every level, train nothing — is accessible without it.

Authors’ abstract

Continuous video and clinical records provide complementary information for understanding how patient behavior evolves over a hospital stay. A central challenge is how to represent behavior across the entire stay while preserving access to fine-grained visual and clinical evidence. We introduce TRACER, a training-free multimodal harness that constructs minute-level action records from video and combines them with clinical records into hourly reports. The harness equips an LLM agent with tools to query this hierarchy, inspect relevant video, and retrieve clinical records. To evaluate TRACER, we construct two benchmarks. The first contains 397 annotated one-minute clips spanning 15 atomic actions; the second poses 144 clinician-informed multiple-choice questions, grounded in 264 hours of video and clinical records from two ICU patients. TRACER achieves up to 84.8% accuracy for action detection and 61.0% QA accuracy versus 49.3% for the strongest baseline using the same backbone. Across 280 patients spanning approximately 60,000 camera-hours, TRACER identifies significantly less purposeful activity in patients with a positive delirium assessment during their stay than in those without (p &lt; 0.0001). Our results show that combining continuous video with clinical records can support understanding of patient behavior over time and across patient cohorts without training on protected health data.