Sensor-Language-Action Models

AI summary

Overview

Research area: Multimodal machine learning for sensor data — specifically the intersection of physiological/wearable time-series modeling, clinical decision prediction, and language models. Technical level: Advanced. Scope: The paper proposes Sensor-Language-Action (SLA) modeling, a framework that treats natural language as the shared interface between heterogeneous sensor observations and the actions they inform, and instantiates it in a model called OpenSLA with a benchmark of more than 116,000 individuals, 79 sensor modalities, and 60 action groups across clinical, operating-room, and continuous glucose monitoring settings.

What This Paper Is About

Existing sensor models largely stop at perception: they recognize states or predict outcomes, but the actions that follow are modeled separately, each with its own task-specific and often closed label space. The authors ask whether sensor data, language, and action can be modeled through a single unified framework, using language as a bridge that can express task intent, individual context, physiological state, and action semantics in a common form. Their goal is a model that not only predicts what action should be taken from sensor evidence but also explains the evidence supporting that action.

Key Contributions

1. Formulation of Sensor-Language-Action (SLA) modeling, a framework that unifies sensor-action problems through a multimodal language interface, together with a large-scale benchmark spanning more than 116,000 individuals, 79 sensor modalities, 5 tasks, and 60 action groups.
2. A multi-faceted sensor captioning pipeline that jointly captures local sensor dynamics, cross-signal relations, individual-level context, and action-relevant evidence, providing language supervision tailored specifically to sensor-action modeling.
3. OpenSLA, a unified SLA model that jointly supports hierarchical action prediction, state understanding, and action explanation, informed by systematic studies of how sensor signals, language, and actions should be modeled together.
4. Extensive evaluation against state-of-the-art methods across six development datasets and one external cohort on five tasks, plus analyses of language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.

Main Findings

• Joint action-language fine-tuning outperforms every other paradigm tested. The authors compare four language-model-centric paradigms — zero-shot multimodal LLMs, VLA-style action-only fine-tuning, SensorLM-style sensor-language alignment, and joint fine-tuning with action and language supervision — on VitalDB action prediction. Joint fine-tuning performs best, ahead of action-only training, sensor-language alignment, and zero-shot multimodal LLMs including GPT-5.6-Luna and DeepSeek-V3.2.
• Direct LoRA adaptation beats other sensor-into-LLM integration strategies. Among LLaVA-style adapter training, Flamingo-style cross-attention injection, and direct LoRA, LoRA (the strategy OpenSLA uses) achieves the strongest performance on most action-prediction tasks.
• Strong hierarchical action prediction across all three domains. On the clinical ICU datasets, OpenSLA-B reaches 77.6 AUROC for action necessity, 72.5 for category, and 61.2 balanced accuracy for fine-grained labels on MC-MED, and 84.0 / 75.4 / 73.7 respectively on MIMIC-III. On the operating-room data, OpenSLA-H reaches 85.9 necessity AUROC and 80.6 category AUROC on MOVER, and 77.1 necessity AUROC and 86.3 category AUROC on VitalDB. On CGM data, OpenSLA-B reaches 73.3 necessity AUROC on MetaboNet and 81.2 on PEDAP.
• The largest gains appear in the clinical and OR domains. The authors attribute this to individual-level textual context being especially informative for context-dependent clinical decisions, whereas sensor-only action prediction is most challenging in the OR setting.
• CGM action-category prediction is the notable exception. The authors report that insulin decisions can depend on future carbohydrate intake that is unavailable in the observed context, making category prediction harder there.
• Generated captions recover physiological state accurately. Using mean absolute error against ground-truth sensor values, OpenSLA-H reaches 1.01 for Cardiac Index, 15.9 for ECG-II Rate, 1.17 for RESP Rate, 1.41 for operating-room respiratory rate, 0.47 for temperature, 3.18 for ETCO2, 0.35 for basal rate, and 0.38 for carb amount — the most accurate estimates among the compared methods.
• Action explanations align well with reference evidence. Using BERTScore F1, OpenSLA-B scores 81.5 (MC-MED), 78.6 (MIMIC-III), 81.5 (MOVER), 80.9 (VitalDB), 85.6 (MetaboNet), and 87.6 (PEDAP); OpenSLA-H scores 82.0, 80.9, 81.0, 80.7, 80.4, and 76.7 respectively.
• Unseen actions sit near semantically related ones. In a PCA-reduced representation space learned on MIMIC-III, semantically related actions occupy nearby regions; the held-out antibiotic Imipenem (excluded from training) is positioned near other antibiotics, and OpenSLA achieves the highest balanced accuracy on zero-shot prediction of that unseen action.
• Continuous action structure emerges. Interpolating between representations of two distinct insulin actions and retrieving nearest real samples yields retrieved insulin actions that vary monotonically along the interpolation trajectory.
• SLA representations retain future-state information. With the backbone frozen, ridge readouts on 2-hour CGM windows predict the mean, minimum, maximum, and slope of glucose over the subsequent 2 hours under individual-grouped nested cross-validation, even though the backbone was not trained with future-state supervision.
• Transfer to an unseen cohort holds up. Applied directly to the external MIMIC-IV cohort without adaptation, OpenSLA-B reaches 81.75 necessity AUROC and 74.40 balanced accuracy, and 74.64 category AUROC and 56.35 balanced accuracy, versus 69.30 / 54.33 for OpenTSLM and 67.70 / 50.02 for SensorLM.
• Language supervision is the single biggest driver. Ablations show the largest degradation under VLA-style action-only training; removing action conditioning or the multimodal fusion decoder further reduces performance. In the caption-facet ablation, using all four facets gives the best results on both MOVER (83.4 necessity AUROC, 80.3 category AUROC) and VitalDB (74.8, 85.9), and action captions provide the strongest single addition.

Methodology in Plain English

The authors first assemble a testbed from six development datasets plus one external dataset: MC-MED, MIMIC-III, and MIMIC-IV for clinical ICU prediction; MOVER and VitalDB for operating-room decision making; and MetaboNet and PEDAP for daily-life continuous glucose monitoring. MIMIC-IV is held out entirely for cross-cohort generalization testing.

For each dataset they define a shared three-level action hierarchy: whether to act at all (necessity), what type of action (category), and which specific action (fine-grained label). For example, administering Imipenem becomes "Yes" at the necessity level, "antibiotics" at the category level, and "Imipenem" at the fine-grained level.

They then generate four complementary types of captions to serve as language supervision: observational captions describing channel-level physiological findings; relational captions describing relationships across physiologically related signals; global captions integrating individual background and history; and action captions verbalizing which preceding evidence supports a recorded action. They test different combinations of these facets.

The OpenSLA model has three parts. A hierarchical sensor encoder processes each channel with a frozen signal encoder, then compresses the tokens across multiple resolutions using learned queries while a parallel set of global queries captures sample-level semantics. An LLM backbone (Qwen-3.5-2B, adapted with LoRA) carries structured prediction heads for the three action levels, where fine-grained prediction works by matching the LLM representation to embeddings of textual action descriptions. A gated residual caption decoder then cross-attends to the predicted action profile and to retrieved sensor evidence, refining the hidden states used for autoregressive caption generation. Training combines an action prediction loss with an autoregressive cross-entropy text loss.

Evaluation covers three capabilities: action prediction (macro-AUROC and balanced accuracy for necessity and category, balanced accuracy for fine-grained labels), state understanding (mean absolute error between sensor values extracted from generated captions and ground truth), and action explanation (BERTScore computed with DistilBERT). Baselines span supervised models (ConvTrans, MMIM), time-series foundation models (Chronos-2, Moirai, Gluformer), sensor-language models (OpenTSLM, SensorLM), and prompted multimodal LLMs (GPT-5.6-Luna, DeepSeek-V3.2). All methods trained on the authors' cohorts use the same training data and compute budget, with training capped at 24 hours on H200 GPU nodes.

Why This Matters

Impact on research. The paper reframes sensor modeling from a perception problem into a perception-plus-action problem, and argues that language is the mechanism that makes heterogeneous sensor-action tasks expressible in one framework. It also releases a benchmark whose scale (more than 116,000 individuals, 79 modalities, 60 action groups, 7 datasets, and a training pool of more than 2.3 million windows across the six development datasets) substantially exceeds the representative sensor-language studies compared in the paper's Table 1 — several of which report no action supervision at all.

Real-world applications.

• Clinical ICU decision support: predicting medication, procedure, and treatment-adjustment decisions from waveforms and measurements, with explanations grounded in the sensor evidence.
• Operating-room decision making: anticipating interventions from intraoperative signals such as ECG, plethysmography, respiration, blood pressure, SpO2, and heart rate.
• Metabolic health and insulin management: anticipating insulin bolus delivery from continuous glucose monitoring and basal insulin delivery data.
• Cross-site deployment: applying a trained model to a new hospital cohort without adaptation, as demonstrated on MIMIC-IV.

Industry relevance. The pipeline is released as an open dataset, code, and project page (OpenSLA on Hugging Face, GitHub, and a project site under CC BY-NC-ND 4.0), which lowers the barrier for groups building sensor foundation models. The finding that direct LoRA adaptation outperforms adapter and cross-attention alternatives is directly actionable for teams integrating sensor encoders into pretrained language backbones. The paper is explicit that OpenSLA has not undergone clinical validation and is not intended for direct clinical decision-making, so deployment would require prospective validation, safety evaluation, and regulatory review.

Future Directions

• Clinical validation. The authors state that OpenSLA has not undergone clinical validation and that real-world deployment would require prospective validation, safety evaluation, and appropriate regulatory review.
• A generalist cross-domain model. The current implementation trains separate models per domain, partly because action spaces are defined independently; the authors identify a generalist SLA model operating across domains through shared representations of heterogeneous actions as an important next step.
• Robustness to sensor heterogeneity. Experiments assume a fixed set of sensor modalities and input channels, while real-world sensing systems vary across devices, signal availability, and acquisition settings.
• Open question from the results. Why CGM action-category prediction remains difficult — the paper points to dependence on future carbohydrate intake not present in observed context — and whether that information gap can be addressed.

Target Audience

Researchers and practitioners working on sensor foundation models, time-series and physiological signal modeling, and multimodal language models; clinical machine learning groups interested in decision prediction from ICU, operating-room, and CGM data; and industry teams building wearable or medical sensing products that need a shared representation for observation, context, and action. The paper assumes familiarity with LLM fine-tuning methods (LoRA, cross-attention, adapters), transformer architectures, and standard evaluation metrics such as AUROC, balanced accuracy, MAE, and BERTScore, so it is best suited to readers with an intermediate-to-advanced machine learning background.

Authors’ abstract

Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.