TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

AI summary

Overview

Research area: Machine learning for health, specifically time-series language models (TSLMs) and sensor–language alignment on wearable data. Technical level: Intermediate. Scope: The paper introduces TimeCap, an open deterministic pipeline that converts wearable sensor signals into natural-language captions, releases the resulting MHCap corpus of 7,005,763 captions, and tests whether caption-augmented training improves health-outcome prediction.

What This Paper Is About

Time-series language models could help interpret the massive streams of data produced by wearables, but they need paired time-series and text, and the few large paired corpora that exist are closed, limited in sensor coverage, or generated by language models in ways that cannot be reproduced exactly. The authors build a captioning pipeline that writes captions directly from the signal using fixed rules rather than a language model, apply it to a large wearable cohort to create an open caption layer, and measure whether those captions actually help a model predict health outcomes.

Key Contributions

1. An open, deterministic captioning pipeline. TimeCap is released under the MIT license and produces captions from the signal with no language model in the generation loop, so anyone with access to the signals can regenerate the corpus exactly. It has four modular extractor tiers (statistical, structural, semantic, and cross-channel) and extends to new datasets through channel definitions rather than pipeline redesign.
2. MHCap, an open caption layer for wearable data. Applying TimeCap to OpenMHC (11,894 participants, more than 60 million hours, 19 Apple HealthKit channels) yields 7,005,763 captions over 908,944 retained person-days from 2,081 sharable participants, covering all 19 HealthKit channels.
3. A second, cross-domain instantiation. Applying the same pipeline to the WESAD lab-physiology dataset produces WESADCap: 1,461 captions over 15 sessions from 15 participants, covering 20 chest- and wrist-worn physiological channels, with only dataset-specific channel definitions changed.
4. An evaluation with a time-series language model. OpenTSLM trained on MHCap captions plus labels (OpenTSLM-Cap) is compared against an architecturally identical label-only OpenTSLM baseline at the 1B and 3B scales.

Main Findings

• Captions help on most binary evaluations. OpenTSLM-Cap outperforms the label-only OpenTSLM on 29 of 32 per-user binary evaluations across 8 tasks, at both the 1B and 3B scales, spanning both tasks seen during fine-tuning and held-out clinical conditions.
• Held-out clinical gains appear at the per-week level at 3B scale. Macro-F1 improvements over OpenTSLM are +3.4 for Cardiovascular Disease, +6.1 for Diabetes, +2.6 for Coronary Artery Disease, and +2.6 for Atrial Fibrillation. These four conditions are excluded from fine-tuning and evaluated zero-shot.
• Ordinal in-distribution tasks also improve at 3B. OpenTSLM-Cap improves QWK by +4.8 on BMI (5 levels) and +5.0 on Happiness (4 levels) over OpenTSLM.
• Every single caption family beats the label-only baseline on average. In a 1B ablation, any one caption family raises mean per-user AUROC over OpenTSLM by 1.9 to 3.2 points, more than the 1.3 points separating the families (baseline mean over all binary tasks 64.12; structural 67.28; semantic 66.01; cross-channel 67.25; full mix 67.19). Only the full mix beats OpenTSLM on all eight binary tasks. On the held-out tasks, the means are 60.62 for the baseline, 65.27 for structural, 63.02 for semantic, 65.38 for cross-channel, and 65.79 for the full mix.
• Labels alone remain better on the ordinal targets. The authors report that labels-only stays ahead on the ordinal targets in the ablation, and per-week differences are small and mixed.
• The corpus is dominated by structural and cross-channel captions. For OpenMHC-weekly: statistical 1,002,819 (14.3%), structural 3,191,600 (45.6%), semantic 867,948 (12.4%), cross-channel 1,943,396 (27.7%), with a mean of 48.5 captions per user-week. For WESAD: statistical 240 (16.4%), structural 1,071 (73.3%), semantic 75 (5.1%), cross-channel 75 (5.1%), with a mean of 97.4 captions per session.
• No earlier corpus combines openness, determinism, and extensibility. Table 1 compares SensorLM (59.7M hours, 103,731 people, 3-tier, not open, not extensible), Sensor-QA (5,648 QA, 60 people, human-written, open), TSR-Suite (23,605 samples, 2,339 human-curated), ChatTS (134K synthetic samples), and SleepLM (more than 100K hours, more than 10K people, closed).
• Some transfer fails. In additional zero-shot experiments not reported in the main results section, ordinal transfer to held-out blood-pressure and sleep-duration categories failed across all configurations, with parse rates collapsing and predictions defaulting to the majority class.
• Clinical per-user performance remains population-level. Per-user AUROC between 59.4 and 70.5 on the clinical tasks (Hypertension, Sleep Diagnosis, and the four held-out conditions) indicates population-level patterns rather than reliable individual predictions.

Methodology in Plain English

The pipeline first maps each dataset's raw sensor streams into one shared representation: a channel-by-time matrix plus metadata such as channel names, display names, units, and data-availability summaries. Missing values are kept explicit in the matrix rather than filled in, so every downstream step sees the same interface regardless of the original schema.

Four extractor tiers then write captions from that matrix:

• Statistical: one summary caption per continuous channel and recording, reporting mean, maximum, minimum, and standard deviation through fixed templates.
• Structural: finds events inside a channel, including spikes (using scipy.signal.find_peaks with channel-specific constraints) and sustained upward or downward trends detected by scanning the signal with multiple window sizes, fitting a linear model per window, and keeping segments that pass thresholds on span, coverage, effect size, and goodness of fit.
• Semantic: reads dedicated binary channels such as sleep or workout labels, keeps contiguous active intervals above a minimum duration, and writes time-bounded statements such as "running" or "sleeping".
• Cross-channel: combines signals so an activity or sleep interval is described together with concurrent evidence, for example pairing a walking interval with heart rate, step count, distance, and active energy, or distinguishing time in bed from time asleep.

All hyperparameters are fixed before generation and there is no language model in the loop, so the same inputs always give the same captions. Evaluation uses a week of hourly data as a 19 × 168 tensor over 19 HealthKit channels, posed to the model as a natural-language question with a fixed answer set. Ten participant-level outcomes are used (8 binary, 2 ordinal): six in-distribution targets (Biological Sex, Hypertension, Employment, Sleep Diagnosis, BMI, Happiness) and four held-out clinical conditions (Diabetes, Cardiovascular Disease, Coronary Artery Disease, Atrial Fibrillation). Both models share the OpenTSLM-SP soft-prompting architecture on Llama-3.2 1B and 3B and are fine-tuned with LoRA (r = 16, α = 32) for one epoch, differing only in training mix. Statistical captions are deliberately omitted from training because the prompt already includes per-channel summary statistics, so any gap reflects caption supervision rather than architecture. Splits are participant-level at 60/10/30 with seed 42; 2,081 of the 11,894 sharable participants remain after data-quality filtering, with 604 in the test partition. Training ran on a single NVIDIA H100 80 GB for approximately 35 hours per configuration, roughly 140 GPU-hours across the four reported configurations.

Why This Matters

Paired sensor–text data is the bottleneck for language models that reason over wearable signals, and the largest such corpora have been closed or generated in ways that cannot be reproduced or audited. TimeCap turns corpus construction into a repeatable procedure: captions can be regenerated bit-for-bit, traced back to a rule and threshold, and extended to new devices by writing channel definitions. That matters both for scientific reproducibility and for health applications, where training data should be inspectable.

Real-world applications:

• Wearable health monitoring: describing activity, sleep, and physiology from consumer devices in grounded language that a model can learn from.
• Clinical cohort research: predicting self-reported conditions such as hypertension and diabetes from a week of sensor data, while noting the paper's own caution that these labels are survey-based and not diagnostic.
• Auditable model development: regenerating training data from source signals so that results can be re-derived and checked.
• Cross-device transfer: adapting the pipeline to lab physiology (WESAD) or new sensor schemas through configuration rather than a new annotation effort.

Industry relevance: the pipeline and caption layer give device makers, digital-health companies, and health-AI teams a reproducible way to build and extend sensor–language training data without depending on a closed third-party corpus or on non-deterministic model-generated annotations. The paper's stated concern about misuse, for example by insurers or employers, is relevant to any organization deploying such models.

Future Directions

• A per-channel data-availability gate before structural detection, so that missing or zero-filled periods are not captioned as events.
• Constrained language-model post-processing that could make captions more readable while keeping them grounded in the signal, alongside the untested comparisons against LLM-generated captions and frontier Vision-Language Models prompted on rendered signal plots.
• Broader linguistic diversity, since the current captions come from fixed template banks with 20 variants per caption type.
• Extension to other cohorts, devices, and domains, which the authors say requires only new channel definitions, with high-frequency signals such as ECG additionally needing dedicated feature extraction, and with the same pipeline applicable to industrial and energy time series.

Target Audience

Machine learning researchers working on time-series language models and sensor–language alignment; health-data engineers who need reproducible, auditable annotation pipelines; and clinical or population-health researchers who want an open paired corpus for wearable data. Familiarity with language model fine-tuning and wearable sensor channels helps, but the pipeline description is accessible to readers without a background in captioning.

Authors’ abstract

Time-series data generated continuously across industrial, environmental, and especially health domains produce multimodal signals whose scale and heterogeneity make their processing and interpretation particularly challenging. Time-Series Language Models (TSLMs) promise scalable interpretation of these signals, yet remain bottlenecked by a fundamental lack of text-annotated time series pairs that are signal-grounded, reproducible, and extensible across modalities and schemas. Existing paired corpora are either closed or limited in scale and modality coverage. Annotation with large language models could scale them up, but its outputs cannot be regenerated exactly, which prevents others from re-deriving, extending, or auditing the resulting corpus. To address this, we release TimeCap, an open deterministic captioning pipeline with three reusable extractor tiers and a cross-channel synthesis layer that operates directly on the signal with no language model in the loop, extensible to new schemas by writing channel definitions alone. This positions TimeCap as a reproducible mechanism for future dataset generation rather than a one-off release. Applying TimeCap to OpenMHC, the MyHeart Counts wearable dataset, yields MHCap, the largest open caption layer for multimodal wearable time-series data. MHCap spans 19 Apple HealthKit channels and 7,005,763 captions over 908,944 person-days. To showcase the impact of the corpus, we train OpenTSLM, a time-series language model, on MHCap and benchmark on health-outcome prediction. Caption-augmented training (OpenTSLM-Cap) outperforms a label-only baseline on 29 of 32 per-user binary evaluations across 8 tasks.