BALMS: Benchmarking Agentic LLMs for Longitudinal Mental Health Sensing

AI summary

Overview

Research area: Natural Language Processing / LLM-based agentic systems applied to longitudinal mental health sensing from smartphone and wearable data.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, retrieval-augmented generation, ReAct-style tool loops, mean absolute error, and LLM-as-a-Judge evaluation, but explains each paradigm clearly enough for readers from adjacent fields.

Scope in one sentence: BALMS is the first systematic benchmark of LLM-based agentic systems for longitudinal mental health sensing, spanning 3 real-world datasets, 2 task families, 3 agentic paradigms, and 5 LLM backbones.

What This Paper Is About

Mental health assessment today depends on episodic self-report questionnaires, which turn subjective states such as stress into numbers but capture only sparse snapshots. Wearables collect continuous behavioral and physiological signals over weeks to years, and recent LLM-driven health agents can answer natural-language queries over those signals — but existing systems mainly handle short-term factual lookups (for example, the highest step count over a week) rather than predicting wellbeing scores from long-term patterns with evidence-grounded explanations. BALMS fills this gap by formalizing longitudinal mental health sensing as an agentic benchmark where systems must jointly produce a numerical wellbeing score and a rationale that can be checked against the sensing window.

Key Contributions

1. Task formulation: The authors formalize longitudinal mental health sensing as an agentic benchmark requiring models to jointly generate verifiable numerical wellbeing scores and evidence-grounded rationales over long-term passive sensing histories.
2. Unified evaluation: They implement and evaluate 3 core agentic paradigms (prompt-based, tool-based, and memory-based) across 5 open- and closed-source LLM backbones, establishing standardized infrastructure for clinical agent assessment.
3. Empirical agent design insights: They show zero-shot agentic prediction remains challenging except with stronger backbones or compact, semantically meaningful sensor features, and that chain-of-thought prompting significantly boosts reasoning-tuned backbones — with agents benefiting far more from selective memory and semantically meaningful features than from raw sensor streams or extended context windows.
4. Failure taxonomy for tool-based agents: A dedicated analysis of PHIA explains why tool-mediated reasoning fails on some datasets, documenting mode collapse, silent code execution, lost state across action blocks, hallucinated rationales at force-finish, invented thresholds, and schema-blind aggregation.

Main Findings

• Zero-shot agents rarely beat a mean baseline. The mean predictor achieves 0.58 MAE on DiversityOne (mood), 0.48 on PMData (stress), and 0.80 on GLOBEM (anxiety). Claude-Haiku-4.5 with Health-LLM reaches 0.42 MAE on DiversityOne, and PHIA reaches 0.54 MAE on PMData, close to the 0.48 baseline. Smaller models such as Mistral-7B frequently fail to match the mean predictor.
• No single paradigm wins on all three datasets. Table 3 shows that prompt-, tool-, and memory-based systems each lead in different cells, and scaling the backbone helps prompt-based and memory-based agents but not consistently the tool-based PHIA.
• Tool-based agents are schema-sensitive, not just language-limited. PHIA excels only on PMData, where Fitbit-style aggregates match its design, and fails on DiversityOne and GLOBEM where raw mobile streams make generated code brittle.
• Chain-of-thought helps reasoning-oriented backbones. With DeepSeek-R1-Distill-Qwen-14B and Claude-Haiku-4.5, +CoT consistently reduces MAE across most configurations, with improvements up to 41.4%.
• Fluency is not temporal grounding. Open-source instruct backbones produce fluent rationales but rarely invoke temporally grounded evidence — invocation of C1 alignment and C5 structure is near zero. Only Claude-Haiku-4.5 closes both the rationale-quality and temporal-grounding gaps, reaching the highest overall rationale quality while consistently invoking every temporal reasoning dimension.
• Longer histories help selectively. With Qwen2.5-14B-Instruct, RAG MAE decreases about 29% on PMData as more historical records become available, but only about 9% on GLOBEM. Health-LLM degrades as the look-back window grows. PHIA stays stable on PMData because preloaded DataFrames decouple tool execution from prompt length, but remains weaker and noisier on GLOBEM and incurs higher latency.
• More sensors do not automatically help. On GLOBEM, dropping mobile streams to a smartwatch-only Fitbit configuration frequently matches or surpasses the full multimodal setup (Table 4), suggesting contemporary LLMs reason better over high-level, semantically rich behavioral primitives than over raw high-frequency streams.
• Tool-based agents collapse to a single label. With Qwen2.5-14B-Instruct (seed 0), PHIA's mode rate is 85.9% (1409/1640) on GLOBEM with MAE 1.52, 77.4% (1132/1463) on PMData with MAE 0.57, and 99.5% (757/761) non-parseable answers on DiversityOne with MAE 2.11. The apparent PMData success reflects agreement between a collapsed prior and the dataset mean rather than grounding in the wearable record.
• Safety scores are saturated. In the Tier-2 general-quality judge results on GLOBEM, Safety is omitted because every cell scores 5.0 on the 1–5 scale.

Methodology in Plain English

The researchers built a benchmark from 3 already-collected longitudinal datasets. DiversityOne covers 782 university students across 8 countries over 28 days with 10 phone-derived sensor channels and a daily evening mood self-report; evaluation is restricted to the Mongolia cohort (highest daily mood-answering rate), yielding 1,355 target-day/user samples and a 7-day look-back. PMData tracks 16 participants over 5 months with 5 Fitbit-derived channels (steps, resting heart rate, burned calories) and daily PMSys 1–5 wellness self-reports, using stress as the target — 1,448 samples with a 14-day look-back. GLOBEM pairs smartphone and Fitbit sensing from 497 college students with weekly EMA PHQ-4 anxiety self-reports across 10-week study windows over four consecutive years; the authors use the INS-W_1 cohort and the PHQ-4 anxiety subscale (0–3), giving 1,640 samples with a 14-day look-back.

Two tasks are defined. T1 asks for a single integer wellbeing score on the dataset's native Likert scale given a target day and look-back window, following Health-LLM's zero-shot setting and scored by MAE. T2 asks for a free-form chain-of-thought rationale alongside the score, citing specific sensor channels and values and reasoning about multi-day patterns such as trends, weekly cycles, or recovery; because this cannot be graded numerically, it is scored by a single Llama-3.3-70B-Instruct judge given the sensing window, gold score, agent prediction, and rationale.

Three agentic paradigms represent the design space. Prompt-based reasoning uses Health-LLM's template verbatim, formatting each sensor modality as a per-day numeric array with demographics and task instructions in one prompt. Tool-based agents follow PHIA: each user's record is preloaded as in-memory pandas DataFrames, and the model drives a ReAct loop generating executable Python at each step, capped at 10 iterations. Memory-based agents chunk memory at the day level rather than at token level — Chunk RAG encodes each historical day with a sentence-transformer and retrieves the top-3 most similar days by cosine similarity, while RAPTOR builds a two-level memory tree whose leaves are sub-day segments (half-hour slots on PMData, four time-of-day segments on GLOBEM) and whose daily internal nodes are LLM-generated summaries, retrieved with the same top-3 budget. Each system runs across 5 backbones: Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Mistral-7B-Instruct-v0.3, DeepSeek-R1-Distill-Qwen-14B, and Claude-Haiku-4.5. Open-source models are served locally with vLLM on 4 A6000 GPUs. The T2 rubric has two tiers: Tier 1 adapts TemporalBench to check six temporal operations (C1 Alignment, C2 Slicing, C3 Difference Judgment, C4 Lag, C5 Structure, C6 Interaction), scored correct, incorrect, or not invoked, and reported as invocation rate, conditional accuracy given invocation, and coverage; Tier 2 adapts PHIA's expert rubric to score Faithfulness, Evidence Grounding, Domain Knowledge, Safety, Clarity, and an Overall holistic score on a 1–5 Likert scale.

Why This Matters

Impact on research. BALMS is presented as the first systematic benchmark of agentic LLMs for longitudinal mental health sensing, providing a standardized comparison of agentic paradigms that prior work evaluated only in isolation. It moves the field beyond short-window factual lookups and case-study explanations toward paired score prediction and verifiable rationales, and Table 2 explicitly positions prior systems (Health-LLM, Englhardt et al., Chunk RAG, RAPTOR, PHIA, GLOSS, PHA, LifeAgentBench) along axes of rationale, raw passive sensing data, tools, memory, and dataset-centric design.

Real-world applications:

• Continuous, low-burden wellbeing monitoring that returns feedback to users between clinical visits rather than replacing diagnosis, so users can make sense of a wellbeing score and its textual explanation.
• Clinical decision support, where an agent's rationale can be audited against the actual sensing window instead of accepting an opaque label.
• Personalization across sensing setups, helping systems work whether a user has only a smartwatch or a full phone-plus-wearable stream.
• Deployments across heterogeneous cohorts and countries, where sensor schemas and observation horizons differ substantially.

Industry relevance. Wearable and mobile platform vendors, digital mental health companies, and clinical tooling providers all stand to gain from knowing which agentic design actually extracts signal from consumer sensor data. The finding that a compact, smartwatch-only Fitbit configuration often matches the full multimodal setup is directly relevant to product decisions about which data streams are worth collecting, transmitting, and storing. The efficiency results also matter operationally: prompt- and memory-based paradigms avoid recursive execution workflows, whereas tool-based agents carry an extensive latency penalty.

Future Directions

• Design agents that explicitly build in temporal operations for comparing behavioral baselines, detecting sustained changes, and analyzing sensor interactions, since open-source instruct backbones rarely invoke alignment (C1) or structure (C5).
• Extend retrieval beyond context reduction so memory is used for temporal reasoning over personal baselines and behavioral shifts, rather than only for fitting within context limits.
• Move beyond single-agent paradigms. The authors state they evaluate only single-agent prompt-, tool-, and memory-based systems and leave multi-agent collaboration, planner-executor architectures, and agents that dynamically combine tools, memory, and reflection unstudied.
• Validate outside retrospective prediction. The limitations note that evaluation does not test agents in real-world interactive or intervention settings, and that the LLM-as-Judge rubric does not replace expert clinical or human-subject evaluation; future work should incorporate clinician and user assessments, prospective deployment, and study safety, calibration, and personalization.
• Handle cross-term discontinuities. Because GLOBEM is collected as separate 10-week terms, the authors evaluate only one continuous term and only the anxiety subscale, leaving changing participants, calibration drifts, and term-specific behavioral baselines to future work.

Target Audience

This paper is most useful to researchers building LLM agents over health and sensor data, benchmark designers working on evaluation of agentic reasoning, and clinical informatics groups interested in continuous wellbeing monitoring. Digital health and wearable product teams will find the paradigm comparisons, latency discussion, and sensor-set sensitivity analysis directly actionable. It is also relevant to clinicians and mental health researchers who want to understand what current LLM agents can and cannot do with longitudinal sensing data — bearing in mind the authors' caution that these systems are intended as clinical decision-support tools, not autonomous diagnostic replacements, and that the benchmark remains a retrospective, LLM-judged evaluation rather than clinical validation.

Authors’ abstract

Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing. Wearable devices offer longitudinal behavioral and physiological signals for continuous, low-burden monitoring. Recent LLM-driven personal-health agents enable natural language queries over wearable signals, but mainly handle short-term, retrieval-based lookups (e.g., highest step count over a week). They do not evaluate whether agents can reason over long-term signals to predict wellbeing scores paired with evidence-grounded rationales. To address this gap, we introduce BALMS, the first systematic benchmark of LLM-based agentic systems for longitudinal mental health sensing. BALMS spans 3 real-world longitudinal datasets, 2 task families (closed-form wellbeing-score prediction and rationale generation auto-graded by an LLM-as-Judge), 3 agentic paradigms evaluated across 5 open- and closed-source LLM backbones. We find that zero-shot agents rarely outperform a simple mean baseline, except with stronger backbones or compact, semantically meaningful features. Chain-of-thought prompting improves reasoning-oriented backbones, but does not guarantee temporal grounding or numerical correctness. Together with more analysis on efficiency and temporal scaling, BALMS highlights the need for longitudinal mental health agents that selectively retrieve history, ground temporal evidence, and reason over interpretable behavioral features.