Offline Policy Evaluation of Multi-Turn LLM Health Coaching with Real Users

AI summary

Overview

• Research area: Large language model (LLM) agents for health coaching, offline policy evaluation (OPE), multi-turn dialogue personalization, and reinforcement-learning-style decision making over tool use and response style.
• Technical level: Intermediate. The paper uses standard RL and causal-inference vocabulary (POMDP, SNIPS, AIPW, importance weighting, calibration), but the core arguments are stated in plain terms and the empirical base is small and descriptive.
• Scope: A workshop paper (Multi-Turn Interactions in Large Language Models) reporting early findings from a seven-user deployment of a tool-augmented LLM health coach, plus an OPE analysis of two-factorized decision heads (Tool, Style) and a lightweight archetype simulator testing an early information-gain bonus.

What This Paper Is About

Most LLM health-coaching research is evaluated on synthetic benchmarks, so little is known about how such agents perform with real users over multiple turns. This paper analyzes logs from a deployed, tool-augmented LLM health coach with seven real users, uses offline policy evaluation to ask whether different tool-use and response-style policies would have produced better outcomes, and tests in simulation whether a brief early "exploration" phase helps the agent learn what kind of user it is talking to. The goal is an evaluation-first path to personalization: keep the LLM generator frozen, learn subgroup-aware decision heads on typed rewards, and always report per-subgroup metrics.

Key Contributions

1. Offline policy evaluation on real deployment logs. The authors reconstruct the logging behavior policy for two factorized decision heads (Tool and Style), calibrate it, and compare four counterfactual policies (NoTool, AlwaysTool, HeuristicGated, PersonalizedWeights) using self-normalized importance sampling (SNIPS) and augmented inverse propensity weighting (AIPW) with ratio clipping at c=50 and session-level bootstrap confidence intervals.
2. A typed reward and action formulation for health coaching. The system is modeled as a user-conditioned POMDP where each turn's action factorizes into Tool (∅, Search, Code, Email) and Style (concise, detailed), and reward combines a user/satisfaction component, a rubric-scored tool-outcome component, and a small engagement component, with an optional early information-gain (curiosity) bonus.
3. Demonstration of subgroup harm hidden by averages. The paper shows that a policy maximizing average reward on the logs produces large, structured differences across literacy-by-self-efficacy archetypes, including strongly negative effects for low-health-literacy/high-self-efficacy users on both objective and satisfaction measures.
4. A lightweight hidden-archetype simulator. The simulator tests whether a bounded early curiosity term (applied for the first two turns) speeds up trait identification and improves goal success, pass@3, and final return, alongside released de-identified logs and evaluation/simulation code.

Main Findings

• Tool use is high-variance with typed, fixable failure modes. Tool-invoked responses averaged 4.02 versus 4.38 for non-tool responses (Cohen's d = -0.40), tool responses had 1.40 times higher variance (σ² = 0.94 vs. 0.67), and the overall success-versus-failure rating gap was +0.50. Per-category gaps were +0.48 for Web Searcher, +0.76 for Code Executor, and 0.00 for Email (n = 7 turns, 1 failure).
• Success rates by tool are similar and imperfect. Web search 81.6%, code execution 80.7%, email 85.7%, with top failures labeled no_url_provided and no_visualization.
• A uniform heavy-tool policy wins on average but harms a specific subgroup. AlwaysTool attained the highest literacy-weighted R_total on the logs (0.304, 95% CI [0.001, 0.524]), compared with NoTool 0.044 [-0.045, 0.198], HeuristicGated 0.006 [-0.111, 0.174], and PersonalizedWeights 0.113 [-0.016, 0.284].
• Per-archetype results reverse the average story. For AlwaysTool minus NoTool, L_high × E_high was +0.575 objective and -0.107 satisfaction; L_high × E_low was +0.595 and +0.525; L_low × E_low was +0.165 and -0.431; L_low × E_high was -0.315 objective and -1.436 satisfaction. Heavy tool use therefore helped high-literacy/low-efficacy users on both components and harmed low-literacy/high-efficacy users on both.
• Performance degrades over the course of a conversation. Ratings declined from 4.36 (turns 1-5) to 4.12 (turns 15+); tool usage peaked at 70% (turns 5-10) then fell to 26.3% for turns 15+; latency peaked at 25s at turns 6-10; and message length decreased 3.4% from early to late turns.
• Baseline user traits correlate with behavior. Health literacy versus tool usage r = 0.302; self-efficacy versus conversation length r = -0.441. Users with above-median self-efficacy averaged 4.3 ± 0.5 ratings versus 3.9 ± 0.7 for below-median. Low-literacy/high-efficacy users reported the highest median ratings (4.5) and high-literacy/low-efficacy the lowest (3.8).
• Unconditional between-user variance is tiny, but context-conditioned heterogeneity is large. Only 1.6% of variance in unconditional ratings was attributable to user identity (ICC = 0.016), while per-archetype differences under counterfactual policies were large.
• Implicit feedback tracks satisfaction. Response time negatively correlated with ratings (r = -0.18): responses under 10s averaged 4.25 versus 3.98 for those over 30s. Message length showed a small positive association (r = 0.07), with high-rated messages (4-5) averaging 2,123 characters versus 1,275 for low-rated (1-2). Ratings showed no meaningful lag-1 temporal autocorrelation (r = 0.10).
• A small early exploration bonus helps in simulation. Adding the curiosity term for the first two turns improved goal success (0.935 to 0.970), pass@3 (0.950 to 0.980), and final return (-3.162 to -2.329), and reduced trait-identification time (6.415 turns to 5.655 at λ=0.10, and 5.860 at λ=0.20), with a small expected dip in archetype-aligned action rate (0.424 to 0.410-0.412) caused by early probing.
• The OPE pipeline appears stable enough to guide offline iteration. Diagnostics reported 23 sessions, rating rate 0.80, rating-propensity AUC 0.712, clipping rate at c=50 of at most 0.29%, Expected Calibration Error of 0.157 for the Tool head and 0.050 for the Style head, and 82.6% agreement between conv_len and turn_index with 1.1% boundary conflicts.

Methodology in Plain English

The authors deployed a web-based health coach built on Qwen3-235B-A22B, which ingests a user's Apple Health export, preprocesses it into daily features (sleep, HRV, VO2 max, activity), and generates machine-learning predictions for stress, soreness, and injury risk (rolling-origin cross-validation, R² = 0.50, 0.28, 0.40 on 1,280 day-samples with N = 28). The agent can call three tools: Code Executor, Web Searcher, and Email. Seven students completed baseline surveys (eHEALS for health literacy and the General Self-Efficacy Scale), then held multi-turn dialogues, typically over more than an hour and with 30+ messages, rating each response from 1 to 5 stars.

Each tool call is scored with a fixed rubric: 0 when no tool is used, +1 for a success (for example, a web result with at least one URL and consistent evidence, code that runs and produces a labeled plot or table, or an email with an ISO8601 date/time), and -1 for failure. A small bounded engagement signal is added on all turns. The total per-turn utility is a weighted combination of user satisfaction, tool outcome, and engagement, with weights set deterministically by literacy stratum (low-literacy: 0.6, 0.2, 0.2; high-literacy: 0.3, 0.5, 0.2).

To compare policies without new user trials, the authors treat the problem as a per-turn contextual bandit and reconstruct the logging policy for each decision head with cross-fitted, calibrated classifiers, checking calibration with Expected Calibration Error. They then reweight logged outcomes, clip importance ratios at 50, and compute SNIPS for the objective component and AIPW for satisfaction, with a rating-propensity model to correct for missing ratings and session-level bootstrap confidence intervals. They are explicit that they do not claim unbiased long-horizon policy value.

Finally, they built a simulator with four hidden archetypes (low/high literacy crossed with low/high self-efficacy) that returns simulated replies, rubric-based ratings, and tool outcome labels, and compared policies with and without an information-gain bonus defined as the reduction in entropy of the archetype posterior, applied only in the first two turns with λ of 0.10 or 0.20. The base generator stays frozen throughout; policies act only through the discrete decision heads. No end-to-end RL policy is trained.

Why This Matters

For research, this paper argues that average reward is the wrong headline metric for personalized agents. The same policy that tops the leaderboard on mean value can systematically harm a subgroup, and the paper shows this on real logs rather than in a synthetic benchmark. It also offers a concrete, reproducible OPE recipe — behavior-policy reconstruction, calibration diagnostics, clipping, selection correction, and per-archetype reporting — for teams that cannot run repeated user trials.

Real-world applications:

• Consumer health and wearable coaching apps: products that decide turn by turn when to run code over health data, search the web, or set a reminder, and how much detail to give.
• Chronic-condition and behavior-change programs: interventions where over-tooling a user with low health literacy but high confidence could erode engagement.
• Clinical triage and patient-education assistants: settings where typed, verifiable tool outcomes (did the calculation run, did the source resolve) can serve as an objective reward signal.
• Any deployed conversational assistant with logged ratings: the subgroup-aware OPE recipe generalizes to customer support, education, and fitness coaching, where a proxy metric (satisfaction) plus typed outcomes can be combined into a reward.

Industry relevance is direct: the pipeline lets product teams compare candidate decision policies offline against frozen generative models, gate releases on diagnostics such as propensity AUC, ECE, and clipping rate, and avoid the cost and risk of on-policy A/B tests before a policy has been screened for subgroup harm. The authors also publish de-identified logs and code, lowering the barrier to reproducing the analysis.

Future Directions

• Add self-efficacy to the weighting and routing. The current evaluation weights, gating, and personalized routing are conditioned on literacy only; the authors state that incorporating self-efficacy is a planned extension, which matters because the worst-hit subgroup in their results is defined by that axis.
• Move from reconstructed to logged propensities. Behavior propensities were reconstructed rather than recorded, and Tool-head calibration is moderate (ECE 0.157), which the authors note can bias IPS/SNIPS estimates; future deployments should log propensities directly.
• Train and prospectively validate the policy. The paper does not train an end-to-end RL policy and provides no prospective validation; it proposes an offline-first loop in which population head policies are fit from logs while the generator stays frozen, user weights initialize from cluster priors, and deployments are gated by OPE diagnostics.
• Broaden the evidence base. The deployment sample is small (N = 7) and demographically narrow, and the authors note this limits generalizability; a larger, more diverse study would test whether the archetype-level harm patterns and the probe-then-personalize benefit hold.

Target Audience

Researchers and practitioners working on LLM agents, conversational health coaching, and personalization, especially those interested in offline policy evaluation and subgroup fairness in deployed systems. It is most useful to applied scientists and engineers who need to evaluate counterfactual decision policies from logged data without new user trials, and to product and clinical stakeholders who need to understand why average satisfaction or engagement numbers can mask harm to specific user groups. Readers without a background in causal inference or bandit evaluation will need to look up SNIPS, AIPW, and calibration terminology, since the paper uses them without extended explanation.

Authors’ abstract

We study a web-deployed, tool-augmented LLM health coach with real users. In a pilot with seven users (280 rated turns), offline policy evaluation (OPE) over factorized decision heads (Tool/Style) shows that a uniform heavy-tool policy raises average value on logs but harms specific subgroups, most notably low-health-literacy/high-self-efficacy users. A lightweight simulator with hidden archetypes further shows that adding a small early information-gain bonus reliably shortens trait identification and improves goal success and pass@3. Together, these early findings indicate an evaluation-first path to personalization: freeze the generator, learn subgroup-aware decision heads on typed rewards (objective tool outcomes and satisfaction), and always report per-archetype metrics to surface subgroup harms that averages obscure.