Causal Reinforcement Learning based Agent-Patient Interaction with Clinical Domain Knowledge

AI summary

Overview

Research area: Causal reinforcement learning for human-robot interaction in healthcare, specifically socially assistive robotics for dementia care.

Technical level: Intermediate. The paper combines standard Q-learning with causal discovery tools (PC, LiNGAM, EconML), so readers need some familiarity with RL and causal inference terminology, but the environment design and reward structure are described concretely.

Scope: The paper proposes a causal structure-aware reinforcement learning (CRL) framework that embeds a learned causal directed acyclic graph (DAG) of patient states and robot actions into the policy optimization loop, validated in a simulated person-with-dementia (PwD) robot-assisted reminiscence therapy scenario.

What This Paper Is About

Reinforcement learning is attractive for adaptive healthcare interventions, but clinical settings such as dementia care have scarce data, require safe exploration, and demand decisions that can be clinically justified — properties that standard model-free RL lacks. Existing dementia-care systems fall into two unsatisfying camps: opaque model-free RL agents, or rigid rule-based and data-heavy agents that cannot adapt to individual patients. The paper's goal is to close that gap by having the agent learn an explicit causal graph of how its actions affect patient cognitive-emotional states, and then use that graph as clinical domain knowledge to guide — and accelerate — policy learning.

Key Contributions

1. A causality-aware RL framework that embeds clinical domain knowledge. The authors integrate causal discovery and causal reasoning directly into policy optimization, which they describe as the first work to integrate expertise in this manner for efficient reasoning in agent-patient interactions.

2. Improved sample efficiency and generalization. The causality-informed agent infers the consequences of actions rather than relying on exhaustive and potentially unsafe trial-and-error, reducing data dependence.

3. Validation in a simulated cognitive rehab (reminiscence therapy) environment. The CRL agent reportedly outperforms traditional RL baselines in performance, sample efficiency, and transparency.

4. A lightweight LLM deployment path. A fixed policy is embedded into a system prompt that maps inferred states to actions, producing consistent, supportive dialogue without any LLM finetuning.

Main Findings

• Learned causal structure matched clinical expectations. The DAG learned via LiNGAM revealed dependencies between the patient's current states, robot actions, and the patient's subsequent states. Expert evaluation by two dementia nurse practitioners confirmed the DAG's alignment with dementia care domain knowledge and suggested directions for refining the state representation.

• Supportive actions showed large estimated treatment effects. Estimated CATE values indicate that Explain increases expected reward by +4.3 under IR_Neu_Yes states, while Comfort improves outcomes by +2.3 under NR_Neg_Yes.

• CRL-Dynamic accelerated early policy learning. Under RL-only execution, CRL-Dynamic achieved superior early performance by leveraging causal priors, while Model-Free RL eventually approached comparable returns with sufficient training. DAG-Only remained limited because it lacks reward-driven adaptation.

• Causal priors helped learning but not execution. Under strategy-consistent evaluation (where the training-time RL+DAG weighting is retained at test time), the proportion of high-return episodes dropped significantly for CRL and DAG, whereas RL sustained the highest success rate. The authors conclude that causal priors are most beneficial during learning and should be gradually phased out for execution.

• High-return episode analysis. Using a threshold of return > 150, CRL-Dynamic consistently maintained a higher proportion of high-return episodes than DAG-only and approached pure RL performance while converging faster — under RL-only execution.

• Episode length and persistence. Under RL-only execution, both CRL variants achieved stable dialogue lengths comparable to pure RL while converging faster. Under strategy-consistent evaluation, CRL-Static and DAG-only degraded: average episode length shortened and the proportion of long episodes (≥ 40 steps) declined, while RL sustained the highest persistence rates.

• Low-return episodes. Tracking episodes with return < 50, CRL-Dynamic quickly reduced the frequency of low-return episodes to stability comparable with RL, while DAG-only maintained a higher failure rate.

• LLM integration worked without finetuning. The GPT-5 policy generated consistent, state-aware dialogue grounded in a Glacier Bay image used as the session hook. It produced coherent and emotionally appropriate responses across five of the six reachable actions in the policy array (Comfort, Explain, ProvideModeratelyPrompt, Repeat, ProvideEasyPrompt).

• A noted LLM behavior. Qualitative inspection showed GPT-5 sometimes interpreted a patient's inability to elaborate on a prior topic (for example, struggling to describe the sky-color) as a confused state rather than a disengaged or irrelevant one — a "semantic continuity preference" that enhances interactional smoothness but slightly reduces state discriminability.

• Interpretable, clinically aligned behavior. The authors report that CRL agents not only achieve higher cumulative rewards but maintain desirable patient states more consistently and exhibit behavior aligned with clinical expectations.

Methodology in Plain English

The researchers built a simulated robot-assisted reminiscence therapy setting in which a virtual patient with dementia responds to a robot's conversational prompts with dynamic emotional, cognitive, and engagement states.

State space. The patient's condition is captured in three dimensions: response relevance (no response NR, irrelevant response IR, or relevant response RR), emotional state or pleasure (negative, neutral, positive), and confusion condition (no or yes). Combining these gives 3 × 3 × 2 = 18 discrete states.

Action space. The action space is described as six robot actions — EasyPrompt, ModeratePrompt, DifficultPrompt, Repeat, Explain, and Comfort — designed to stimulate engagement, support comprehension, or provide emotional reassurance. The experiments section refers to an 18-state space with 7 robot actions, where the 7th action is deterministic; the algorithm also references a sixth-indexed action, a6 (GiveChoice).

Reward function. The reward combines a response–action term (for example, −2 for no response regardless of action; 3 for a relevant response following DifficultPrompt), an emotion term (−3 negative, 1 neutral, 2 positive), and a confusion term (−2.5 if confused, 2 if not). Additional terms include an optional per-round shaping term η (default 0), a bonus δ for using a new memory trigger, an early-termination penalty λ_stop = −200, and a goal bonus λ_goal = 15 for completing all triggers or reaching the 50-round cap.

Causal discovery. The researchers first generated simulated interaction trajectories in which patient states evolve in response to random robot actions, logging pre-action states, actions, post-action states, subsequent actions, and rewards. This random-action dataset maximizes coverage of the state–action space. They then applied causal discovery algorithms such as PC and LiNGAM to infer a DAG linking current states, actions, and subsequent states, and used the EconML library to estimate Conditional Average Treatment Effects (CATE) — the relative impact of each action versus a baseline action a0 (EasyPrompt) on reward, given different patient states.

Causal-guided Q-learning. The CRL algorithm (Algorithm 1) is a causal structure-aware Q-learning procedure. At each step, if the agent observes negative emotion or confusion persistence, it takes action a6 (GiveChoice). Otherwise it computes mixed policy weights (w_RL, w_DAG) according to a CRL-static or CRL-dynamic rule, samples a coin from [0,1), and then either takes an ε-greedy action with respect to Q, takes the action suggested by the learned DAG, or takes a random action. The Q-value is updated after each executed action.

Training and evaluation. All methods were trained for 1500 epochs of 30 episodes each, with evaluation averaged over 100 independent test runs using the final policy from the last training epoch. Shared hyperparameters were a learning rate of 0.05, discount factor γ = 0.95, and ε-greedy exploration with ε = 0.1. Comparisons covered Model-Free RL (no causal information), DAG-Only (pure DAG action selection ignoring rewards), CRL-Static (fixed weighting of DAG guidance and RL signals), and CRL-Dynamic (adaptive weighting that shifts reliance from DAG to RL over training epochs). Weight configurations were: RL 0.9/–/0.1; CRL-Static 0.45/0.45/0.1; CRL-Dynamic 0.2→0.7 and 0.7→0.2 with 0.1 exploration; DAG –/0.9/0.1.

LLM deployment. For dialogue generation, a fixed policy was embedded into a system prompt mapping inferred states (RP, E, C) to actions, enabling a GPT-5 policy to produce supportive dialogue without gradient-based optimization.

Why This Matters

Impact on research. The paper demonstrates a concrete way to combine causal discovery with reinforcement learning in a human-centered clinical domain, rather than in the synthetic benchmarks where causal RL is usually tested. Its most transferable insight is a negative result as much as a positive one: causal structure is valuable as an inductive bias during training, but retaining it at execution time degrades high-return episode rates and interaction persistence. That finding has direct implications for how other causal-RL-for-HRI systems should be designed and evaluated.

Real-world applications:

• Socially assistive robots supporting reminiscence therapy and cognitive rehabilitation for persons living with dementia.
• Conversational AI companions in long-term care that adapt dialogue to inferred emotional and confusion states.
• Clinical decision-support for care staff, where the explicit DAG and CATE estimates provide interpretable, clinically justifiable rationales for suggested interaction strategies.
• Safe online learning in high-stakes settings where exhaustive trial-and-error is unacceptable, using causal priors to reduce the exploration burden.

Industry relevance. The LLM integration path is notable for deployment economics: because the fixed policy lives entirely in a system prompt, no finetuning is required, which lowers the barrier to embedding causal policy mappings into existing conversational systems. The paper's code and datasets links are marked "will be released upon publication," so reproducibility currently depends on the authors following through.

Future Directions

• Validation beyond simulation. The authors state their validation is limited to simulation and that future work will test the framework with human stakeholders.
• Expanding the causal model. The paper calls for extending the causal model to capture more complex, long-term patient dynamics.
• Refining the state representation. Expert feedback from the two dementia nurse practitioners suggested directions for refining the state representation, which the authors explicitly note is beyond this work's focus.
• Better state discriminability in LLM-driven dialogue. GPT-5's tendency to read a patient's inability to elaborate as confusion rather than disengagement or irrelevance introduces a trade-off between conversational smoothness and accurate state inference that the paper surfaces but does not resolve.

Target Audience

This paper suits researchers and practitioners working at the intersection of reinforcement learning, causal inference, and human-robot interaction — particularly those building socially assistive robots or conversational agents for healthcare. It is also relevant to clinical informatics and nursing researchers interested in how causal models can encode domain expertise, and to engineers considering hybrid CRL–LLM control architectures for deployment where interpretability and data efficiency matter more than raw asymptotic performance.

Authors’ abstract

Reinforcement Learning (RL) faces significant challenges in adaptive healthcare interventions, such as dementia care, where data is scarce, decisions require interpretability, and underlying patient-state dynamic are complex and causal in nature. In this work, we present a novel framework called Causal structure-aware Reinforcement Learning (CRL) that explicitly integrates causal discovery and reasoning into policy optimization. This method enables an agent to learn and exploit a directed acyclic graph (DAG) that describes the causal dependencies between human behavioral states and robot actions, facilitating more efficient, interpretable, and robust decision-making. We validate our approach in a simulated robot-assisted cognitive care scenario, where the agent interacts with a virtual patient exhibiting dynamic emotional, cognitive, and engagement states. The experimental results show that CRL agents outperform conventional model-free RL baselines by achieving higher cumulative rewards, maintaining desirable patient states more consistently, and exhibiting interpretable, clinically-aligned behavior. We further demonstrate that CRL's performance advantage remains robust across different weighting strategies and hyperparameter settings. In addition, we demonstrate a lightweight LLM-based deployment: a fixed policy is embedded into a system prompt that maps inferred states to actions, producing consistent, supportive dialogue without LLM finetuning. Our work illustrates the promise of causal reinforcement learning for human-robot interaction applications, where interpretability, adaptiveness, and data efficiency are paramount.