ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

AI summary

Overview

Research area: Natural Language Processing, specifically reinforcement learning for large language model dialogue agents (hierarchical RL, RLHF-style post-training, conversational strategy modeling).

Technical level: Advanced. The paper assumes familiarity with Markov decision processes, Q-learning/DQN, PPO and actor-critic methods, the options framework, and LLM post-training pipelines.

Scope: The paper proposes and evaluates a two-level hierarchical reinforcement learning framework, ToSCA, that lets an LLM-based conversational agent first pick an explicit textual dialogue strategy at the utterance level and then generate the response token by token conditioned on that strategy.

What This Paper Is About

Most reinforcement learning methods for conversational LLMs operate only at the token level, which produces sparse rewards and expensive exploration, especially in multi-turn dialogue where meaningful user feedback arrives only after a full utterance is generated. Humans instead reason at two scales: they choose a communicative intent (such as question, inform, or directive) and then word a response to fit it. ToSCA models both of these scales inside a single hierarchical reinforcement learning system, with a high-level strategy chooser over a discrete, human-readable strategy set and a low-level token-level policy that is conditioned on the chosen strategy.

Key Contributions

1. A hierarchical RL dialogue framework that bridges explicit utterance-level strategic planning and token-level response generation in one architecture, following the options formalism in which low-level actions (response tokens) are conditioned on high-level actions (strategies).
2. A dual-granularity reward mechanism that pairs the utterance-level user satisfaction score with two token-level components — intrinsic self-consistency and a KL-divergence penalty — to alleviate reward sparsity and improve convergence.
3. A specific architecture choice and an accompanying theoretical result: rather than a high-level actor, the framework uses a Q-network trained with DQN for value-based selection over discrete strategies (the authors call it Critic^H-Actor^L-Critic^L), with a theorem stating that the value function converges as long as the high-level policy is deterministic and the high- and low-level critics are solved in an interleaved manner.
4. Extensive experiments across two conversational domains plus an out-of-domain transfer test, covering daily-life conversation (DailyDialog) and emotional support conversation (ESConv), with zero-shot evaluation on EmpatheticDialogues, showing gains over prompting, supervised, and RL baselines.

Main Findings

• Return optimization on DailyDialog. Averaged over five random seeds, ToSCA reaches Q^H of 616.5 ± 1.35 and r^L of 4.03 ± 0.34, compared with Direct at 486.2 ± 1.22 and 3.21 ± 0.32, and the Raw Dataset at 551.4 ± 1.33 and 3.53 ± 0.35. The significance test H0: Metric_X ≥ Metric_ToSCA yields p-values below 0.01 for both baselines and both metrics.
• Strategy determination on DailyDialog. ToSCA achieves the highest accuracy (63.64) and the best Macro-F1 (58.91) even though it never receives reward from ground-truth strategy labels during training. It does not achieve the best bias score (0.63), but the combination of high accuracy and competitive bias gives it the best MaF1. For reference, straQ* reaches Acc 54.01 / MaF1 50.10, ArCHer 50.41 / 42.67, SFT 60.19 / 44.82, and EmoFSM 60.03 / 46.02.
• Response quality on DailyDialog. ToSCA attains the highest Bleu-2 (16.35), Rouge-L (35.22), and Distinct-2 (62.67), outperforming prompting, finetuning, and RL baselines; the closest RL competitor on B-2/R-L is PPO at 7.85 / 25.16.
• In-domain emotional support (ESConv). ToSCA achieves the best or second-best result on every metric: Acc 39.26, MaF1 36.85, bias 0.48, B-2 6.93, R-L 16.28, D-2 52.42. SFT and ArCHer remain competitive here (SFT B-2 6.97, ArCHer D-2 54.80), which the authors attribute to ESConv's narrower topics and more structured behavior patterns.
• Out-of-domain transfer (EmpatheticDialogues). Under a zero-shot setting trained on ESConv, ToSCA leads on all three reported metrics with B-2 4.49, R-L 12.93, and D-2 46.53, ahead of ArCHer (3.84 / 10.75 / 45.19) and PPO (3.91 / 11.16 / 41.59). Finetuning baselines were excluded from this comparison.
• Ablation results. Removing the high level reduces the framework to token-level PPO (B-2 7.85, R-L 25.16, D-2 50.59); removing the low level reduces it to a DQN strategy planner guiding a frozen LLM, equivalent to straQ* (Acc 53.39, MaF1 49.02, B-2 4.17, R-L 13.15, D-2 59.44). Removing intrinsic self-consistency ("w/o isc") lowers every metric relative to full ToSCA (Acc 60.04 vs 63.64, MaF1 53.01 vs 58.91, B-2 16.21 vs 16.35, R-L 35.06 vs 35.22, D-2 60.95 vs 62.67).
• Strategy-following behavior. A qualitative case shows that without intrinsic self-consistency the model selects the Question strategy but produces a statement ("I bet there's something fun to watch on it."), whereas with it the response is a genuine question ("Got anything exciting on there?").
• Learned strategy dynamics. The most frequent same-turn transitions from user to agent are "Question → Inform" and "Directive → Commissive". Across adjacent agent turns, a prominent upper-triangle pattern appears — Inform → Question → Directive → Commissive — which the authors connect to Helping Skills Theory as described in the ESConv work.

Methodology in Plain English

The researchers split the conversation decision into two timescales. At the utterance level, a small LLM (LLaMA3.2-1B-Instruct) is given a multiple-choice prompt containing the session description, the user's emotion, the dialogue history, and the query, and must select one strategy from a fixed list. Instead of adding a value head, the state-action value is read off from the model's averaged logits over the candidate strategy tokens, and this Q-network is trained with the standard DQN objective. At the token level, a second LLM (LLaMA3.1-8B-Instruct) generates the response one token at a time, with the selected strategy inserted into the prompt so that the response is conditioned on it; this actor and its critic are trained with PPO.

Rewards are supplied at both scales. The high-level reward is simply the user satisfaction score, produced automatically by GPT-4o acting as an LLM-as-the-Judge on a 0-to-5 scale. The low-level reward reuses that same satisfaction score but adds two token-level terms: a KL-divergence penalty that keeps the policy from drifting too far from its reference policy, and an intrinsic self-consistency term, defined as the log-likelihood the model assigns to its own response given the low-level context and the chosen strategy, encouraging responses that actually follow the strategy.

Training runs in roll-out/update cycles: the agent plays multi-turn sessions, stores high-level transitions in one buffer and token-level transitions in another, updates the Q-network from the high-level buffer with DQN, and jointly updates the low-level actor and critic with PPO. The framework was trained on OpenRLHF using 16 A100 GPUs, with learning rates of 5e-6, 9e-7, and 9e-4 for Q^H, π^L, and V^L respectively, a batch size of 64, and a maximum decoding length of 128. Evaluation combines automatic strategy classification metrics (accuracy, Macro-F1, and a Bradley-Terry-based bias score), generation metrics (Bleu-2, Rouge-L, Distinct-2), and human ratings on Acceptance, Effectiveness, Sensitivity, Fluency, Emotion, and Satisfaction.

Why This Matters

The paper shows that giving a conversational RL agent an explicit, human-readable middle layer of strategic decisions — rather than either pure token-level optimization or continuous latent intents — can improve both which strategy is chosen and how good the resulting response is, and that these gains carry across domains. It also offers a practical recipe for combining DQN and PPO in a two-level MDP for language agents, which is relevant to anyone building multi-turn dialogue systems with RL post-training.

Real-world applications implied by the work include:

• Emotional support and mental-health-adjacent chat systems, where selecting an appropriate supporting strategy (drawn from ESConv's 8 annotated strategies) matters as much as phrasing.
• Customer service and task-oriented assistants, where agents must decide between informing, asking, directing, or committing before generating a reply.
• Domain transfer scenarios, where a system trained on one dialogue corpus (ESConv) must operate on another without strategy annotations (EmpatheticDialogues).
• Interpretable agent design, since the high-level actions are explicit text strategies that can be inspected, annotated, and audited, unlike continuous latent intents.

Industry relevance centers on the cost profile of RL post-training: because the high-level decision happens once per turn and the low-level policy is conditioned on it, the framework is presented as a way to make reward signals denser and exploration cheaper than purely token-level RLHF-style optimization.

Future Directions

• The paper reports human evaluation of the ESC setting only in Appendix F.2; the truncated content does not give those numbers, and extending full human evaluation across all domains and against the OOD setting is a natural next step.
• Sensitivity to the discount factor γ and the reward weights β1 and β2 is analyzed only in Appendix B.1, leaving open how these must be retuned for new domains.
• The authors observe that baselines vary substantially across domains (for example, SFT and ArCHer on ESConv versus DailyDialog) and explain this partly by topic breadth and behavioral structure; predicting which hierarchical configuration suits which domain remains unresolved.
• Since the high-level reward is supplied by GPT-4o as an automatic judge, replacing or validating that oracle — Appendix F.3 reports consistency checks between oracle and human scores — is an open question for deployment without a strong external judge.

Target Audience

This paper is most useful to RL and NLP researchers working on LLM post-training and multi-turn dialogue, particularly those interested in hierarchical reinforcement learning, reward design, and strategy-aware conversational agents. It also suits applied engineers building dialogue systems that need interpretable intermediate decisions, and graduate students already comfortable with PPO, DQN, and the options framework who want a worked example of hierarchical RL applied to language.

Authors’ abstract

Humans have multiple levels of temporal abstractions on daily interaction and thinking, such as concept perception and strategic planning. Inspired by this nature, we propose a two-level hierarchical reinforcement learning (RL) framework for conversational agents, bridging the gap between previous token-level or utterance-level RL methods. Developed on a two-level MDP, the token-level response decoding is conditioned on the utterance-level action, the explicit textual strategies. Based on theoretical derivation and efficiency consideration, we use DQN to solve the high-level critic and PPO to solve the low-level actor-critic. To further alleviate the reward sparsity and facilitate the convergence, we also design the dual-granularity reward mechanism, in which the utterance-level satisfaction score is integrated with token-level intrinsic motivation and K-L penalty. Experiments on both daily and emotional support conversations show that our method outperforms versatile baselines in strategy determination and response quality. Our implementation is available at https://github.com/AaronJi/ToSCA.