Implicit Beliefs as Scalable Process Rewards for Strategic Persuasion

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents in multi-turn strategic dialogue, specifically persuasion, combined with psychologically grounded user simulation.

Technical level: Intermediate. The paper assumes familiarity with policy-gradient RL for LLMs (GRPO-style group-relative advantages, KL-regularized token-level objectives) and with prompting-based agent simulation, but the core idea — reading a frozen model's token probabilities to build a reward — is explained clearly enough for readers with basic LLM background.

Scope: The paper introduces PersuArena, a theory-grounded persuasion simulator with 120 scenarios across five domains, and Probe-RPO, an online RL framework that converts a frozen persuadee's internal belief probes into dense per-turn process rewards, evaluated against prompted LLMs and reward ablations.

What This Paper Is About

Training an LLM to be a good persuader over multiple conversation turns is hard for two reasons: simulated persuadees usually behave unrealistically (because their resistance is left to the model's own priors rather than any psychological structure), and the only reliable training signal is often the final accept/reject outcome, which does not reveal which conversational move actually worked. The authors argue these two problems share one solution: if the simulated persuadee is built with explicit psychological mechanisms, then its internal state can be read directly from its own token probabilities and used as a dense, cheap reward signal at every turn. The paper builds both the simulator and the training method that exploits it.

Key Contributions

1. PersuArena, a theory-grounded persuasion simulation environment. It operationalizes established psychological constructs (resistance types, persuasion dynamics, trust components) into executable prompt constraints and a deterministic episode state machine, yielding non-monotonic persuasion trajectories across 120 scenarios in five domains: healthcare, financial advising, workplace management, education, and personal relationships.

2. Probe-RPO, an online RL framework that turns implicit beliefs into process rewards. Instead of adding an LLM judge per turn, it reads K = 3 binary introspective probes from the frozen persuadee's logits in a discarded fork of the context, and rewards the per-turn change in the resulting belief state.

3. A composed advantage estimator that combines a standard GRPO group-relative outcome advantage with a dense return-to-go over belief changes, with a single hyperparameter λ that recovers outcome-only GRPO exactly at λ = 0.

4. Empirical demonstration that belief-elicited process rewards outperform both prompted LLMs and outcome-only or judge-based reward ablations, transfer to persuadee models unseen during training, and produce an emergent "diagnose-then-act" conversational structure.

Main Findings

• Large gain in full acceptance: Probe-RPO reaches 52.5% full acceptance on the 120 held-out scenarios, versus 35.8% for the strongest GRPO ablation (GRPO with Goal+Strategy reward) and 25.8% for the best off-the-shelf prompted LLM (Claude Haiku 4.5). The paper frames this as a 16.7 percentage point improvement over the strongest outcome-only variant.

• Combined acceptance: Counting full and conditional acceptance together, Probe-RPO reaches 73.3%, versus 61.6% for Claude Haiku 4.5 and 56.6% for GRPO (Goal+Strategy).

• Indecision drops sharply: Probe-RPO's pending rate is 17.5%, less than half of the 39.2% recorded for GRPO (Goal+Strategy). Rejection rates are similar across systems (reported as 4.2–10.8%); Probe-RPO's own rejection rate is 9.2%, higher than several comparisons. The authors therefore characterize the main effect as converting unresolved conversations into decisions, not as uniformly reducing negative outcomes.

• Both reward levels are necessary: GRPO (Goal) reaches 24.2% acceptance with 49.2% pending. GRPO (Goal+Strategy) raises acceptance to 35.8%. Training with process reward only (no terminal anchor) yields 37.5% conditional acceptance but only 19.2% full acceptance — receptiveness rises but commitment does not follow. Combining both gives the highest acceptance and lowest indecision.

• Intermediate process weight is best: Three runs differing only in λ showed an inverted-U. At step 60, held-out acceptance was 21.7% / 52.5% / 25.0% for λ = 0 / 0.5 / 1; each run's best checkpoint preserved the shape at 35.8% / 52.5% / 30.0%. Too little weight starves the gradient (GRPO groups with identical outcomes across all G rollouts contribute nothing); too much lets a bounded but accept-blind signal dominate — the shaping term barely separates acceptance from conditional acceptance (AUC 0.62), and the best λ = 1 checkpoint (step 50) moves the persuadee as far as the released model (mean final belief +0.67) yet closes far less often (30.0% vs 52.5%).

• Dense signal correlates with policy quality: The dense term's share of the advantage magnitude is reported as the strongest correlate of policy quality measured, with Pearson r = 0.786 against training accept rate.

• Transfer to unseen persuadees: Trained only against Qwen3-14B, Probe-RPO achieves 52% acceptance with Qwen3-14B, 75% with GPT-5.6 Luna, and 45% with Claude Haiku 4.5 on a fixed 60-scenario held-out subset, exceeding all prompted persuader baselines shown in every column. The spread from 45% to 75% shows transfer is substantial but not invariant.

• Emergent diagnose-then-act structure: Probe-RPO opens with diagnostic questions in roughly 80% of first-turn utterances and then sharply reduces diagnosis, with plan details and commitment asks peaking in the middle turns. The untrained Qwen3-4B initialization also reduces questions over time but relies on generic autonomy cues and produces substantially fewer concrete plans. Prompted Claude Haiku 4.5 shows a similarly staged pattern.

• Probe trajectories decompose failure modes: Under Probe-RPO all three probes climb from near −1 at the start to roughly +0.7 by mid-conversation, with CA (concern-address) rising first and PE (persuasion-effectiveness) and AC (acceptance-shift) following. Under the untrained policy, CA plateaus near +0.3 while PE and AC remain negative throughout — the persuadee feels heard but is not moved. Trajectories ending in acceptance and conditional acceptance separate from the rest within roughly two to three turns; rejections stay negative; pending trajectories rise slightly then flatten near zero, an internal signature of stalling.

• Trajectories are genuinely non-monotonic: At least 15% of turns carry a negative process reward for every persuader evaluated, rising to 36.1% for the untrained policy.

Methodology in Plain English

Step 1 — build a persuadee with rules, not vibes. Each persuadee is an LLM with a hidden psychological profile the persuader never sees: one of four resistance types (Entrenched Belief, Identity Threat, Autonomy Resistance, Ambivalent Avoidance), a motivational orientation, a trust baseline split into ability, benevolence, and integrity, and theory-of-mind parameters. Profile-specific instructions in the prompt induce three dynamics: reactance escalation (pressure and repetition increase resistance, and excessive pressure can cause stage regression), inoculation saturation (repeating a strategy more than twice without progress makes it permanently ineffective), and trust ratcheting (negative trust events outweigh positive ones). A deterministic parser and state machine end episodes in one of five states: ACCEPT, CONDITIONAL ACCEPT, REJECT, PENDING, or CONCLUDE–PENDING, using a strict priority-ordered decision procedure.

Step 2 — read the persuadee's mind cheaply. Each persuadee turn uses two passes over the same frozen model. Pass 1 produces the visible reply and is the only pass that changes the dialogue. Pass 2 forks the context, appends each probe question, decodes exactly one token, and reads the top-n log-probabilities; the fork is discarded. The three probes are fixed in advance and never tuned — CA ("Has the persuader shown genuine understanding of what actually worries me about this?"), PE ("Is the persuader persuasive, e.g., using effective and suitable persuasion strategies?"), and AC ("Am I finding myself more open to what is being proposed than I was before?"). All three are positively valenced, so "Yes" always favors the persuader.

Step 3 — turn readings into rewards. For each probe the belief score is the probability mass on the Yes token family minus that on the No token family, bounded in [−1, +1]; the aggregate belief is the mean over the three probes. The per-turn dense reward is the change in that belief (divided by 2 to keep the range in [−1, +1]) rather than its level, so the persuader is not paid for receptiveness it did not create. This telescoping sum resembles potential-based shaping and is bounded per episode. A terminal anchor — a mechanical goal score of +3, +1, −1, or −3 for ACCEPT, CONDITIONAL_ACCEPT, PENDING, and REJECT, plus a frozen judge's 1–5 strategy score mapped to [−2, +2] — prevents the policy from merely making the persuadee feel good forever.

Step 4 — optimize. The composed advantage adds a group-relative outcome advantage (normalized by the group standard deviation, as in GRPO) to a dense return-to-go over belief changes (not normalized), with λ the single dial between them. Training uses Qwen3-4B-Instruct as the persuader, separate frozen Qwen3-14B instances as persuadee and judge, 1,800 generated training instances, 2 epochs at learning rate 1×10⁻⁵, group size G = 4 with a batch of 16 scenarios (64 trajectories), λ = 0.5, KL coefficient β = 0.05, entropy bonus 0.01, and T_max = 6 during training (10 at evaluation). Persuadee replies and probe tokens are masked and never receive gradient. Decision tokens from the persuader are stripped before the persuadee sees them, so the persuader cannot forge an outcome.

Step 5 — evaluate. The 120 held-out scenarios (disjoint identifiers from the 1,468 training and 200 validation scenarios) are compared against five prompted LLMs (Qwen3-4B, Qwen3-14B, Llama-3.1-8B, GPT-5.6 Luna, Claude Haiku 4.5) and three ablations: GRPO (Goal), GRPO (Goal+Strategy), and Process Reward Only. The authors note the held-out set is deliberately harder and differently distributed — 95.0% high-stakes versus 48.4% in training, 75.0% colleagues/acquaintances versus 44.3%, mean persuadee age 52.1 versus 36.8.

Why This Matters

Research impact. The paper offers a different route to dense supervision for dialogue than the dominant one. Process reward models and per-turn LLM judges add models and cost to the training loop; Probe-RPO replaces a per-turn judge with a few single-token read-outs from a model already in memory. It also makes a conceptual point: a simulator built for psychological realism can double as the reward source, unifying two problems usually treated separately. The result that a 4B policy learns a staged diagnose-then-act strategy purely from RL, without being told about conversational phasing, is a notable secondary finding.

Real-world applications (domains the paper names):

• Clinical counseling and public-health intervention, where shifting beliefs and intentions is the point.
• Negotiation and crisis de-escalation, where pressure backfires and sequence matters.
• Financial advising conversations, where trust components (ability, benevolence, integrity) are explicitly modeled.
• Workplace management, education, and personal-relationship conversations, the remaining PersuArena domains.

Industry relevance. Any deployed conversational agent that must move a user toward a decision — onboarding, retention, health coaching, sales, mediation — faces the same credit-assignment problem: outcome-only signals are sparse and uninformative. A reward that costs one forward pass per turn and requires no judge model is operationally attractive at scale.

Future Directions

• Human interlocutors and human evaluation. All persuadees here are LLMs, and the authors explicitly have not tested whether humans perceive Probe-RPO-trained agents as more competent. Cross-model transfer was measured (45%–75% acceptance), but cross-species transfer is open.

• Probe validity and probe-set design. The reward is only as faithful as the persuadee's calibration and only as broad as the questions asked; the three probe levels are strongly correlated, and only their turn-level changes carry partly independent information. Adding a negatively valenced probe such as felt pressure, whose signed form the appendix sketches, could penalize manipulative pressure directly.

• Ethical guardrails. The authors pair the capability with a proposed guardrail model conditioned on an explicit anti-manipulation policy to screen each persuader turn, and PersuArena includes manipulation-detection metrics, but integrating ethical constraints into Probe-RPO itself is left as future work. Misinformation and youth online safety are named as dual-use concerns.

• Better reward parameterizations and λ selection. Moving from log-odds to probability space discards magnitude information; hybrid parameterizations remain future work. Separately, λ cannot be selected on training reward, and the audit of outcome–strategy decoupling can only detect failures within the scope of the judge's questions.

Target Audience

Researchers and practitioners working on RL post-training for multi-turn dialogue agents, especially those building social or strategic conversational systems; designers of LLM-based user simulators for evaluation and training; and applied teams who need dense supervision for long-horizon interaction without paying for a per-turn judge model. Readers interested in the psychology-of-persuasion side will find the taxonomy in PersuArena useful even if they skip the optimization details.

Authors’ abstract

Training large language models (LLMs) for multi-turn strategic persuasion faces two coupled obstacles: (i) realistic simulation of psychologically grounded persuadees, and (ii) dense, faithful supervision for credit assignment in strategic dialogues. Outcome-only rewards are sparse and uninformative about which conversational moves drove success, while LLM-judge process rewards are expensive and decoupled from the underlying interaction dynamics. We address both challenges with a unified framework. First, we introduce PersuArena, a persuasion simulation environment that produces realistic, non-monotonic trajectories. Second, building directly on this environment, we propose Process Rewards via Online Belief Elicitation for Relative Policy Optimization (Probe-RPO), an online RL framework that converts the persuadee's internal state into dense process rewards. At each turn, three introspective probes are read from the frozen persuadee's logits, yielding a multi-dimensional reward signal that is faithfully aligned with persuasion dynamics. With extensive experiments on PersuArena spanning healthcare, financial advising, workplace management, education, and personal relationships, we demonstrate that our Probe-RPO achieves the full acceptance rate of 52.5%, compared with the 35.8% for the strongest GRPO ablation and 25.8% for the best off-the-shelf LLM. We also conduct comprehensive analyses on the generalization across persuadee models and the persuasion dynamics across conversations. Our work offers a promising paradigm that can provide effective process supervision for strategic dialogue.