Training Proactive and Personalized LLM Agents

AI summary

Overview

Research area: LLM agents, multi-turn human–AI interaction, multi-objective reinforcement learning, tool-use agents.

Technical level: Advanced (assumes familiarity with RL, GRPO/PPO-style policy optimization, agent scaffolds, and benchmark design).

Scope: The paper introduces an interactive environment (UserVille) and a multi-objective RL framework (PPP) for training LLM agents that are productive, proactive, and personalized when working with ambiguous user requests.

What This Paper Is About

Most LLM agents today are trained only to maximize task completion on self-contained, precise prompts, which ignores how real users actually communicate—vaguely, with hidden preferences, and expecting the agent to ask questions when needed. The authors reframe agent training around three collaborative dimensions (Productivity, Proactivity, Personalization) and build an environment plus a reinforcement learning recipe that explicitly rewards good agent–user interaction, not just correct answers. They show that training agents on these dimensions produces better task outcomes and better user satisfaction than strong frontier-model baselines.

Key Contributions

1. UserVille, an interactive evaluation and training environment. It converts static benchmarks (SWE-Bench, BrowseComp-Plus) into interactive settings by (a) "vaguenizing" precise prompts into under-specified ones, (b) running preference-aware LLM user simulators with 20 configurable interaction personas, and (c) producing user-centric metrics for proactivity and personalization.

2. The PPP framework. A multi-objective RL method that optimizes a composite reward combining task success (R_Prod), interaction quality via user-effort estimation (R_Proact), and adherence to stated user preferences (R_Pers).

3. A user-effort taxonomy for proactivity. Each agent question is classified as low-, medium-, or high-effort based on whether the answer is derivable from the hidden precise prompt, requires the user to refuse, or requires the user to do external work.

4. Empirical validation across two domains plus a human study. The trained agent (PPP-36B, based on Seed-OSS-36B-Instruct) outperforms GPT-5 and other baselines on average across dimensions, generalizes to unseen preferences and simulators, and improves real-user satisfaction in a Prolific coding study.

Main Findings

• Interaction is essential under vague prompts. Without interaction, vague prompts drop SWE-Func-Loc F1 from 64.50 to 44.11. Enabling an ask_user tool alone does not help base models; RL training on interaction lifts performance back to 64.50.

• PPP improves all three dimensions. Across SWE-Bench (Func-Loc) and BrowseComp-Plus, PPP achieves a +16.72 average point gain over baselines including GPT-5. On SWE-Func-Loc it scores 56.26 Productivity, 75.55 Proactivity, and 89.26 Personalization.

• Frontier models are productive but not collaborative. GPT-5 scores 40.40 average, with strong productivity on SWE (55.83) but weak proactivity (36.60) and personalization (12.96 on BrowseComp). Personalization rankings do not track productivity rankings (GPT-4.1 beats GPT-5 on personalization in some settings).

• Ablations confirm each objective is necessary. Removing R_Proact drops proactivity from 75.55 to 37.75. Removing R_Pers drops personalization from 89.26 to 47.25. Training only on task success causes proactivity and personalization to degrade over training.

• Agents learn strategic, ambiguity-sensitive questioning. The PPP agent raises its ask ratio from 50% to 100% on vague SWE prompts (51% to 85% on Deep-Research) while keeping a low ask ratio on precise prompts. Question quality follows an increase-then-decrease pattern: the agent first asks more, then refines toward low-effort, high-value questions.

• Strong generalization. Performance varies only slightly across simulators (GPT-5, GPT-5-Mini, GPT-4.1, GPT-4o, GPT-5-Nano). Personalization improves on 8 held-out unseen preferences. Localization-trained models transfer to full SWE-Bench, improving success rate from 0.29 to roughly 0.36.

• Sim-to-real transfer holds in a human study. With 33 programmers and 300 evaluations, PPP-36B scored 3.75 overall versus GPT-5's 3.79 and Seed-36B's 3.45. Its ask rate was 98% versus 35% for baselines, and 79% of participants rated its question volume as "about right." 67.7% of PPP sessions followed user preferences.

Methodology in Plain English

The authors start from existing agent benchmarks (SWE-Bench, BrowseComp-Plus) where prompts are deliberately complete. They use an LLM to rewrite those prompts into short, vague versions—keeping at most one key detail—creating an information gap the agent must close by asking questions. The precise original prompt is hidden and used only by the user simulator to answer.

Each simulated user is parameterized by one of 20 interaction preferences (e.g., "never ask me questions," "ask only at the beginning," "respond in JSON format"). During interaction, the agent can call task tools (code search, file editing, web search) or the ask_user tool. Every question is graded for user effort, and every trajectory is graded for preference adherence, using either hard-coded rules or an LLM-as-a-judge with preference-specific rubrics.

Training uses GRPO—a group-relative RL algorithm—with DAPO's Clip-Higher and token-level policy gradient loss. The reward is the sum of three signals: a verifiable task reward, a proactivity reward (+0.05 bonus for all-low-effort questioning; −0.1 per medium-effort and −0.5 per high-effort question), and a personalization reward (+0.05 bonus for full compliance, penalties for violations). Training ran for 200 steps on a Seed-OSS-36B-Instruct base with a batch size of 64 and group size of 8.

For evaluation, productivity is F1 (function localization), exact match (deep research), or unit-test pass rate (full SWE). Proactivity is a binary score (1 if session-level effort is low, else 0). Personalization is a binary score averaged only over sessions where the agent asked at least one question.

Why This Matters

The paper argues that benchmark-driven agent research has over-optimized a narrow objective (task success) and under-optimized the skills that make agents usable in practice: surfacing uncertainty, asking cheap questions, and respecting how a user wants to be interacted with. It shows that these skills can be trained with simulated users and that the gains transfer to real humans—an increasingly important claim as agents move from demos into production coding and research assistants.

Real-world applications:

• Coding assistants on underspecified bug reports. Agents that ask targeted clarifying questions instead of guessing, reducing wasted edits and review effort.

• Deep-research and search agents. Agents that resolve scope ambiguity with the user before spending retrieval budget, and that respect constraints like "answer in one sentence."

• Customer support and IT automation. Systems that adapt question frequency and format to different user personas (experts vs. novices, verbose vs. terse).

• Enterprise copilots with strict interaction policies. Agents trained to honor hard rules such as never asking questions, or only asking at session start, which matters for compliance and workflow fit.

Industry relevance: The ablations show that pure task-reward optimization actively degrades collaboration quality over training, which is a direct warning for teams fine-tuning agents on benchmark success. The finding that a 36B open model can match GPT-5 on user satisfaction—while losing on raw resolution rate—suggests that interaction skills are a cheaper, higher-leverage axis than scaling alone.

Future Directions

• Closing the sim-to-real effort gap. The user study notes participants perceived more effort from PPP-36B's questions than the simulator predicted, suggesting simulator effort estimates need calibration against real humans.

• Scaling beyond function localization. The authors acknowledge PPP-36B lags GPT-5 on full SWE resolution (71.0% vs. 85%), partly because training focused on localization with a smaller base model. Training directly on full tasks is an open question.

• Incorporating real human feedback in the loop. The paper's ethics section explicitly proposes including real user feedback rather than only simulated users, which raises privacy, consent, and data-security design questions.

• Expanding and diversifying objectives and personas. The framework is described as domain-agnostic; natural extensions include additional interaction objectives (e.g., teaching, negotiation) and broader demographic coverage to test fairness across communication styles.

Target Audience

Researchers and engineers working on LLM agent training, RLHF/RLAIF, and human–AI interaction evaluation. It is also relevant to product teams building coding assistants, search agents, or enterprise copilots who need to decide what to optimize beyond task success, and to benchmark designers interested in user-centric rather than task-centric metrics. The paper is dense with formalism and assumes comfort with reinforcement learning and agent scaffolds.

Authors’ abstract

Despite rapid progress, current AI agents are primarily optimized for isolated task completion. We argue for a paradigm shift toward training agents as collaborators that communicate and adapt to people. To facilitate this shift in real-world complex applications, we first formalize three dimensions of collaborative AI agents: Productivity, Proactivity, and Personalization (PPP). We introduce UserVille, an interactive environment with configurable LLM-based user simulators and user-centric feedback to evaluate these dimensions, and propose a multi-objective reinforcement learning framework that optimizes them using rewards from task outcomes, question effort, and preference adherence. On two real-world agentic tasks (SWE-Bench and BrowseComp-Plus), PPP-trained agents outperform strong LLM baselines (including GPT-5) by an average of 16.7 points, ask more targeted questions, and generalize to unseen preferences and tasks. A follow-up user study further highlights the importance of user-centric feedback for training collaborative agents that are both more effective and easier to supervise.