PROPER Agents: Proactivity Driven Personalized Agents for Advancing Knowledge Gap Navigation

AI summary

Overview

Research area: Proactive conversational AI agents, personalized assistance, and user knowledge-gap modeling (cs.LG / human-centered NLP).

Technical level: Intermediate. The paper is formally written with set-theoretic definitions and an optimization objective, but the underlying idea — deciding which unspoken considerations an assistant should raise — is explained in accessible terms.

Scope (one sentence): The paper proposes ProPer, a modular agent framework that explicitly models what a user has not thought of (their "unknown unknowns") and decides selectively when to surface those considerations in a response.

What This Paper Is About

Most "proactive" AI assistants either interrupt users with clarifying questions or guess at what the user needs from context, which frequently produces irrelevant or badly timed help. The authors argue the missing piece is an explicit model of the user's knowledge gaps — task-relevant considerations that lie outside the user's awareness and cannot be inferred from context. ProPer addresses this by first generating candidate "dimensions" of a task, then deliberately calibrating which of them to bring into the final answer rather than dumping all of them on the user.

Key Contributions

1. Proactivity reframed as a calibration problem. Instead of maximizing how much an agent says, the paper formalizes proactivity as selectively intervening over a user's knowledge gaps, balancing personalization against disruption.
2. A dimension-based representation of tasks. Task-relevant considerations are represented as short free-form "dimensions," split into user-explicit, system-explicit (already covered by a baseline response), and implicit (latent gap) dimensions — enabling explicit modeling of what is missing.
3. The ProPer architecture. A modular pipeline that decouples gap discovery from response generation: a fine-tuned Dimension Generating Agent (DGA), a post-hoc calibrated reranker with a fixed intervention budget, and a Response Generating Agent (RGA) that edits a baseline response.
4. A gap-aware evaluation framework. A rubric scoring coverage of task-relevant gaps, appropriateness of initiative, and alignment with user intent, applied across medical, recommendation (shopping), and coding domains.

Main Findings

• Single-turn gains are large in open-ended domains. ProPer outperformed its base LLM on 84% of entries across all datasets, with mean quality scores rising from 2.19 to 3.86 (LlaMA-8B) and 2.93 to 4.03 (Qwen-8B) on the Medical dataset, and from 2.34 to 4.06 (LlaMA-8B) and 3.12 to 4.29 (Qwen-8B) on PWAB. Win rates on PWAB reached 93.17% and 87.50% respectively.
• Gains are smaller on tightly specified tasks. On Code-Contests, LlaMA-8B moved from 1.26 to 2.13 and Qwen-8B from 2.24 to 2.84, which the authors attribute to diminishing returns from proactive intervention when a task has well-defined correct solutions.
• ProPer beats GPT-4 in some domains. On Medical (3.73 vs. 3.28 against LlaMA-8B+ProPer) and PWAB (4.11 vs. 3.46), ProPer variants scored higher than GPT-4; on Code-Contests, GPT-4 scored 3.19 against LlaMA-8B+ProPer's 2.08.
• Chain-of-thought prompting is not enough. ProPer outperformed CoT-prompted LlaMA-8B and Qwen-8B even though CoT improved the base models, with the authors describing those CoT gains as marginal.
• The DGA matters most. Ablations show removing the Dimension Generating Agent causes a substantial drop across all datasets, while removing the post-hoc reranker causes a smaller but consistent degradation. DGA also produced higher-quality dimensions than LlaMA-8B or Qwen-8B, with Qwen-8B outperforming LlaMA-8B on that sub-task.
• Calibration parameters matter only in some domains. Sweeping (λ1, λ2) from (8.0, 1.0) to (0.0, 0.2) lowered quality in Medical (e.g., 4.00 to 3.70 for LLaMA+ProPer) and in PWAB, while Code-Contests stayed comparatively stable (2.11 to 2.08 for LLaMA+ProPer).
• Advantages persist across turns. In a small multi-turn evaluation of 12 sampled conversations per domain, ProPer was preferred in 11/12 Medical, 9/12 Code-Contests, and 12/12 PWAB conversations. Baseline wins occurred mainly in narrowly specified tasks where a conservative response suffices.
• Judge-based scoring. All comparisons used Gpt-5 as an external judge assigning a 0–5 quality score with brief justifications; statistically significant differences (p ≤ 0.01) are marked in the results table.

Methodology in Plain English

The pipeline runs in three stages on top of an ordinary instruction-tuned LLM.

First, a Dimension Generating Agent (DGA) is fine-tuned on interaction data annotated with dimension-level labels. Critically, it is trained only on observed dimensions — ones the user stated or the assistant addressed — and then at inference time is prompted to propose dimensions that appear to be missing from the current query and baseline response. It outputs candidate dimensions with log-probability confidence scores. (The extra training sets and any dataset split sizes are not reported in the content provided.)

Second, a post-hoc calibrated reranker builds an activation pool from (a) dimensions the user asked about that the baseline answer did not cover and (b) non-redundant implicit dimensions. Using BGE-small embeddings, it selects a fixed budget of k dimensions by maximizing a score that rewards high DGA confidence, rewards alignment with unmet explicit user needs, and penalizes similarity among the selected dimensions.

Third, the Response Generating Agent (RGA) edits the baseline response rather than rewriting it. Its prompt distinguishes explicit from implicit dimensions, instructs it to address explicit gaps by default and implicit ones only when the context warrants, favors concise additions over full rewrites, and allows at most one clarifying question when a gap needs user-specific information. Domain-specific prompt versions exist for coding assistance, clinical support, and recommendation settings.

Evaluation compares base models (LlaMA-3.1-8B-Instruct and Qwen-3.1-8B) against the same models wrapped in ProPer on three datasets: Medical (MD), Code-Contests, and PWAB (an online Amazon-shopping dataset). GPT-4 is included as a reference point, and GPT-5 acts as judge. Experiments ran on an NVIDIA H100 GPU.

Why This Matters

Impact on research. The paper shifts proactive-agent research away from "detect context and intervene" toward explicitly representing what the user does not know, and treats when to speak as a tunable calibration problem rather than an all-or-nothing behavior. It also provides reusable machinery: dimension-level supervision, an activation/reranking objective, and a gap-aware rubric for measuring proactive assistance rather than surface correctness.

Real-world applications:

• Patient-facing health information systems, where surfacing unasked considerations (severity framing, glycemic control, vaccination, precautions, comorbidities) can matter more than answering the literal question.
• Programming assistants, where proactivity should be restrained because tasks have well-defined optimal solutions and extra guidance adds little.
• E-commerce and recommendation agents, where preference trade-offs and comparison criteria are the typically unarticulated gaps.
• General conversational assistants and tutoring tools, where the goal becomes helping users discover the questions they should have asked.

Industry relevance. The modular design — gap generation, budgeting, and response editing as separate components — is directly implementable as a wrapper around existing production LLMs, and the budget parameter k gives product teams a single knob for controlling how forward the assistant is. Code is available at https://github.com/i-kiran/ProPer-Agent.

Future Directions

• Replace LLM-judge evaluation with human-centered validation measuring trust, perceived intrusiveness, and long-horizon utility, as the authors explicitly call for.
• Ground dimensions in structure. The current version uses free-form text dimensions for generality; the authors suggest concept bottlenecks, ontology-aligned slots, or hybrid symbolic-neural representations to improve interpretability and auditability.
• Learn calibration instead of sweeping it. Rather than fixed (λ1, λ2) regimes, turn them into a dynamic policy conditioned on uncertainty, conversation stage, and user feedback.
• Scale up multi-turn evaluation and persistent user models. The 12-conversation-per-domain study is described as a small-scale robustness check; the authors want longer trajectories, adaptive per-user proactivity thresholds, and multi-modal signals.
• Address epistemic and safety limits. The framework does not yet distinguish known unknowns from deeper gaps where the system itself may not know what is missing, and the authors note that their evaluation does not assess safety or appropriateness in real clinical workflows or regulatory requirements.

Target Audience

Researchers and practitioners working on proactive or personalized conversational agents, human-AI interaction, and LLM agent architecture will get the most from this paper. It is also useful for product teams evaluating whether proactive assistant features help or annoy users, and for applied ML engineers who want a concrete recipe for separating "what is missing" detection from "what to say" generation. Readers comfortable with basic set notation and an optimization objective will follow the formal sections; newcomers can read the introduction, the framework overview, and the results tables without the math.

Note: dataset sizes, DGA training hyperparameters, and appendix details beyond the truncated portion of the fine-tuning data pipeline are not reported in the content provided above.

Authors’ abstract

Current approaches to proactive assistance move beyond the ask-and-respond paradigm by anticipating user needs. In practice, they either burden users with clarifying questions or rely on context-based extrapolation, often leading to unnecessary or mistimed interventions. Such systems lack explicit mechanisms to model users' knowledge gaps, resulting in incomplete or suboptimal task outcomes. To address this, we propose PROPER, a framework that explicitly models user-specific knowledge gaps in a controlled manner. Central to our approach is the notion of dimensions: structured, task-relevant factors that define the considerations required for effective task completion. Given a user query, the DGA (Dimension Generating Agent) identifies explicit dimensions (from the user's query) and generates a set of candidate implicit dimensions capturing unarticulated aspects of the task. The RGA (Response Generating Agent) integrates both explicit and implicit dimensions selectively to produce personalized, context-aware, and proactively informative responses. We evaluate PROPER across multiple domains using a structured, gap-aware rubric that measures coverage, initiative appropriateness, and intent alignment. PROPER improves on quality scores and win rates across all domains, achieving up to 84% gains in single-turn evaluation and consistent dominance in multi-turn interactions. All code for PROPER is available at: https://github.com/i-kiran/ProPer-Agent.