PACE: Towards Surfacing Hidden Conflicts in User Requests

AI summary

Overview

Research area: Natural Language Processing; specifically personalized LLM assistants, retrieval-augmented generation (RAG), multi-hop knowledge graph retrieval, and context-aware safety/conflict reasoning over egocentric knowledge bases.

Technical level: Intermediate. The paper assumes familiarity with retrieval metrics (Recall@K, MRR), dense vs. sparse retrieval, embedding models, and LLM agent pipelines, but its central idea is stated in accessible terms.

Scope: The paper introduces PACE, a benchmark for deciding whether an apparently ordinary user request conflicts with hidden facts in a personal knowledge base, and PaceMaker, a training-free multi-agent retrieval framework for surfacing that evidence.

What This Paper Is About

Personalized assistants are usually evaluated on whether they correctly execute a request, not on whether the request is actually appropriate given the user's circumstances. PACE addresses this gap by pairing user requests grounded in synthesized personas with egocentric knowledge-base (KB) facts, so that a model must retrieve and combine distributed evidence to decide whether the request conflicts with the user's situation. The accompanying PaceMaker framework is designed to retrieve that decisive evidence when it is not lexically or semantically obvious from the query itself.

Key Contributions

1. The PACE dataset. A retrieval-grounded benchmark (Personalized Assistants for Conflict Evaluation) in which each request looks normal in isolation, and its conflict status emerges only from hidden situational facts spread across a user-centric KB containing alters (family, colleagues, friends) and surrounding conditions. Reported statistics: 376,448 facts, 3,249 queries, 185 profile instances, an average of 2,035 facts per instance, 18 queries per instance, and 4.01 gold facts per query.

2. A feasibility-status and situation-type taxonomy. Every request is labeled Conflict or Non-conflict, and further categorized as Temporal (schedules, travel time, routines), Personal (health conditions, values, accessibility needs of the ego or an alter), or State (external conditions such as road conditions, posted restrictions, or facility operating issues).

3. The PaceMaker framework. A training-free multi-agent pipeline combining conflict-aware query planning, hybrid dense/sparse retrieval with Weighted Reciprocal Rank Fusion, multi-hop k-NN graph traversal, and conflict-aware evidence filtering.

4. An empirical demonstration that the task remains hard. Experiments across open-source and closed-source model configurations show that surfacing hidden situational constraints is a substantial open challenge, with conflict queries consistently harder than non-conflict ones.

Main Findings

• Oracle sets a clear ceiling, and full context underperforms it. Providing the model with only gold documents yields Pass rates of 86.89% (Qwen configuration), 87.13% (GPT configuration), and 91.26% (Gemini configuration). Passing the entire KB instead yields only 57.49% in the open-source setting and about 73% in the two closed-source settings (73.10% GPT, 73.07% Gemini), showing that unfiltered context impairs conflict reasoning.

• PaceMaker beats retrieval baselines in all three configurations. Pass rates: 68.82% (Qwen3-Embedding-8B / Qwen3-4B-Instruct-2507) versus 62.73% for Sparse and 62.39% for Dense; 75.35% (text-embedding-3-small / GPT-5.4-mini) versus 65.53% Sparse and 62.63% Dense; 77.44% (gemini-embedding-2 / Gemini 3.1 Flash-Lite) versus 67.44% Sparse and 63.59% Dense.

• Retrieval metrics also improve. PaceMaker's Recall@5 / Recall@10 / MRR are 26.23 / 29.51 / 50.14 (Qwen configuration), 36.05 / 44.34 / 58.23 (GPT configuration), and 35.37 / 42.29 / 57.48 (Gemini configuration), against Sparse MRR of 37.01, 37.01, and 37.82 respectively.

• Conflict queries are the hard case. In the Qwen configuration, PaceMaker reaches 53.20% Pass on Conflict queries versus 41.80% for Sparse, 37.78% for Dense, and 30.77% for Full KB; Oracle reaches 79.46%. In the GPT configuration, PaceMaker reaches 59.17% Conflict Pass versus 55.70% for Full KB and 33.88% for Dense; Oracle reaches 76.54%. In the Gemini configuration, PaceMaker reaches 75.93% versus 71.91% Full KB and 49.85% Dense; Oracle reaches 92.63%.

• Gains over the strongest non-oracle baseline on Conflict queries are 11.40, 3.47, and 4.02 percentage points in the Qwen, GPT, and Gemini configurations respectively, while Non-conflict performance stays comparable.

• The conflict/non-conflict gap persists even under Oracle in the Qwen and GPT configurations, implying conflict resolution is difficult even when the relevant facts are directly provided.

• Full evidence coverage helps. Pass rates on Conflict queries rise when all gold documents are retrieved (Gold@10=1) compared to partial coverage (Hit@10=1 but Gold@10=0), indicating that atomically distributed, non-semantically-salient evidence must be gathered completely for reliable judgment.

• Every agent component matters. In the GPT-configuration ablation, removing planning drops overall Pass to 73.31% and Conflict Pass to 54.17%; removing traversal drops them to 71.65% and 52.41%; removing selection drops them to 75.19% and 58.87%; full PaceMaker reaches 75.35% and 59.17%.

• PaceMaker outperforms structured retrieval baselines, particularly on Conflict queries. Using Qwen3-4B-Instruct-2507 and NV-Embed-v2: PaceMaker 68.67% Pass / 54.42% Conflict / 26.78 Recall@5; HippoRAG 2 65.13% / 43.57% / 20.15; GraphRAG 58.26% / 34.98% with Recall@5 not reported.

• Human validation supports the dataset and judge. Feasibility-status agreement with humans is 93.3% overall (88.3% Conflict, 98.3% Non-conflict), with 86.7% inter-annotator agreement. For response quality, human agreement with the automatic judge is 93.5% overall (97.0% Pass, 90.5% Wrong, 93.0% Fail), based on 600 sampled instances with three annotators each.

Methodology in Plain English

Building PACE. The authors start from persona seeds taken from MSC and Synthetic-Person-Chat, and use GPT-5.4-mini to expand them into richer narrative descriptions of routines, environments, and behavioral tendencies. Expanded narratives are randomly paired to form ego-alter relationships, from which structured ego and alter profiles are generated, including occupation, health conditions, values, and everyday places and devices. From these profiles, the same model generates user requests (reservations, activity planning, recommendations), a gold context holding the situational background, and a reference judgment explaining the correct decision. Distractor contexts that are topically related but do not reveal the decisive evidence are also generated, consistent with the persona and timeline. Gold and distractor contexts are atomized into single-fact KB entries, mirroring how personal KBs store scattered discrete facts. Everything is anchored to a bounded timeline around a reference date, so pre-date facts are observed or confirmed and post-date facts are included only if already scheduled or announced. Quality is checked by manual author review plus automated filtering for taxonomy compliance, constraint satisfaction, core-cause diversity, and whether the gold facts are sufficient and unambiguous.

Building PaceMaker. At indexing time, the framework encodes all KB documents and builds a k-nearest-neighbor document graph over cosine similarity with 10 neighbors per document; no LLM calls are used during indexing. At inference, a conflict planner agent proposes up to three decision-relevant probing cues, and a multi-view query generator produces the original query view plus up to three counter views targeting potentially conflicting conditions, all conditioned on the reference date. Each view goes to both a dense retriever and BM25 sparse retrieval, with the top 10 documents per view merged by Weighted Reciprocal Rank Fusion (fusion constant 60, counter-view weight 1.2 versus 1.0 for the original view). A pre-hop filter agent selects the top 10 of the top 20 fused candidates as seeds; breadth-first traversal then expands 3 neighbors per document up to a depth of 5 hops. A post-hop filter agent narrows the entire pool to 10 final documents, which an answer generator uses to state whether the request can be fulfilled and why. FAISS handles dense indexing and retrieval, and open-source agent models are served via vLLM on NVIDIA RTX A6000 4EA GPUs.

Evaluation. Retrieval quality is measured with Recall@5, Recall@10, Hit@5, Hit@10, Gold@5, Gold@10, and MRR. Response quality is judged by GPT-5.4-mini against the gold rationale on a three-level Pass / Wrong / Fail scheme, with Pass as the primary metric. Baselines are Oracle (gold documents only, no retrieval), Full KB (entire KB, no retrieval), Sparse (BM25), and Dense (vector search). Appendix D.4 additionally reports an evaluation using Gemini 3.1 Flash-Lite as an independent judge to check for self-preference bias.

Why This Matters

Research impact. Most personalized-assistant and RAG work optimizes for retrieving supportive or relevant information; PACE reframes retrieval as diagnostic evidence selection, where the goal is the fact that determines whether a request should be refused. The results show that retrieving topically relevant documents, including with structured multi-hop methods like GraphRAG and HippoRAG 2, is not enough: completeness of the conflict-inducing evidence is what drives judgment quality. This gives the community a benchmark and a baseline for implicit-constraint reasoning.

Real-world applications:

• Personal scheduling and booking assistants that must notice a hidden conflict, such as a prior commitment or travel time that makes a reservation infeasible, instead of silently confirming it.
• Dietary, health, and accessibility-aware recommendations, for example declining a restaurant when a companion has a shellfish allergy.
• Venue and facility recommendations that account for known external conditions, such as a cafe hosting live music when the user asked for a quiet place to work.
• Personal memory and knowledge-base systems that must decide what to surface from thousands of scattered atomic facts without over-refusing legitimate requests.

Industry relevance. The framework is training-free and model-agnostic, so it can be layered onto existing retrieval stacks without fine-tuning. The paper notes that PaceMaker uses no LLM calls during indexing and far fewer calls in total than HippoRAG 2, giving a lower cold-start cost that is advantageous for personalized KBs initialized or refreshed frequently. At the same time, the persistent gap between Full KB and Oracle Pass rates indicates that simply scaling context is not a substitute for evidence selection.

Future Directions

1. Extending beyond feasibility judgment to full task execution. The authors state the benchmark does not evaluate whether an assistant can complete downstream tasks such as recommendation, scheduling, or planning, and suggest richer persona worlds, actionable entities, and dynamic task environments to assess end-to-end personalized task solving.

2. Training specialized agents. The current method is evaluated without task-specific training; the authors propose training agents for query reformulation, evidence selection, graph traversal, and decision calibration.

3. Improving evidence completeness. Because full coverage of gold documents yields clear additional gains on Conflict queries, methods that reliably surface the entire set of atomically distributed conflict facts remain an open problem.

4. Closing the non-oracle gap on Conflict queries. Conflict pass rates remain far below Oracle across all configurations, and even Oracle leaves room in the Qwen and GPT settings, raising the question of how to calibrate refusal so that assistants neither blindly execute nor over-refuse.

Target Audience

Researchers and engineers working on personalized LLM assistants, retrieval-augmented generation, agentic retrieval, and user-centric memory or knowledge-graph systems. It is also relevant to practitioners building safety- or appropriateness-aware assistants who need a benchmark for context-dependent refusal, and to dataset and evaluation researchers interested in how conflict labels and judge-based response evaluation are validated with human annotation. Readers should be comfortable with retrieval metrics and multi-agent LLM pipelines to get the most out of the experiments.

Authors’ abstract

Personalized assistants should not only comply with user requests but also assess whether those requests are appropriate given the user's current circumstances. However, prior work has primarily focused on accurately executing requests, overlooking the need for assistants to account for context and engage in conflict-based refusal. Furthermore, while existing work on conflict or safety detection relies on explicitly provided factors, real-world scenarios often involve implicit factors that must be retrieved from a knowledge base (KB). To this end, we introduce Personalized Assistants for Conflict Evaluation (PACE), a dataset for evaluating whether models can identify latent constraints, expressed as egocentric knowledge or events, that render seemingly reasonable user requests inappropriate. PACE pairs user requests grounded in well-defined personas with egocentric KB facts, requiring models to integrate contextual evidence to determine whether a request is conflicting. This implicit retrieval setting hinders the direct association between user requests and conflict-inducing knowledge, making it difficult for existing models to identify relevant user-specific facts. To address this challenge, we further propose PaceMaker, a multi-agent framework in which specialized agents coordinate across query reformulation, multi-hop graph traversal, and conflict-aware filtering to retrieve contextually decisive evidence. Experiments on PACE evaluate both evidence retrieval quality and conflict decision accuracy, showing that PaceMaker consistently outperforms existing approaches.