ProcessPrefBench: Evaluating Cross-Task Transfer of Individual Process Preferences from Naturalistic Human-AI Interaction

AI summary

Overview

Research area: Natural Language Processing — evaluation of long-term memory and personalization in human–AI dialogue, specifically whether user-specific process standards transfer across tasks.

Technical level: Intermediate. The paper is readable without deep technical background, but it assumes familiarity with LLM benchmarking, rubric-based LLM-as-judge evaluation, and prompting conditions.

Scope: A 66-case, two-user benchmark built from three months of natural dialogue that measures whether a preference expressed in one conversation is actually realized — not merely mentioned — in a later, different task that does not restate it, while tracking task failure and fabricated evidence separately.

What This Paper Is About

When users correct an AI assistant, request a different angle, or describe how they like to think, they reveal reusable standards about how work should be done — what counts as evidence, how deep an explanation should go, what to focus on, and how to deliver results. The paper asks whether such a standard, learned from one conversation, can shape an assistant's answer to a later, unrelated task when the user never repeats it. The authors build ProcessPrefBench to measure precisely that, and to check whether improving preference-following comes at the cost of factual reliability.

Key Contributions

1. A source-to-target construction protocol. Each of the 66 cases links a preference-bearing source dialogue to a different later task from the same user, with source evidence and target applicability judged separately — so topical similarity alone cannot create a case.

2. Target-specific 0–2 rubrics that separate surface mention from functional integration. Each case contains one to three source-anchored "preference atoms" with criteria for not applicable, 0 (absent or contradicted), 1 (local or cosmetic), and 2 (the preference changes the answer's reasoning, organization, conclusion, or delivery).

3. An evaluation that decouples preference realization from severe task failure. A separate task gate flags non-answers, major factual errors, fabricated evidence, hard-constraint violations, and false claims of external execution, so a personalized-but-unusable answer cannot count as success, and missing personalization alone cannot count as task failure.

4. A controlled comparison of memory interfaces. Three models (DeepSeek V4 Flash, Qwen-Max, GPT-4o) are tested under target-only, full-history, imperative distilled-preference, conditional distilled-preference, and the mem0 external-memory system, across 1980 final answers.

Main Findings

• Relevant history improves preference realization. Imperative distilled-preference prompting beats target-only prompting by 0.244 points on the 0–2 scale (95% CI [0.177, 0.313]; 48/8/9 wins/ties/losses; exact sign test p < .0001), with positive effects for DeepSeek V4 Flash (+0.427), Qwen-Max (+0.114), and GPT-4o (+0.192). Full-history prompting gains 0.204 points (95% CI [0.143, 0.269]; 44/12/9; p < .0001).

• Distillation offers no reliable advantage over raw history. The imperative-versus-full-history difference is +0.026 (95% CI [−0.034, +0.087]; p = .451), with model-level effects of +0.080 for DeepSeek V4 Flash, −0.058 for Qwen-Max, and +0.071 for GPT-4o. The authors explicitly note this does not establish equivalence.

• Fabricated evidence roughly doubles under memory. Failures rise from 5.8% under target-only prompting to 10.1% under full-history (difference 4.3 points; 95% CI [+1.0, +7.8]; p = .010) and 10.4% under imperative prompting (difference 4.5 points; 95% CI [+2.0, +7.6]; p = .006). The two memory conditions differ by only 0.3 points (p = 1.0).

• Overall task failure rises but not conventionally significantly. Imperative prompting moves the overall task-failure rate from 15.2% to 18.9% relative to target-only (exact McNemar p = .086).

• Fabrication increases are uneven across models. The increase is largest for Qwen-Max under imperative prompting (+9.1 points, p = .004), while GPT-4o shows no same-direction increase. The mem0 comparison shows an 8.3-point increase in fabrication (p = .019).

• Per-model preference scores vary widely. On the 0–2 scale, DeepSeek V4 Flash goes from 0.79 (target-only) to 1.15 (full-history) to 1.23 (imperative); Qwen-Max from 0.62 to 0.77 to 0.73; GPT-4o from 0.47 to 0.59 to 0.66. Task-failure rates for the same conditions are 8.3%/9.1%/14.4%, 13.6%/23.5%/22.7%, and 23.5%/14.4%/19.7% respectively.

• mem0 shows no consistent gain. Its cell means exceed target-only prompting for DeepSeek V4 Flash (1.06 vs 0.79) and GPT-4o (0.49 vs 0.47) but not for Qwen-Max (0.59 vs 0.62).

• The pipeline can remember the wrong thing. In the audited Figure 1 case, all three tested models compressed the source interaction into an adjacent data-credibility standard rather than the user's actual point about how and by whom a product is used — so the imperative condition faithfully executed the wrong standard.

• Transfer can succeed cleanly. In the second audited case, the standard changed a flat two-column pros-and-cons list into named structural dimensions with flip conditions; the case rises from 0.17 under target-only prompting to 1.33 under imperative prompting (the largest paired gain in the benchmark) with no fabricated-evidence flag.

• Robustness holds for the main comparison. The imperative-versus-target-only effect stays positive for user1 (n = 51, +0.194, p < .0001) and user2 (n = 14, +0.429, p = .0034), after excluding temporally inverted cases (n = 62, +0.252, p < .0001), under source-cluster (n = 30, +0.268) and target-cluster (n = 58, +0.249) aggregation, and under normal termination only (n = 52, +0.274).

• No evaluated interface is both personalized and reliable. The authors conclude that no tested interface combines reliable preference transfer with preserved factual reliability, and that autonomous selection from long histories remains untested.

Methodology in Plain English

Building the cases. The authors collected three months of real assistant usage from two knowledge workers using Margin, a webpage reading, annotation, and AI-discussion product (responses came from locally installed Claude Code and Codex at their then-default models). Each user annotated their own interactions and kept only exchanges where their own words — a correction, a prescription, or a self-description — supported a reusable standard for future work. Factual repairs and one-off task requests were excluded.

Pairing sources to targets. Annotating a source does not automatically make any later task a valid target. The preference owner separately judged another same-user task and kept the pairing only when the target did not restate the preference but applying the standard should still improve the answer. This is why topical similarity alone cannot generate a case.

Scoring. Each case has one to three preference atoms with target-specific 0/1/2 criteria. A "not applicable" option exists for atoms whose trigger is absent from the target. Crucially, a 2 requires the answer's reasoning, organization, conclusion, or delivery to actually change — mentioning the preference is not enough.

Conditions. All conditions share the same system prompt, page title, highlighted context, verified supplied material, and verbatim user question. Target-only adds no memory; full-history adds the complete source dialogue; distilled conditions have the tested model distill each source once and reuse it (applied always under "imperative," only when relevant under "conditional"); mem0 extracts from the source and retrieves against the target. Every case is generated twice per model and condition, giving 1980 final answers.

Judging. Claude Sonnet 4.6 serves in isolated preference and task-gate roles, with a model family disjoint from the tested models. The preference judge sees the rubric but not the source dialogue, and casts three evidence-citing votes per atom (majority "not applicable" excludes the atom; otherwise the median applicable score). The task gate sees validity checks without the rubric and uses two votes plus a third on ties.

Judge validation. Two annotators rated 30 answers blind to model, condition, and judge scores. Post-calibration human–human agreement was κ = .72 on preference atoms and .84 on the task gate; agreement between humans and the judge was lower (.52 and .53 on preferences; .37 and .29 on task failure), and the judge scores preferences more conservatively (mean 0.83 vs 1.02 and 1.03 for the two annotators). The authors describe this audit as diagnostic rather than confirmatory.

Why This Matters

Impact on research. The paper reframes "memory" as a transfer problem with separable failure modes: preserving the wrong standard, failing to realize a correctly preserved one, or importing remembered content as unsupported claims. It also supplies a benchmark design (separate source-evidence and target-applicability judgments, functional-vs-cosmetic rubric levels) that existing long-term-memory and personalization benchmarks do not jointly test.

Real-world applications:

• Knowledge-work assistants that need to carry an analyst's standards for evidence quality and analytical framing across many unrelated reading sessions.
• Research and sensemaking tools, where a user's preferred structure (dimensions, flip conditions, cases) should persist into new topics.
• Enterprise agents with persistent memory, where the finding that memory raises fabricated evidence is a direct deployment risk that needs guarding.
• Evaluation and audit pipelines for memory systems, using the task gate concept to keep personalization gains from masking factual degradation.

Industry relevance. Vendors shipping long-term memory and personalization features now have a concrete measurement showing that more memory is not free: fabricated evidence roughly doubled (5.8% to 10.1–10.4%) in exchange for a gain of about 0.2 points on a 0–2 scale. The finding that distillation gave no reliable advantage over raw history (p = .451) also bears on cost and architecture decisions for memory pipelines.

Future Directions

• Close the fabrication gap. No tested interface combined reliable preference transfer with preserved factual reliability; whether any prompting or memory design can do both is open.

• Test retrieval, not just application. Every case supplies the correct source, so the benchmark tests cross-task application rather than selecting the right interaction from a heterogeneous history — which the authors call untested.

• Resolve the distillation question. The +0.026 difference does not establish equivalence, and the two end-to-end audits differ only in what the distillation kept — one preserved the user's actual standard, the other preserved an adjacent one. What distillation should retain is an open design question.

• Broaden beyond two users. The benchmark is idiographic: two knowledge workers, one product, a 52/14 case split, 30 corrective sources and 59 distinct targets, concentrated in research, sensemaking, and decision tasks. The authors state their intervals capture within-benchmark variation, not population uncertainty.

Target Audience

Researchers and practitioners working on LLM memory, personalization, and long-context dialogue evaluation; product teams building persistent-memory assistant features; and evaluation engineers who need rubric designs that distinguish surface compliance from genuine behavioral change. Readers interested in hallucination and grounding will find the fabricated-evidence analysis and its worked examples directly useful.

Authors’ abstract

In natural human-AI dialogue, users' corrections, requests, and self-descriptions carry reusable signals about how they want work to be done. These signals reveal individual process preferences, user-specific standards for evidence, depth, analytical focus, procedure, and delivery. We ask whether a preference revealed in one dialogue can shape an assistant's later answer when the task does not restate it. ProcessPrefBench uses three months of natural dialogues from two knowledge workers to build 66 source-target cases separately judged for source evidence and target applicability. Sources are preselected, so the benchmark tests application rather than retrieval. We test three models with target-only, full-history, and distilled-preference prompts. Target-specific 0-2 rubrics score preference realization, and a task gate flags severe failures. Compared with target-only prompting, full-history and imperative distilled-preference prompting raise preference realization by 0.204 and 0.244 points on the 0-2 scale, respectively, with no reliable advantage from distillation. Fabricated-evidence failures rise from 5.8% under target-only prompting to 10.1% under full-history and 10.4% under imperative prompting. ProcessPrefBench highlights a tension in learning from dialogue, where drawing on prior interactions can improve preference realization yet also invite fabricated evidence. We hope it helps build assistants that honor a user's process preferences without sacrificing factual reliability.