Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization

AI summary

Overview

Research area: Artificial Intelligence — LLM agent personalization, test-time adaptation, and dueling bandits (preference-based online learning).

Technical level: Intermediate. The paper combines bandit theory with LLM prompt-level personalization; the core ideas are accessible, though the acquisition rule and the Bradley–Terry machinery assume some comfort with probabilistic modeling.

Scope: ID-TAP is an information-directed dueling-bandit framework that selects among a large pool of LLM persona prompts at test time using only binary preference feedback, and it is evaluated on seven personalized-generation benchmarks.

What This Paper Is About

LLM agents behave differently depending on the persona described in their prompt, but the best persona depends on the individual user and on the specific query — a "creative collaborator" helps with brainstorming while a "rigorous critic" helps with editing. The paper asks how an agent can discover the right persona for each incoming request from very light interaction, specifically from a small number of binary "which of these two responses do you prefer?" answers. The goal is a method that works when the persona pool is large (hundreds of candidates) and when user preferences must be inferred from noisy, implicit signals rather than explicit instructions.

Key Contributions

1. Scalable challenger acquisition. The authors operationalize an Information-Directed Sampling (IDS)-inspired acquisition criterion for large discrete persona pools, combining neural utility-gap estimates with a lightweight Bradley–Terry (BT) entropy-reduction proxy so that every candidate challenger can be scored cheaply.

2. A dual-system architecture for real-time efficiency. Global utility learning is decoupled from local uncertainty estimation: a query-conditioned neural utility model picks the "champion" persona and estimates gaps, while a lightweight BT maximum-likelihood explorer simulates hypothetical comparison outcomes. The neural model is updated once per observed comparison, while the BT solver handles the 2(K−1) hypothetical re-estimations per round.

3. Cross-query adaptation via similarity-weighted warm starts. Comparison evidence from earlier queries is transferred to a new query using normalized cosine-similarity weights, capped so that the total inherited count mass never exceeds one query-equivalent.

4. Empirical evaluation across seven benchmarks and two offline preference simulators. Under a shared ten-round comparison budget, ID-TAP achieves the strongest aggregate result among the compared methods and overtakes the strongest baseline after approximately 4 simulated-feedback rounds on the aggregate five-task trajectory.

Main Findings

• Best aggregate performance across seven benchmarks. Averaged over LaMP-4, LaMP-5, LongLaMP-2, LongLaMP-3, LongLaMP-4, PrefEval, and UltraChat, ID-TAP reaches a mean normalized ROUGE-L of 0.919 and a mean win rate of 83.6, versus 0.852 / 78.9 for POHF, 0.837 / 79.3 for PDO Sampling, and 0.729 / 76.1 for PersonaAgent.

• Numerically best in 13 of 14 dataset–metric cells. The single exception is LongLaMP-2 win rate, where PersonaAgent is higher (88.8 versus ID-TAP's 84.7).

• Same ordering under raw ROUGE-L. The unnormalized table gives ID-TAP first place on all seven datasets, with a cross-dataset average of 0.2228 versus 0.2068 for the strongest baseline.

• Faster feedback-round efficiency. On the aggregate curve averaged over the five LaMP and LongLaMP generation tasks, ID-TAP exceeds the compared baselines after roughly 4 simulated-feedback rounds.

• Challenger selection contributes modestly but consistently. In matched ablations, Full ID-TAP beats Random Challenger by 0.005 on LongLaMP-3, 0.020 on LongLaMP-4, and 0.011 on PrefEval.

• Both history mechanisms matter on sequential tasks. Along the progression NH → NBT → Full, neural history transfer adds 0.010 on both LongLaMP tasks, and the BT warm start adds a further 0.009 and 0.007; on PrefEval the result is unchanged across these variants.

• Random-collection controls are far worse. Random Response scores 0.601 / 0.497 / 0.448 and Random Acquisition + Neural Final scores 0.615 / 0.516 / 0.507 on LongLaMP-3 / LongLaMP-4 / PrefEval, well below Full ID-TAP's 0.783 / 0.664 / 0.629.

• Robust to denser persona pools. Keyword-conditioned persona construction lowers challenger exploration for POHF and PDO Sampling across four evaluated datasets, while ID-TAP's exploration rate changes by at most 1.5 percentage points.

• Low algorithmic overhead. With K=100 and 40 sampled users per dataset, ID-TAP's algorithmic update takes 0.065–0.571 seconds per round, against 0.95–3.15 seconds of end-to-end wall-clock latency — roughly 7–22% of total latency.

• Robust across preference and evaluation models. ID-TAP is numerically highest across the reported matched-model, cross-model, and BERTScore configurations.

Methodology in Plain English

The setup. Before any interaction, the system builds a fixed pool of personas from the user's history — in the main experiments, K = 400 candidate persona descriptions per user. These candidate strings never change during a session. At test time a query arrives, and the agent has a budget of T = 10 comparison rounds to figure out which persona suits this query.

Each round has two moves. First, the agent picks a champion: the persona its neural network currently scores highest for this query. Then it picks a challenger — the persona that offers the best trade-off between "probably nearly as good as the champion" and "likely to teach us the most about the ranking."

How the challenger is chosen. The rule minimizes a ratio of two quantities: the squared estimated utility gap between champion and challenger (numerator), divided by how much comparing them is expected to reduce uncertainty about the persona ranking (denominator). This is the information-directed idea: don't waste a comparison on a persona that is obviously bad, and don't waste it on one that tells you nothing new. The uncertainty term comes from a cheap Bradley–Terry model: the authors simulate both possible outcomes of the duel, re-fit the model each time, and measure how much the entropy of the normalized strength distribution drops.

Why two separate models. Re-fitting a big neural network for every hypothetical outcome would be far too slow. So the neural model supplies the utility gaps and gets one update per real comparison, while a small, fast Bradley–Terry solver handles all the "what if" simulation. The paper notes that these two systems are actually trained on the same objective (pairwise binary cross-entropy matches the Bradley–Terry–Luce negative log-likelihood), so they are complementary rather than redundant views of the same preference data.

Borrowing from past queries. When a new query arrives, its Bradley–Terry counts are initialized not from zero but from a weighted blend of the counts accumulated on earlier queries, where weights come from a sigmoid over cosine similarity between query embeddings. The weights are normalized so that the total inherited evidence is capped at one query's worth — preventing a long session from drowning out the current query's own data.

How preferences are simulated offline. A real user cannot be in the loop for repeatable experiments, so a proxy user stands in, implemented in two ways. The ROUGE-L simulator scores each response against a held-out reference and samples the binary outcome from a Bradley–Terry–Luce model over those scores, following POHF's offline protocol. The second simulator is an LLM judge given the query, a user-specific reference, and an anonymized response pair (positions randomized), asked to pick the response that better realizes the intended output considering task fulfillment, semantic intent, organization, coherence, and tone. Win rate is measured by pairing each method's final response against a "naive" response generated by the same backbone without persona selection or test-time optimization.

Models and protocol. The backbone is DeepSeek-V3.2 and the embedding model is Qwen3-Embedding-4B. Every configuration is run three times per user; means are taken per user, then aggregated across users. The authors state that persona-pool sizes, dataset statistics, and full hyperparameters are in the appendix, which is not included in the excerpt available here.

Why This Matters

The paper addresses a practical gap: personalization at inference time typically either requires expensive fine-tuning or relies on hand-written prompts. ID-TAP shows that a fairly small number of binary preference judgments — ten per query — can be turned into a meaningful persona-selection signal, without touching model weights and without white-box access to the LLM. It also treats the acquisition problem (which comparison to run next) as first-class, which is where the efficiency gains come from.

Real-world applications:

• Writing and editing assistants that shift between brainstorming, drafting, and rigorous critique personas depending on what the current request calls for.
• Customer-facing chat agents that adapt tone and verbosity per user from simple thumbs-up/thumbs-down comparisons rather than lengthy preference questionnaires.
• Educational tutors where the "stern tutor" versus "encouraging coach" distinction matters and can be learned from a handful of A/B choices.
• Creative tools where the same user wants different stylistic personas for different projects, and the system must transfer what it learned without being reset each time.

Industry relevance: The runtime profile matters here — the algorithmic overhead is 7–22% of end-to-end latency at K=100, meaning the personalization logic is not the bottleneck compared to generating the responses themselves. That makes the approach plausible as a layer on top of an existing black-box model API rather than requiring a bespoke training pipeline.

Future Directions

• Evaluation with real end users. The authors explicitly flag this: all reported results come from automated proxy feedback (ROUGE-L-based and LLM-judge-based), and the interface with actual humans remains untested.

• Finite-sample analysis of the neural/BT procedure. The paper currently motivates its design by citing existing theory for neural dueling, Bradley–Terry estimation, and M-estimation, but does not provide its own theoretical guarantees for the coupled two-system setup.

• Scaling the persona pool further. K=400 is used in the main experiments and K=100 in the runtime analysis; whether the acquisition rule stays efficient at substantially larger K — where 2(K−1) BT re-estimations per round grows linearly — is an open practical question.

• Validating the local-stability assumption. The method assumes user preferences are stable over short interaction horizons and that the historical context is fixed for the session. Behavior under drifting preferences or evolving user context is not reported.

Target Audience

Researchers and practitioners working on LLM personalization, test-time adaptation, or preference-based learning will get the most from this paper. It is particularly relevant to those already familiar with dueling bandits or RLHF-style pairwise feedback who want to see information-directed acquisition applied to a large discrete prompt space. Practitioners building product-side personalization layers over closed-weight model APIs will find the runtime numbers and the fixed-persona-pool framing directly useful. Readers looking for a theoretical treatment or a human-subject study should note that neither is included; the contribution is architectural and empirical.

Authors’ abstract

Effective test-time personalization requires LLM agents to adapt to query-dependent user preferences from lightweight interaction. We study persona selection with binary comparisons between candidate responses as the online signal. We present ID-TAP, a scalable, IDS-inspired neural dueling-bandit framework for large discrete persona spaces. ID-TAP combines a query-conditioned neural utility model with a lightweight Bradley-Terry explorer, and transfers comparison evidence across related queries through similarity-weighted warm starts. The neural model shares statistical strength across queries while preserving query-specific scores, and efficient BT updates score hypothetical challengers. Across seven user-conditioned generation benchmarks under reference- and model-based preference simulators, ID-TAP achieves the strongest aggregate performance among the evaluated baselines within a fixed ten-round comparison budget. These results support information-directed challenger selection as an effective design for test-time persona adaptation.