Online LLM Personalization from Pairwise Preferences Across Users

AI summary

Overview

• Research area: Machine learning for LLM personalization, specifically online learning and contextual dueling bandits applied to inference-time prompt (persona) selection.
• Technical level: Advanced. The paper combines bandit regret theory (strong regret, Bradley-Terry-Luce choice models, Bayesian logistic posteriors) with an LLM-based preference benchmark; the plain-language reading is accessible, but the algorithm and evaluation protocol assume familiarity with online learning.
• Scope: The paper introduces a multi-user contextual dueling bandit formulation of online LLM personalization, releases a tutoring preference benchmark called TutorDuel, and proposes a baseline algorithm, CODuel, that transfers preference information across users.

What This Paper Is About

An LLM tutor can answer the same question in several styles (proofs, worked examples, Socratic questions, and so on), but each user's preferred style is hidden and varies by user and by question. The paper studies a setting where users arrive one after another for short sessions, the system shows two answers per question, and it observes only which of the two the user chose. The goal is to minimize cumulative strong regret — the summed utility gaps of both displayed answers relative to the user's best available persona — while starting with no preference data and exploiting structure shared across users.

Key Contributions

1. Problem formulation. The paper formalizes inference-time LLM personalization as a multi-user variant of the contextual dueling bandit problem, where questions supply contexts, personas supply arms, feedback is a single pairwise choice per round, and the objective is cumulative strong regret (the gaps of both displayed answers, not just the worse one). It also defines the distinct-pair floor G_T and "excess strong regret" R_T − G_T.
2. A reproducible benchmark and dataset (TutorDuel). TutorDuel contains educational questions across four topics, LLM-generated tutor answers in six teaching styles, pairwise judgments from LLM-simulated students with hidden preference profiles, and reference utilities obtained by fitting Bradley-Terry models to the judge's choice probabilities. The paper also provides a numerical simulator for controlled evaluation.
3. A baseline algorithm (CODuel). CODuel learns a mixture prior over user preferences from completed sessions and adapts to each new user within the session, using a lookahead pair rule and an evidence gate that falls back to a per-user no-transfer learner.
4. An evaluation protocol. Performance is bracketed between random pairs and a per-user linear oracle fitted in hindsight, and generalization is tested on new topics and new students never used in development.

Main Findings

• CODuel beats every comparison method on all three held-out rosters. On the held-out-profile, Composite, and Quirky rosters (u = 350 arrivals each, b = 24), CODuel removes 48%, 44%, and 46% of random pairs' excess regret, against 33%, 38%, and 38% for the best comparison method. Its excess regret is 1197.8 (±5.1), 1276.3 (±6.1), and 1250.2 (±5.4).
• The per-user linear oracle is far ahead. The oracle, which picks each user's parameters in hindsight on CODuel's features, removes 85%, 81%, and 82% of random pairs' excess regret. CODuel therefore reaches 54% to 57% of the oracle's benchmark score. The best fixed pair per user removes 76%, 66%, and 68%.
• Cross-user transfer is the main driver. The no-transfer ablation, which learns each user independently, scores 30, 27, and 28. Removing the learned population while keeping recalibration raises excess regret by 29% to 31% relative to full CODuel.
• Sharing information is not automatically better. On the held-out profiles, CoLSTIM (which learns each user separately) has a higher benchmark score (33) than any comparison method that shares information across users, and scores close to the no-transfer ablation on every roster. Meta-TS and HierTS score 23 there, below the no-transfer ablation's 30.
• Strongest head-to-head reductions. Relative to each comparator's excess regret, CODuel reduces excess regret by 22.5% against CoLSTIM on the held-out profiles and by 9.3% and 12.5% against COLDB on Composite and Quirky.
• New topics are easier than new students. On the held-out-topic roster, which keeps the development student profiles and changes only the questions, CODuel scores 54; new students lower its score by 6 to 10 points.
• Rebuilt rosters give similar results. Rerun on Composite and Quirky rosters redrawn with replacement from the same 350 students, CODuel scores 45 (42 to 47) and 46 (43 to 49), on average 7 points above COLDB.
• A second judge confirms the ranking. With TypeSafe Jev as the judge instead of gpt-6-luna, CODuel again has the lowest excess regret on all three rosters, with 25.4%, 20.4%, and 21.8% less excess regret than the best comparison method on each roster. The two judges agree with each other on 81.4% of the 204 author-labeled comparisons, and Jev matches gpt-6-luna's observed average agreement with the two authors (73.5%).
• The simulated judge tracks human choices reasonably. On 204 comparisons labeled by two authors, the judge agrees with each author on 74.0% and 73.0% of comparisons, versus 69.6% agreement between the two authors. Averaging over the two authors, the paired difference is +3.9 percentage points (95% cell-bootstrap interval: −2.5 to +10.0), which includes zero.
• Second-persona sampling is not the bottleneck for baselines. Drawing the second persona from a second posterior sample instead of the top-two rule moves Meta-TS's and HierTS's excess regret by at most 2.1% in either direction on each evaluation roster.

Methodology in Plain English

The researchers set up a tutoring scenario. Six "personas" are fixed system prompts representing teaching styles: a rigorous prover, a worked-examples tutor, a Socratic guide, an intuition-and-analogy tutor, a concise cheat-sheet, and a patient step-by-step coach. Tutor answers are generated with gpt-5.6-terra at temperature 1, and questions with gpt-5.6-sol (24 distinct questions per session across four topics).

Each simulated student is a natural-language profile. Eight hand-written profiles are used for development and eight held-out ones for evaluation; two additional rosters (Composite and Quirky) give each of 350 arrivals its own student, built from combinations of seven traits (Composite uses 350 distinct combinations among 3,072; Quirky draws traits uniformly and adds two of ten personal quirks). A judge model, gpt-6-luna with reasoning off, role-plays the student and compares two tutor answers. To reduce sensitivity to answer order and A/B labels, both orders and both label assignments are covered, giving 16 sampled choices across four calls per comparison; the four prompts' smoothed log-odds are aligned and averaged through a sigmoid.

For features, a rater model (TypeSafe Jev) scores how well each persona's answer style suits each question before any answer is generated — one call rates all six personas at about $0.03 per 1,000 questions. Scores are centered and scaled on development questions, and the feature vector for a persona-question pair is a 7-dimensional vector: six persona indicators plus the scaled rater score.

Reference utilities for regret come from fitting Bradley-Terry scores to each user's 15 pairwise probabilities per question by ridge-regularized maximum likelihood, then converting to utilities on a [0, 1] scale with a constant κ = 8 logits chosen on the development roster. The benchmark score is the share of random pairs' excess regret a policy removes, where 0 is random pairs and 100 is the distinct-pair floor G_T (779, 699, and 679 on the three rosters).

The proposed algorithm, CODuel, has four parts. A population model describes user preference vectors as a mixture of Gaussians fitted to completed sessions, with an isotropic component whose weight is at least α/(α + n) (α = 20, at most M_max = 16 learned components). For a new user, this mixture is the prior of a Bayesian logistic learner that updates a Laplace posterior after each choice and reweights components by how well each explains the session. A lookahead pair rule scores candidate pairs by their immediate belief-averaged regret plus the expected regret of greedy play over the remaining session, using a bank of the 16 most recent questions. An evidence gate opens only when, on completed sessions, the population learner predicts choices better than the no-transfer learner (z = 0.5, n_min = 8). In TutorDuel, an extra step learns a shared linear recalibration A = aI_K + A' of the rater's scores, with a ridge penalty of 10 on A', starting at A = I_K until 20 sessions are complete and refitting every 10 further sessions.

For controlled testing, a numerical simulator covers the case where the linear utility and Bradley-Terry assumptions hold exactly: d = 8 dimensions, u = 300 users, b = 20 questions, β ∈ {4, 12}, and four populations including a null population with no shared structure as a negative control for transfer. In TutorDuel, all design choices were made on a 300-arrival development roster before evaluation rosters were run.

Why This Matters

• Impact on research: The paper turns LLM personalization into a well-posed online learning problem with a regret objective that charges for both displayed answers, not just the loser. It provides a benchmark with hidden reference utilities, a per-user linear oracle as a target, and a negative-control roster that tests whether transfer hurts when there is nothing to share. The finding that some cross-user methods (Meta-TS, HierTS) underperform even a per-user learner is a concrete caution for the personalization literature.
• Real-world applications:
  • Tutoring and educational assistants that adapt explanation style to each student's discipline and preferences within a short session.
  • Conversational assistants and writing tools that learn tone, length, and format preferences from simple A/B choices rather than explicit preference surveys.
  • Customer-support or technical-help bots that adapt answer style (step-by-step vs. concise summary) per user and per question type.
  • Content and interface selection where a system must choose among a fixed catalog of presentation options and can only observe one pairwise preference per interaction.
• Industry relevance: The setup matches how deployed products actually collect feedback — a single click on one of two responses — and the paper shows that a modest pairwise signal, combined with a shared population prior, meaningfully reduces regret. The cost-conscious details (a rater at roughly $0.03 per 1,000 questions, cached judge probabilities, reusable utility tables) make the benchmark practical to reproduce. The gap to the oracle (CODuel reaches 54% to 57% of it) is framed explicitly as a target for future methods.

Future Directions

• Closing the oracle gap. CODuel reaches 54% to 57% of the per-user linear oracle's benchmark score; the paper offers this difference as a target for future methods, especially since the oracle's advantage comes from adapting the pair to each question rather than from a fixed per-user pair.
• Better use of the second arm. The paper notes that switching to a double-Thompson-style second draw moves Meta-TS and HierTS by at most 2.1%, but richer exploration rules for the second displayed persona remain open.
• Robustness to judge and simulation limits. The paper cites prior work showing LLM simulations can misrepresent group opinions and response variability; the human validation covers 204 comparisons with two authors and a confidence interval that includes zero, so broader human validation is a natural next step.
• Settings beyond a fixed small catalog. The problem assumes K = 6 fixed personas, known linear features, and a known β. Extending to larger or generated persona catalogs, unknown feedback noise, and preferences that drift within a session is not addressed here.
• The transfer question when users do not cluster. The null-population control and the negative-control roster raise the open question of how to detect and avoid harmful transfer in populations with no shared structure, which the evidence gate handles only heuristically.

Target Audience

Researchers working on bandits, online learning, and reinforcement learning from human feedback who want a concrete LLM-facing application; practitioners building personalization or preference-learning systems for chat assistants and tutoring products; and benchmark designers interested in how to construct reproducible preference datasets with hidden reference utilities, human-validated LLM judges, and explicit performance brackets from random baselines to hindsight oracles. Readers without a background in bandit regret will find the setup and results accessible, but the method section assumes comfort with Bayesian logistic regression, Laplace approximations, and mixture priors.

Authors’ abstract

Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 44-48% of random pairs' excess regret and achieves over half the performance of a per-user linear oracle fitted in hindsight. We release code, prompts, and the benchmark.