Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward

AI summary

Overview

• Research area: Machine learning / LLM post-training theory and alignment (learning from deployment feedback, offline RL, preference optimization).
• Technical level: Advanced. The paper is primarily a theoretical investigation with regret and sample-complexity bounds, accompanied by empirical validation on two LLM tasks.
• Scope: One-sentence scope: it formalizes fine-tuning LLMs from natural user-edit logs, derives learning guarantees for three feedback interpretations of edits (supervision, preference, reward/cost), and shows that ensembling them outperforms any single approach.

What This Paper Is About

When people use LLM writing assistants or coding agents, they often edit the model's output to fix it. Those edits are generated naturally by users, so they are cheap and plentiful, unlike annotator-labeled preferences. The paper asks how to fine-tune an LLM from such (context, agent response, user edit) logs, and it observes that the very same edit data can be read three different ways: as supervised targets, as a preference of the edit over the original response, and as a cost (edit distance) signal. The goal is to give principled reasons for choosing among these interpretations and to build a single method that combines their strengths.

Key Contributions

1. Formalizes learning from user edits as a two-phase protocol (Protocol 1). A dataset of n edits D = {(x_i, y_i, y_i')} is used for an offline learning phase, followed by T online episodes in which the agent is penalized by edit cost c_t = Δ_edit(y_t, y_t') and the objective is to minimize total edits.
2. Derives suboptimality and sample-complexity bounds for each of the three feedback types — SFT on edits (Theorem 1), DPO on edit-induced preferences (Theorem 2), and pessimistic cost-model RL (Theorem 3) — and shows they carry different trade-offs depending on the user, data distribution, and model class.
3. Proposes two ensembling strategies: early ensembling, which jointly optimizes preference and supervised losses with a mixing coefficient λ (Equation 7), and late ensembling, which trains a set of policies Ψ = [π̂_SUP, π̂_PREF, π̂_RL, π̂_EF] offline and selects among them online with a UCB bandit (Algorithm 1).
4. Validates the approach empirically on two domains adapted from Gao et al. (2024a) — email writing and summarization — showing the ensembling procedure outperforms methods that learn from individual feedback and can robustly adapt to different user-edit distributions at test time.

Main Findings

• User behavior is modeled by a balance equation. Assumption 1 states that for all x, y, y' the ratio q(y'|x,y)/q(y|x,y') equals π*_β(y'|x)/π*_β(y|x), and that the optimal response has probability at least γ_min(x) > 0 of being produced by the user — small, but not required to scale with |𝒴|.
• Edits induce a contraction. Lemma 1 shows the user distribution satisfies a total-variation contraction with factor (1 − γ_min(x)), meaning repeated application moves policies toward a fixed point.
• DPO is theoretically justified on edit data. The balance equation implies the induced preference distribution satisfies a Bradley-Terry form and equals σ(c(x,y) − c(x,y')), even though only one of the two responses comes from the reference policy rather than both being i.i.d. samples of π_ref.
• SFT bound. Under Assumptions 1 and 2, SubOpt(π̂_SUP) ≤ ε + O(min(η_max · D(π_ref, π*_β), η̄_max · D^(1/2)(π_ref, π*β))) with probability at least 1 − δ when n ≥ Ω(log(|Π|/δ)/ε²), where η_max = 1 − min_x γ_min(x) and η̄_max = sqrt(E{x∼ρ}[(1 − γ_min(x))²]).
• DPO bound. Under Assumptions 1, 2, and 3 (preference concentrability C_PREF), SubOpt(π̂_PREF) ≤ ε with probability at least 1 − δ when n ≥ Ω(C_PREF · log(|Π|/δ) / (β² (σ'(−V_max))² ε²)), paralleling techniques from Rosset et al. (2024) and Xie et al. (2024).
• RL bound. Under Assumption 2, SubOpt(π̂_RL) ≤ ε with probability at least 1 − δ when n ≥ Ω(log(|Π|/δ)/ε² + C̄² · d_Elud(ℱ) log(|ℱ|/δ)/ε²), where C̄² is the policy concentrability coefficient of π_ref and d_Elud(ℱ) is the Eluder dimension of the cost function class.
• Trade-off guidance. When the preference coverage coefficient C_PREF is small, DPO can be far more sample-efficient than SFT and RL. When the weighted approximation error is small, SFT is preferred. When approximation error and preference concentrability are large but policy concentrability is small and the cost function is simple (for example, a low-dimensional linear function), learning the cost function first and optimizing it is the most sample-efficient path.
• Empirical result stated in the abstract. On two domains adapted from Gao et al. (2024a), the ensembling procedure outperforms methods that learn from individual feedback, and it adapts robustly to different user-edit distributions at test time.
• Strong vs. weak users. A strong user edits according to every individual preference in the latent preference string (higher γ_min, faster convergence to the optimal policy); a weak user samples a subset of preferences per interaction (smaller γ_min, slower convergence). Both share the same optimal behavior.
• Experimental detail reported. Offline datasets of 20,000 examples for summarization and 10,000 for email writing were collected separately with strong and weak users, giving 4 separate offline datasets; evaluation uses T = 200 examples per task with 3 seeds, always with the strong user at test time.
• Not reported in the provided excerpt. The full list of compared methods beyond 𝙱𝚊𝚜𝚎 (the text is truncated at "We consider the following approaches: (i) 𝙱𝚊𝚜𝚎 whi...") and the quantitative results tables are not included in the truncated content, so specific performance numbers per method are not reported here.

Methodology in Plain English

The authors set up a two-stage protocol. First, an offline dataset of user interactions — each consisting of a context, the agent's response, and the user's edited version — is used to adapt the model. Second, the adapted model is deployed for T episodes and is charged the edit distance between its response and the user's edited response; the agent wants to minimize the total charge.

From that single dataset they build three baselines. The supervised baseline simply fine-tunes the model to produce the edited response (Equation 3). The preference baseline treats the edit as evidence that the edited text is preferred over the original and runs DPO (Equation 4). The cost baseline fits a regression model to the observed edit distances (Equation 5), then uses a pessimistic version of that cost model (the least favorable function consistent with the data, f̄) inside a KL-regularized RL objective (Equation 6). A fourth method, early ensembling, adds the supervised loss to the preference loss with weight λ (Equation 7). A fifth, late ensembling, trains all of these policies, then during deployment uses a UCB bandit that picks policies based on their average cost so far and an uncertainty bonus α·sqrt(log(t)/N(π)) (Algorithm 1). The theoretical analysis assumes a smooth model of how users edit (the balance equation) and that the optimal policy is representable, and it measures success via sub-optimality against the KL-regularized optimal policy π*_β, accumulated as regret.

For experiments, the authors use two tasks from Gao et al. (2024a), email writing and summarization, over articles from 4 domains, each with a hidden preference string (for example, "structured, straight to the points, respectful, professional greeting and closing") that the agent cannot see and must infer from context. An LLM user simulates edits; edit distance is Levenshtein distance normalized by the number of tokens in the agent's response, computed with the NLTK word tokenizer. The agent model is Llama 3.1 8b Instruct with greedy decoding; the simulated user is Qwen 3 32B instruct with think tokens removed. The paper also mentions a pure online variant of the protocol with large T and no offline phase, for which a no-regret algorithm is derived in Appendix A.

Why This Matters

Impact on research: This is described as the first theoretical investigation of learning from user edits. It connects and compares three literatures that are usually studied in isolation — imitation learning from actions, preference learning à la RLHF/DPO, and reward-based RL — inside one protocol, and it supplies a formal justification for applying DPO to edit data whose pairwise distribution differs from the standard RLHF setup. It also gives concrete conditions for when each method should win, which is useful for interpreting and designing experiments.

Real-world applications:

• Writing assistants that continuously adapt to a specific person's tone, formatting, and length conventions from edits alone.
• Coding agents that learn a team's or repository's style from accepted diffs rather than from human-written preference labels.
• Personalization of enterprise LLM deployments, where edits accumulate as free "deployment logs" without hiring annotators.
• Test-time adaptation when a deployment's user population shifts away from the population that supplied the offline logs.

Industry relevance: The paper's central practical claim is that user edits are a freely generated, naturally occurring, and self-interested (hence reasonably high-quality) source of training signal, in contrast to compensated annotator data that needs quality checks. The late-ensembling algorithm is deliberately simple — train several policies, choose among them with a bandit — which makes it appealing for production deployment pipelines where retraining frequently is impractical.

Future Directions

• Scale and validate the empirical study. The excerpt is truncated before the results tables; a natural next step is a fuller characterization of which ensemble members contribute most under each domain and user type, and whether findings hold at larger model and dataset scales.
• Relax the assumptions. The theory relies on the balance equation, policy realizability, preference concentrability, and a γ_min lower bound. Testing behavior when users edit inconsistently, adversarially, or with multimodal context is an open question.
• Move beyond the reward-only online variant. Appendix A handles a pure online setting without an offline phase; extending no-regret guarantees to richer bandit or full-fine-tuning online phases is left open.
• Refine ensembling and cost modeling. The pessimistic cost function f̄ in Equation 6 is described as computationally impractical in the general setting, so developing tractable approximations and better ways to combine early and late ensembling are clear practical openings.

Target Audience

This paper is best suited to researchers and practitioners working on LLM post-training, RLHF/DPO, offline RL theory, and personalization from deployment data. It will be most valuable to readers comfortable with concentration bounds, suboptimality and regret definitions, Eluder dimension, and the standard RLHF formalism, as well as to applied engineers at companies that collect user-edit logs in writing or coding assistant products and want guidance on how to convert those logs into training signal.

Authors’ abstract

We study how to fine-tune LLMs using user-edit deployment data consisting of a set of context, an agent's response, and user edits. This deployment data is naturally generated by users in applications such as LLMs-based writing assistants and coding agents. The _natural_ origin of user edits makes it a desired source for adapting and personalizing LLMs. In this setup, there emerges a unification of various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper, we initiate the theoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains adapted from Gao et al. 2024, we show our ensembling procedure outperforms these methods that learn from individual feedback. Further, we show that our proposed procedure can robustly adapt to different user-edit distributions at test time.