Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach

AI summary

Overview

Research area: Post-training of large language models for online decision-making, spanning online learning theory (regret minimization) and LLM agent training.

Technical level: Advanced. The paper assumes familiarity with online learning concepts such as regret, full-information feedback, bandit feedback, non-stationarity, Follow-the-Regularized-Leader (FTRL), and supervised fine-tuning of LLMs.

One-sentence scope: The paper proposes Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a self-improving supervised fine-tuning procedure that repeatedly trains a model on its own lowest-regret decision trajectories across full-information online learning, multi-armed bandits, and non-stationary multi-armed bandits, and demonstrates it on numerical Transformers, open-weight LLMs, and GPT-4o mini.

What This Paper Is About

LLMs are increasingly deployed as agents that make decisions in interactive environments, but they were not designed for decision-making and have been observed to perform poorly even on basic online tasks — failing to explore adequately, showing linear regret growth, and failing to exploit consistently. Existing training fixes either distill action sequences from known expert algorithms (requiring predefined input/output formats and known optimal algorithms) or rely on manually crafted chain-of-thought reasoning structured around fixed algorithms. This paper asks whether regret — the standard performance metric of online decision-making — can serve as a general training signal to post-train LLMs into better decision-makers without any hand-specified expert algorithm.

Key Contributions

1. A new post-training meta-algorithm, Iterative Regret-Minimization Fine-Tuning (Iterative RMFT). At each iteration, the model samples trajectories across scenarios, the k lowest-regret trajectories per scenario are selected, and the model is fine-tuned on those trajectories via supervised fine-tuning. The regret metric is used as the selection criterion rather than a reward signal or distilled expert actions.

2. A language-space training procedure that preserves model-generated reasoning. Unlike prior work that distills from known algorithms or hand-crafts chain-of-thought output to mimic specific algorithms, Iterative RMFT integrates the model's own self-generated reasoning rationales, avoiding rigid output format engineering and providing flexible natural-language training signals.

3. Empirical demonstration across three model classes and three decision-making environments. The paper instantiates Iterative RMFT for Transformers with numerical input/output, for open-weight LLMs, and for the closed-weight model GPT-4o mini, covering full-information online learning (FOL), multi-armed bandits (MAB), and non-stationary multi-armed bandits (NS-MAB).

4. A theoretical insight linking single-layer attention to no-regret learning. In a simplified setting, the paper provides an analysis suggesting that a single-layer attention Transformer may become a no-regret learner under this training paradigm, with the imitation-based process converging to Follow-the-Regularized-Leader (FTRL).

Main Findings

• Regret-based self-imitation improves decision-making: Iterative RMFT is reported to improve LLMs' decision-making performance across a spectrum of models — Transformers with numerical input/output, lightweight open-weight LLMs, and the more advanced closed-weight LLM GPT-4o mini. Specific regret values and numeric comparisons are not reported in the available content.

• Enhanced exploration-exploitation behavior emerges without explicit guidance: Beyond lowering regret values, Iterative RMFT is reported to automatically elicit enhanced online decision-making behavior, such as an improved exploration-exploitation (E-E) tradeoff, despite not being explicitly guided by or distilled from existing algorithmic heuristics.

• Generalization across task structures: The flexibility Iterative RMFT offers regarding output and reasoning formats allows trained models to exhibit generalization across tasks varying in time horizon, action space size, reward generation processes, and decision-making contexts/scenarios described in natural language.

• Autonomous algorithm discovery framing: Because the method does not rely on pre-defined expert algorithms, it can be viewed as empowering LLMs to autonomously discover decision-making algorithms, incentivized by the regret metric.

• Contrast with direct regret optimization: In prior work (Park et al., 2025b) the regret objective itself was directly optimized through backpropagation, analytically yielding FTRL. In this paper the model instead imitates its own best-performing trajectory as measured by regret, and in a simplified single-layer attention setting this imitation-based process again converges to FTRL, suggesting no-regret behaviors may emerge naturally.

• Distinction from reward-maximization post-training: The paper argues reward maximization alone cannot automatically elicit efficient exploration, whereas regret minimization is positioned as a source of signals for learning to explore at inference time, with generalizability across action sets, rewards, and horizons.

• Specific quantitative results are not reported in the available content — no numeric regret values, regression coefficients (β̂, p_reg), SuffFailFreq or MinFrac values, dataset sizes, or training hyperparameters appear in the retrieved text.

Methodology in Plain English

The paper starts from a simple idea: in online decision-making, the standard way to score a sequence of decisions is regret — how much worse the agent did compared to the best possible choice in hindsight. Regret is used across full-information online learning, multi-armed bandits, and non-stationary bandits, which makes it a unifying metric across environments.

Iterative RMFT works as a self-improvement loop. In each iteration, for each of M language-described scenarios, the current model rolls out L decision trajectories. Because the trainer generates the environment, the trainer knows the rewards and can compute each trajectory's regret. The k lowest-regret trajectories for each scenario are collected into a training dataset. At the end of the iteration, the model is fine-tuned by supervised fine-tuning on that dataset, and the loop repeats.

The authors instantiate this meta-algorithm three ways:

• Numerical Transformers (Section 4): A single-layer linear attention model with an output operator mapping to the policy space (Softmax for the probability simplex in the Experts Problem, projection for the ℓ2-ball). Because this model is deterministic, diversity is created by adding Gaussian noise perturbations to the output policy at each round, producing L distinct trajectories.
• Open-weight LLMs (Section 5): Phi-3.5-mini-instruct, Gemma-2-9b-it, and Qwen3-8B are fine-tuned for language-grounded numerical decision-making, using stochastic decoding to sample trajectories.
• Closed-weight LLM (Section 6): GPT-4o mini is fine-tuned through a training API for language-grounded decision-making with real-world contexts across FOL, MAB, and NS-MAB, again using stochastic decoding.

Evaluation uses regret, a regression-based test for sublinear regret growth (fitting g(t) = β log t + α and reporting β̂ with its p-value p_reg), and the exploration metrics SuffFailFreq(t) and MinFrac(t) from Krishnamurthy et al. (2024). For NS-MABs, only SuffFailFreq(t) is reported because MinFrac(t) is stated not to provide meaningful insight there.

Why This Matters

Impact on research. The paper positions itself as an initial exploration and calls for more principled and novel post-training paradigms for LLMs on decision-making tasks. It argues that regret is a more suitable training signal than reward maximization for eliciting exploration, and it offers a bridge between online learning theory (where FTRL and no-regret guarantees are classical) and practical LLM post-training. It also offers a route to training decision-making agents when the optimal expert algorithm is unknown or unclear, which is common in real-world applications.

Real-world applications (the paper names these domains as sources of language-grounded scenarios):

• Healthcare, where language-grounded decision scenarios with sequential choices arise.
• Resource allocation, where agents must balance exploration and exploitation over time.
• Marketing, where decisions unfold in multi-turn interactions with changing rewards.
• Additional domains named in related work include software engineering, enterprise operations, and cybersecurity.

Industry relevance. Because the approach is supervised fine-tuning based, it is compatible with all existing post-training interfaces, including training APIs for proprietary closed-weight models such as the GPT series — meaning organizations without access to model weights can still apply it. The method does not require predefined input/output formats (such as fixed action-space size, reward-vector dimension, or time horizon), which reduces engineering overhead when deploying agents across varied tasks. Its improvement of the exploration-exploitation tradeoff is directly relevant to any deployment where an agent must try new options while capitalizing on known good ones.

Future Directions

• Extending beyond the three canonical environments. The paper's instantiations cover FOL, MAB, and NS-MAB; applying the meta-algorithm to richer real-world decision problems with variable problem structures remains open.
• Scaling the theoretical analysis. The convergence-to-FTRL argument holds only in a simplified single-layer attention setting; whether analogous guarantees hold for deeper Transformers and full LLMs is unresolved.
• Choosing the selection hyperparameters. The procedure depends on the number of trajectories L, the number selected k, and the number of scenarios M. Optimal or adaptive choices for these are not resolved in the available content.
• Establishing a more principled post-training paradigm. The authors explicitly frame their work as an initial exploration and call for new paradigms; questions about the limitations of regret as a signal when the environment deviates from the canonical online decision-making formulation remain open.

Target Audience

Researchers working at the intersection of online learning theory and LLM post-training; machine learning engineers who need to deploy LLM agents in interactive decision-making settings; and practitioners in applied domains such as healthcare, resource allocation, and marketing who need agents that explore and adapt rather than just exploit. Readers should be comfortable with concepts such as regret, bandit feedback, supervised fine-tuning, and attention mechanisms.

Authors’ abstract

Large language models (LLMs) are increasingly deployed as "agents" for decision-making (DM) in interactive and dynamic environments. Yet, since they were not originally designed for DM, recent studies show that LLMs can struggle even in basic online DM problems, failing to achieve low regret or an effective exploration-exploitation tradeoff. To address this, we introduce Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a post-training procedure that repeatedly distills low-regret decision trajectories back into the base model. At each iteration, the model rolls out multiple decision trajectories, selects the k-lowest regret ones, and fine-tunes itself on them. Unlike prior methods that (a) distill action sequences from known DM algorithms or (b) rely on manually crafted chain-of-thought templates, our approach leverages the regret metric to elicit the model's own DM ability and reasoning rationales. This reliance on model-generated reasoning avoids rigid output engineering and provides more flexible, natural-language training signals. Empirical results show that Iterative RMFT improves LLMs' DM performance across diverse models - from Transformers with numerical input/output, to open-weight LLMs, and advanced closed-weight models like GPT-4o mini. Its flexibility in output and reasoning formats enables generalization across tasks with varying horizons, action spaces, reward processes, and natural-language contexts. Finally, we provide theoretical insight showing that a single-layer Transformer under this paradigm can act as a no-regret learner in a simplified setting. Overall, Iterative RMFT offers a principled and general post-training framework for enhancing LLMs' decision-making capabilities.