DARA: Few-shot Budget Allocation in Online Advertising via In-Context Decision Making with RL-Finetuned LLMs

AI summary

Overview

Research area: Online advertising under the AI-Generated Bidding (AIGB) paradigm, specifically few-shot budget allocation, combining large language model (LLM) in-context learning with reinforcement learning (RL) fine-tuning.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO-style clipped objectives, KL regularization, GRPO), auction/advertising terminology (RTB, ROI, marginal ROI), and LLM fine-tuning.

Scope (one sentence): The paper proposes DARA, a dual-phase LLM framework with a new GRPO-Adaptive post-training method, for allocating advertising budgets across time periods when only a few historical episodes are available.

Note: the paper content provided is truncated partway through Section 5.4.1. Results from Section 5.4.2, the appendices, and the hyperparameter tables are referenced but not included in the supplied text.

What This Paper Is About

Advertisers must decide how to spread a fixed budget across time periods to win the most valuable impressions, but each advertiser has personalized goals and little historical data, creating few-shot or cold-start conditions where traditional RL policies fail to generalize. LLMs can generalize from a few in-context examples, but they are not numerically precise enough for fine-grained budget optimization. The paper's goal is a framework that separates the "learn the pattern" part of the problem from the "precisely tune the numbers" part, and then sharpens both with a modified RL fine-tuning algorithm.

Key Contributions

1. DARA, a dual-phase LLM architecture for budget allocation. The task is decomposed along the temporal axis into two specialized agents: a Few Shot Reasoner that produces the first episode's allocation from a few-shot prompt, and a Fine-grained Optimizer that refines subsequent allocations using marginal ROI feedback and a sliding window of recent episodes. The stated aim is robust and interpretable decision making under few-shot conditions.

2. GRPO-Adaptive (GRPO-A), a new RL algorithm. An extension of GRPO in which the reference policy is periodically replaced with a snapshot of the current policy (after every K GRPO updates), with KL regularization reset against the new baseline. The motivation is that a static reference policy causes sensitivity to structured reasoning and symbolic precision to deteriorate during training.

3. A simulation environment for budget allocation. Modeled after real-world data distributions, built on controllable functional forms where the marginal ROI function is MROI_i(b) = max{F_i(b), 0} for 0 ≤ b ≤ B, with F_i either a polynomial or an exponential function. It continuously generates diverse allocation scenarios to support robust policy learning from limited data.

4. A dual-environment evaluation setup. One environment constructed directly from enterprise-scale real-world advertising data, and one synthetic environment for controlled generalization testing.

Main Findings

• Theoretical grounding: Under the assumption that each period's value function v_t(b_t) is differentiable, strictly increasing, and concave, the optimal allocation equalizes marginal ROI across all time periods (v'_1(b*_1) = ... = v'_T(b*_T)). Because the total budget B is fixed, maximizing ROI is treated as approximately equivalent to minimizing the variance of marginal ROI across periods — the metric the experiments report.

• Superior variance reduction in the real-world environment: After training in the simulation environment and evaluating in the real-world data environment, DARA reduces marginal ROI variance more than all baselines at all steps shown in Figure 3. The authors attribute its largest advantages to the later stages of allocation, where the fine-grained optimization phase has recent allocation outcomes to learn from.

• Advantage over DPO: DARA shows consistently lower variance than DPO. The paper argues DPO's token-level preference alignment is limited by a static modeling structure and no dynamic feedback responsiveness.

• Advantage over ABPlanner': DARA achieves better variance reduction than the RL-based ABPlanner', particularly in later steps, attributed to greater numerical sensitivity and the structural separation of the two phases.

• Ablation (four configurations): Directly applying a single LLM to end-to-end budget planning produced the poorest performance and the highest marginal ROI variance across the entire horizon. Adding RL fine-tuning to that single-phase setup gave only a slight improvement. Adopting the dual-phase architecture without RL improved performance substantially and consistently. The full model — dual-phase plus RL fine-tuning of both LLMs — gave the best overall performance.

• RL gains are amplified by architecture: The paper reports that RL's modest benefit in the single-phase setting became substantially more pronounced once each LLM was structurally disentangled and specialized, suggesting effectiveness depends on how RL is integrated, not only on whether it is used.

• Insensitivity to temporal granularity: Across 5 configurations of 2, 4, 6, 8, and 10 time periods, DARA improved over the strongest baseline (ABPlanner') by approximately 10.6% to 12.2%, with only mild fluctuation. The improvement was most pronounced at 6 periods, which the authors note aligns with typical ad scheduling practice of early, middle, and late phases.

• Why a single static reference policy fails: The paper states that experiments in Section 5.4.2 reveal that keeping the reference policy π_θ0 static throughout training causes sensitivity to structured reasoning and symbolic precision to gradually deteriorate, which motivated GRPO-Adaptive.

• Not reported in the supplied text: Absolute variance values, the specific LLM backbone used, the values of hyperparameters (referred to Appendix D), dataset sizes, the number of few-shot episodes n, and the specific environment constants for the real or synthetic environments.

Methodology in Plain English

The authors start from the observation that budget allocation naturally splits into two jobs with different demands. Early on, you need to generalize from a handful of past campaigns to set a broad allocation shape. Later on, you need to make small numerical corrections based on how the actual market responded. Rather than asking one model to do both, they assign one LLM to each job.

The Few Shot Reasoner receives a prompt built from a few historical episodes and outputs an initial allocation vector. The prompt contains four blocks: the task objective, the few-shot historical data, records of previous attempts, and a specified output format. The model is also asked to provide reasoning alongside the vector, for interpretability.

The Fine-grained Optimizer takes over from the second day onward. It sees the prior allocation plus the marginal ROI observed for each period, and adjusts the budget locally. A sliding window mechanism feeds it the most recent episodes' feedback so its adjustments track current conditions rather than stale ones. The full loop is given in Algorithm 1.

To move the LLMs beyond fluent-but-numerically-sloppy output, the authors fine-tune both with reinforcement learning. They use a group-based scheme: for each prompt, G candidate allocation vectors are sampled in parallel, each is scored by a composite reward, and the advantages are normalized within the group. A clipped, PPO-style update then raises the probability of the better completions relative to the worse ones.

Their modification, GRPO-Adaptive, targets the KL term. Standard GRPO anchors the policy to a fixed reference model, which the authors found suppresses reasoning sharpness over time. Instead, every K updates they snapshot the current policy, install it as the new reference, and reset the KL penalty against it. This lets the model keep improving while retaining a stability constraint.

The reward is composite: a term penalizing inconsistency in marginal reward across periods (-α Σ|r_i - r̄|), a constraint penalty for wrong output dimensionality or a total allocation far from B, and a bonus term that rewards moving budget toward periods with high historical reward and away from periods with low historical reward, subject to a minimum adjustment threshold δ and a clipping bound τ.

Training uses a multi-environment sampling procedure. Every fixed number of steps S, the model switches to a newly resampled or procedurally generated environment with T new marginal ROI curves. The intent is to prevent memorization of one distribution and force generalizable strategy learning.

Evaluation runs in two settings: an environment built from real online advertiser data from a leading global e-commerce platform, and a synthetic environment with diminishing-return MROI functions. Training used 8 NVIDIA H20 GPUs with 96GB of memory each, and each experiment was repeated five times with the mean and 95% confidence interval reported.

Why This Matters

Impact on research. The paper argues for explicitly separating generalization from optimization in sequential decision tasks, rather than training one monolithic policy. It also contributes GRPO-Adaptive as a general post-training idea — periodically refreshing the KL reference — that could be applied to other structured reasoning tasks where static references have been observed to blunt performance. The theoretical framing (equalizing marginal ROI at optimality, then optimizing marginal ROI variance) connects the advertising problem to a tractable objective.

Real-world applications:

• Automated budget pacing for advertisers on e-commerce and retail ad platforms, where campaigns run over a day divided into multiple time periods and the advertiser has little historical data.
• Cold-start onboarding of new advertisers, where a personalized allocation must be produced before any per-advertiser history exists.
• Campaign scheduling for seasonal or promotional events, where the early/middle/late phase structure matches the reported sweet spot of 6 time periods.
• Generalization to new markets or regions where the cost-reward pattern differs from the advertiser's previous experience.

Industry relevance. The work is a collaboration between Peking University, Alibaba Group, and Theta Labs, and evaluation uses enterprise-scale advertising data from a leading global e-commerce platform, with training on enterprise GPU infrastructure. The paper is published at WWW '26 and releases code at https://github.com/mx-song/DARA.

Future Directions

• Closing the remaining performance gap in late-stage allocation. The paper reports the largest gains in later steps, but does not quantify a ceiling. Determining how much variance remains reducible is open.
• Extending the sensitivity analysis. The supplied text covers only the number of time periods (2 through 10). Section 5.4.2 is referenced but its results are not in the provided content, so the full picture of the framework's sensitivity is incomplete here.
• Characterizing when the dual-phase split pays off. The ablation shows RL's benefit is amplified by task decomposition, but the paper does not report the boundary conditions — for example, at what number of few-shot episodes the gain disappears.
• Transferring GRPO-Adaptive beyond advertising. The dynamic-reference insight is presented as a general fix for reasoning degradation during fine-tuning, but it is validated only on this budget allocation task.
• Scaling to longer horizons and richer environments. The environment supports flexible sequence lengths and parameters, which the authors mention as enabling deeper analysis, but longer-horizon behavior is not reported in the supplied text.
• Deployment in live environments. The paper states that once the model shows stable performance in simulation it can be deployed directly in real environments, but no live deployment results are reported.

Target Audience

This paper suits researchers and practitioners working at the intersection of LLM agents, reinforcement learning from feedback, and computational advertising. Specifically: RL and LLM post-training researchers interested in GRPO variants and reference-policy design; ad-tech engineers building budget pacing or auto-bidding systems; and graduate students in machine learning or algorithmic mechanism design who want a worked example of combining in-context learning with structured numerical optimization. Readers without background in RL objectives or advertising auction terminology will find the methodological sections demanding, though the problem framing and ablation logic are accessible.

Authors’ abstract

Optimizing the advertiser's cumulative value of winning impressions under budget constraints poses a complex challenge in online advertising, under the paradigm of AI-Generated Bidding (AIGB). Advertisers often have personalized objectives but limited historical interaction data, resulting in few-shot scenarios where traditional reinforcement learning (RL) methods struggle to perform effectively. Large Language Models (LLMs) offer a promising alternative for AIGB by leveraging their in-context learning capabilities to generalize from limited data. However, they lack the numerical precision required for fine-grained optimization. To address this limitation, we introduce GRPO-Adaptive, an efficient LLM post-training strategy that enhances both reasoning and numerical precision by dynamically updating the reference policy during training. Built upon this foundation, we further propose DARA, a novel dual-phase framework that decomposes the decision-making process into two stages: a few-shot reasoner that generates initial plans via in-context prompting, and a fine-grained optimizer that refines these plans using feedback-driven reasoning. This separation allows DARA to combine LLMs' in-context learning strengths with precise adaptability required by AIGB tasks. Extensive experiments on both real-world and synthetic data environments demonstrate that our approach consistently outperforms existing baselines in terms of cumulative advertiser value under budget constraints.