Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs

AI summary

Overview

Research area: Natural Language Processing / LLM post-training — specifically how to split a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL).

Technical level: Intermediate. The paper is written around a formal optimization setup (allocation ratio, tolerance, near-optimal region), but its central argument is intuitive: the best SFT–RL split is not a single point but a wide band.

Scope: An empirical study of SFT–RL annotation budget allocation across the Llama 3, Qwen 2.5, and Qwen 3 model families up to 14B parameters, four tasks, and both off-policy (DPO) and on-policy (GRPO) RL methods, framed as a transferable "near-optimal region" rather than a single optimal ratio.

What This Paper Is About

Post-training an LLM usually means doing SFT first, then RL or preference optimization, but the field has no principled way to decide how much of a fixed annotation budget each stage should get. The authors show that instead of hunting for one exact best split ratio, you can identify a wide near-optimal region — a set of ratios that all come within a small tolerance of peak performance. Their central claim is that this region widens as models get bigger and, crucially, transfers from small proxy models to large target models, so you can run cheap experiments at small scale and reuse the answer at large scale.

Key Contributions

1. Near-optimal region analysis. The authors show that across tasks and model families, even a small tolerance (a few percent of the optimum) produces a wide near-optimal region rather than a single sharp optimum.

2. Scale-dependent region expansion. At a fixed tolerance, the near-optimal region generally widens with model scale, and regions identified on small proxy models transfer reliably to larger target models — removing the need for exhaustive large-scale search.

3. Generality across settings. The widening and transfer behavior holds across tasks, model families, and both off-policy (DPO) and on-policy (GRPO) RL methods; the authors also show how annotation cost asymmetry between SFT and RL data shifts the near-optimal region.

4. A practical proxy allocation procedure. A three-step recipe — proxy sweep on a 5-point ratio grid, choose a tolerance, select the midpoint of the region — which satisfies the chosen tolerance in 94.3% of cases for ε = 5% and 97.1% for ε = 10%.

Main Findings

• The optimum is a plateau, not a point. Under a 10% tolerance (retaining at least 90% of optimal performance), most tasks admit near-optimal ratios covering 55%–75% of the feasible allocation space.

• The region widens with model scale. Under a fixed tolerance, smaller models often admit only a narrow set of near-optimal ratios under strict tolerances, while larger models accommodate a broader range. The authors interpret this as a fixed-tolerance phenomenon, since a higher absolute peak performance mechanically creates more slack at the same threshold, and they note edge cases where widening can fail under an alternative tolerance definition.

• Near-optimal regions transfer better than the exact optimum. Under point optimum (ε = 0), transferability is sometimes high but inconsistent across tasks, budgets, and model families — particularly for the Qwen 2.5 family. Allowing a modest tolerance substantially and consistently improves transfer.

• Near-optimal ratios are roughly contiguous. Per-ratio hit-rate heatmaps show that as tolerance increases, high-performing ratios expand contiguously, supporting the assumption that the region is approximately an interval.

• One apparent exception. For Qwen 2.5 summarization, the region narrows marginally at larger scales under higher tolerances, which the authors attribute to saturation rather than a collapse in allocation behavior; cross-scale transfer remains strong there.

• On-policy RL behaves similarly. SFT–GRPO results on the Llama family match the SFT–DPO pattern, though with slightly stronger fluctuations, particularly on Helpfulness, consistent with higher variance in on-policy training.

• Cost asymmetry widens the region further. Using ρ = c_SFT / c_DPO ∈ {1, 2, 5, 10} with c_DPO = $0.001, the near-optimal region generally widens at the same tolerance as SFT annotations become relatively more expensive, meaning the choice of ratio matters less.

• Annotation dominates compute cost. Per-example training compute ranges from $2×10⁻⁵ (1B SFT) to $3×10⁻⁴ (8B DPO), which is 3–5 orders of magnitude below human annotation at $0.5–1.0 per example, and still below cheap synthetic annotation at $10⁻³ per example.

Methodology in Plain English

The authors set up the problem formally: a model of size N gets a total annotation budget B, and an allocation ratio r sends r·B samples to SFT and (1−r)·B to RL. They then replace the usual "find the single best r" question with an "ε-near-optimal region" — every ratio that keeps at least (1 − ε) of the best observed performance, where ε ranges over small values such as 2% to 10%.

They run a large sweep: model families Llama 3, Qwen 2.5, and Qwen 3 across scales up to 14B parameters; four tasks (math via GSM8K, instruction following via Tülu3 Persona IF and IFEval, summarization via Reddit TL;DR and ROUGE-L F1, and helpfulness via HelpSteer/HelpSteer2 with a reward model); an allocation grid of {0.00, 0.25, 0.50, 0.75, 1.00}; and budgets up to 15k samples. Because the small-budget regime (B < 5k) is noisy, they restrict scaling analysis to B ≥ 5k. To keep the sweep tractable they use LoRA for training.

They measure region width two ways — range width (span on [0,1]) and count width (fraction of grid points) — and measure cross-scale transfer as the fraction of a small proxy model's near-optimal ratios that remain near-optimal on a larger target model. They check that a denser 9-point grid (adding 0.125, 0.375, 0.625, 0.875) identifies the same regions. Finally, they validate the practical recipe by applying it to their existing 1B–8B runs.

Why This Matters

Impact on research. The paper reframes a costly, ill-conditioned hyperparameter search (the exact SFT–RL split) as a robust region-estimation problem. It also provides evidence that near-optimal regions, unlike point optima, are a stable object for cross-scale transfer — which is relevant to ongoing debate about whether post-training decisions generalize across model sizes.

Real-world applications:

• Choosing SFT–RL budget splits for new model families without exhaustive large-scale sweeps.
• Planning annotation spend in cost-constrained post-training pipelines, including settings where SFT annotations cost more than preference labels.
• Deciding how much freedom a team has in allocation when the exact ratio is driven by non-performance constraints such as annotation logistics or sample reuse.
• Serving as a blueprint for reusing small-model experimental results to guide large-model deployment decisions.

Industry relevance. Annotation is the dominant cost in post-training, and the authors note that retraining or rerunning stages for each allocation change makes full-scale grid search prohibitively expensive. A procedure that reliably selects a transferable ratio from cheap proxy experiments directly reduces the compute and annotation budget required to make principled post-training decisions.

Future Directions

• More complex schedules. The authors restrict analysis to two-stage SFT → RL pipelines and flag extension to multi-stage or interleaved schedules as a promising direction.

• Out-of-distribution settings. All experiments train on task-specific data drawn from the same distribution as evaluation; whether near-optimal regions and their transferability survive OOD evaluation remains an open question, since OOD scaling is less reliable with larger generalization gaps and frequent non-monotonicity.

• Larger models and more algorithms. Verification beyond 14B parameters (Llama 3 70B, Qwen 2.5 32B–72B) is left for the future, as are untested RL algorithms such as PPO and SimPO.

• Richer cost models. The budget is measured only in annotated samples; jointly modeling annotation, training compute, GPU time, and experimental search cost into a single scalar budget is described as non-trivial and without precedent in cross-scale scaling laws. Uncertain or adaptive annotation cost models are also flagged as future work.

Target Audience

Researchers and practitioners working on LLM post-training, data-centric machine learning, and annotation budget planning. It is most useful to readers who already understand SFT, DPO, and GRPO at a working level, and who need practical guidance on allocating limited annotation resources across training stages rather than a purely theoretical treatment.

Authors’ abstract

How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhaustive large-scale search. Our results hold consistently across tasks, model families, and both preference-based off-policy and reward-supervision on-policy RL methods. We further analyze how the asymmetry in annotation costs between SFT and RL data shifts the near-optimal region.