WHALE: A Simple Recipe for Joint Harness-Weight Optimization

AI summary

Overview

• Research area: Machine learning for agentic language-model systems, specifically the joint optimization of model weights and the executable "harness" code that surrounds them (tool schemas, context management, parsing, error handling, termination policy).
• Technical level: Intermediate. The recipe itself is conceptually simple, but the paper assumes familiarity with online fine-tuning, rejection sampling, and LLM-driven search over code.
• Scope: The paper introduces WHALE (Weight-Harness Alternating Learning), a framework that alternates weight updates with search over executable harness code, and evaluates it against weight-only, harness-only, and prompt-restricted joint baselines across three tool-using domains.

What This Paper Is About

An agentic language-model system has two halves: the model weights, and the harness code that decides what the model sees, how tools are called, how errors are handled, and when the run stops. The paper argues that optimizing either half while freezing the other leaves the system bottlenecked, because a weight update can change which harness works well and a harness update can change which model capabilities are usable. WHALE addresses this by alternating two phases — train the model under the current harness, then search for a better harness under the updated model — and by studying how long each phase should run before switching.

Key Contributions

1. A modular alternating framework. WHALE decomposes joint weight–harness optimization into two conditional updates, each performed with respect to a fixed counterpart: θ_{k+1} = ModelUpdate(θ_k; h_k, D_weight) and h_{k+1} = HarnessSearch(h_k; θ_{k+1}, D_harness). It requires only black-box interfaces, so any weight-update and harness-search method can be substituted.

2. A concrete instantiation. The authors pair online rejection-sampling fine-tuning (RSFT) for weight updates with Meta-Harness (Lee et al., 2026) for harness search over executable code, and release code at https://github.com/krafton-ai/WHALE.

3. An adaptive switching rule. Adaptive WHALE replaces the fixed per-cycle budgets (E, I) with a per-phase patience rule driven only by training signals and never by a validation dataset, removing the schedule from the hyperparameter set.

4. A controlled, budget-matched schedule study. The paper compares alternating versus stagewise optimization, sweeps five fixed schedules, and separates harness-search expressivity from prompt-only adaptation via a Fast–Slow Training (Tiwari et al., 2026) control that uses the same update methods and schedule but restricts search to system and user prompts.

Main Findings

• WHALE beats single-component and prompt-only baselines. Using one fixed schedule, (E, I) = (0.6, 6), WHALE achieves the highest accuracy in all three domains, outperforming the stronger single-component baseline by 7.67–10.05 percentage points and Fast–Slow Training by 4.15–13.00 points in best mean@8 accuracy. Across the paper, the reported ranges are 7.67–24.38 points over weight-only and harness-only, and 4.15–13.00 points over FST.

• The bottleneck depends on the domain. SearchQA is harness-dominant: harness search matches the peak accuracy of weight updates using only 5.79% as many rollouts. Mathematical reasoning is model-dominant: harness search reaches only 0.42%, compared with 15.42% for weight updates. These comparisons count target-agent rollouts and exclude proposer compute in the harness search phase.

• Capability-level behavior in SearchQA. Format compliance peaks at 97.07% for weight-only and 99.98% for harness-only. Retrieval accuracy improves only modestly under weight-only, while harness-only raises it from 26.88% to 60.61%; WHALE reaches the highest retrieval accuracy at 65.41%. Answer extraction reverses the balance, with weight-only peaking at 79.93% and harness-only at 57.28%; in WHALE, extraction improves sharply in the cycle 1 harness-search phase and further in the cycle 2 weight-update phase, but is held back by the harness-search phases from cycle 3 onward.

• Capability-level behavior in mathematical reasoning. The bottleneck is response truncation. Weight-only cuts the truncation rate from 95.83% to 30.83%. Harness-only barely moves format accuracy. WHALE's cycle 1 harness-search phase raises format accuracy from 0.83% to 4.38% in 4,608 rollouts, a gain of 3.54 percentage points, whereas harness-only spends 46,080 rollouts for a gain of 0.63 points (0.00% to 0.63%).

• Small alternation beats stagewise. Stagewise reaches (E*, I*) = (2.7, 32) in SearchQA within the 4-epoch and 40-iteration budgets for 43.02% test mean@8, and (E*, I*) = (4.3, 33) in Math within the 6-epoch and 60-iteration budgets for 15.63%. WHALE with (0.6, 6) exceeds both by 5.32 and 9.16 points and passes the final stagewise accuracy after only 29% and 49% as many rollouts. In Math, the 60-iteration stagewise harness search keeps improving its training score yet gains only +0.21 percentage points over weight-only on the test set.

• Phase length has two failure extremes. Among five schedules — (0.2, 2), (0.2, 6), (0.6, 2), (0.6, 6), (1.0, 10) — the same schedule is strongest in both domains: (0.2, 6) reaches 50.09% in SearchQA and 28.33% in Math, exceeding (0.6, 6) by +1.75 and +3.54 points. The Math (0.2, 2) run accepted a chance-inflated candidate, destabilized, and was stopped. Scaling budgets from (0.2, 6) never helps: accuracy falls from 50.09% to 48.34% to 45.93% in SearchQA and from 28.33% to 24.79% to 24.79% in Math. The two budgets interact: in Math, raising E from 0.2 to 0.6 helps at I = 2 (16.67% to 17.92%) but hurts at I = 6 (28.33% to 24.79%).

• Adaptive scheduling works without fixed budgets. Adaptive WHALE reaches 52.82% in SearchQA, +4.48 points over (0.6, 6) and +2.73 points over the best hand-tuned schedule (50.09%), using 23% fewer rollouts than the (0.6, 6) run needs for its own best. In Math it reaches 26.46%, +1.67 points over (0.6, 6) with 4% fewer rollouts but 1.87 points below the best hand-tuned schedule (28.33%). Realized median phase lengths are 0.24 and 0.29 weight-update epochs with I = 7 harness-search iterations in both domains, extending to as much as 1.16 epochs and I = 13 when training signals keep improving.

Methodology in Plain English

The system has two knobs: the model's weights and the harness program. The authors refuse to turn both at once, because each one changes the ground under the other. Instead they take turns.

In the weight phase, the model plays against the frozen harness, generates several attempts per prompt, and keeps only the attempts the verifier marks correct. It then fine-tunes on those kept attempts — an online version of rejection-sampling fine-tuning. The phase budget E is measured in training epochs drawn from a persistent sampler.

In the harness phase, the weights are frozen and an LLM-based proposer inspects an archive of previously written harnesses together with their scores, per-example outcomes, and saved trajectory logs. It proposes M new candidate harnesses per iteration, each is evaluated on the harness-search dataset with one rollout per candidate–example pair, and the highest-scoring harness in the expanded archive is accepted. The phase budget I is the number of such iterations.

The two phases repeat, so each component keeps re-adapting to the other. For adaptive WHALE, the fixed budgets are replaced by patience rules: the weight phase stops once its sliding-window training reward has not set a new best for a fixed number of steps, and the harness phase stops once the best training score in its archive has not improved for a fixed number of iterations, subject to minimum phase lengths. In the reported adaptive configuration, the averaging window, minimum phase length, and patience are all 0.2 epochs, and the harness phase runs at least 6 iterations and stops after 2 without a new best.

The evaluation uses Qwen3.5-2B for SearchQA and Math and Qwen3.5-4B for chess puzzles, with rollouts at temperature 1.0, top-p 1.0, top-k 20; response limits of 8,192, 8,192, and 16,384 tokens; G = 8 sampled trajectories per prompt for weight updates; and 8 trajectories per test example reported as mean@8. All baselines start from the same domain-specific (θ₀, h₀) and receive matched budgets. Domains span 18,946 SearchQA training questions, 17,917 DAPO-Math-17K problems, and 16,384 Lichess puzzles, each with a 256-example harness-search set; SearchQA's test set is 700 questions (100 each from seven benchmarks), and Math tests on AIME 2024 and AIME 2025.

Why This Matters

Impact on research. Prior joint-adaptation work optimizes weights alongside textual prompts only. WHALE extends the adaptable surface to full executable harness code — tool interfaces, output formatting, feedback text, turn allocation, stopping criteria — and gives a controlled, budget-matched account of when that extra expressivity pays off. It also reframes a practical engineering question (how often to retrain versus how often to rewrite the scaffold) as a scheduling problem with measurable failure modes at both extremes.

Real-world applications implied by the evaluated domains:

• Retrieval-augmented question-answering assistants, where the harness controls query post-processing, how many documents return, and when the agent must answer instead of searching again.
• Mathematical and STEM reasoning agents that write and execute code, where the harness governs response caps, turn limits, and final-answer recovery.
• Sequential decision-making agents such as game or puzzle solvers, where the harness formats moves, validates legality, and handles opponent replies

Authors’ abstract

Agent performance depends jointly on the model parameters and the executable harness code that manages context and control flow. Optimizing either component in isolation can leave the system bottlenecked by its frozen counterpart: weight updates can change which harness is effective, while harness updates can change which model capabilities are exposed. Existing joint-adaptation methods optimize weights and textual prompts but leave the broader harness fixed. We propose Weight-Harness Alternating LEarning (WHALE), a simple recipe that alternates two phases: updating the model under the current harness, then searching for a better harness under the updated model. We instantiate these two phases with online rejection-sampling fine-tuning and Meta-Harness, respectively. When to switch is a key design choice: to separate real improvements from noise without over-optimizing against a changing counterpart, WHALE uses either fixed phase durations or an adaptive patience rule over training signals. Using Qwen3.5-2B/4B agents across three domains (search question answering, mathematical reasoning, and chess puzzles), WHALE outperforms weight-only, harness-only, and Fast-Slow Training by 4.15-24.38 percentage points in best mean@8 accuracy. Either component can be the bottleneck: harness search matches peak weight-only accuracy with far fewer rollouts in SearchQA, but improves math accuracy only after a weight update. Small interleaved updates also outperform stagewise weight-then-harness optimization in accuracy and rollout cost. The code is available at https://github.com/krafton-ai/WHALE.