HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL

AI summary

Overview

Research area: Reinforcement learning for large language models, specifically rubric-based RL for open-ended generation (role-play, long-form writing, medical and scientific answers, instruction following).

Technical level: Advanced. The paper assumes familiarity with GRPO-style policy-gradient training, LLM-as-judge reward models, importance sampling, and multi-agent LLM pipelines.

Scope: The paper diagnoses why natural-language rubric rewards degrade during RL training and proposes HarnessRL, a training-time "harness" that co-evolves task sampling, rollout guidance, and rubric criteria alongside the policy, evaluated on a new five-domain suite called Rubric-Mix-10k.

What This Paper Is About

Open-ended tasks have no single correct answer, so researchers reward models with rubrics: lists of natural-language criteria scored by an LLM judge. The paper shows these rewards break down as training proceeds — either collapsing to uniformly low or high scores (no learning signal) or rewarding responses that merely look compliant while quality stalls. HarnessRL's goal is to fix the reward machinery itself during training, rather than fixing a rubric or a set of hints once and hoping it stays useful as the policy changes.

Key Contributions

1. A taxonomy and measurement of rubric-reward failure. The authors define three failure classes — blind (uniformly low), saturated (uniformly high), and spurious (scores separate responses but do not track quality) — and measure them against an oracle judge. In a rubric-based GRPO run these account for 61.6% of response pairs: 27.6% provide no usable contrast and 34.0% provide contrast that does not reliably track quality, including 19.8% that the reward ranks in the wrong direction.

2. HarnessRL, a harness that co-evolves with the policy. Three interfaces — task sampling (p_t), rollout guidance (g_t), and rubric criteria (c_t) — are updated during training by a harness evolver that attributes each failure to the interface that can address it, then validates candidate updates with a just-in-time critic before acceptance.

3. A training-time memory with dual retrieval. The memory stores rollout history, rubric history, and training dynamics per task draw, retrieved through a time index (same task, earlier draws) and a semantics index (related criteria across tasks), so the evolver can distinguish persistent failures from single-step noise and avoid reversing earlier updates.

4. Rubric-Mix-10k. A curated five-domain suite (medical, science, writing, role-play, instruction following) with 8,600 training tasks, 250 held-out tasks per domain, and one OOD benchmark per domain — 110,606 criteria averaging 12.9 per task.

Main Findings

• Strongest results at both scales. On Rubric-Mix-10k, HarnessRL-trained Qwen3.5-4B scores 64.5 in domain and 49.8 OOD, exceeding the strongest baseline (AMARIS++, at 61.5 and 47.0) by 3.0 and 2.8 points, and GRPO (57.8 and 43.9) by 6.7 and 5.9 points. At Qwen3.5-9B it reaches 69.5 and 57.3, margins of 2.9 and 3.1 points over AMARIS++.

• Faster to best checkpoint. The abstract reports that HarnessRL nearly halves the training steps to its best checkpoint, and that it reaches its best OOD checkpoint before every adaptive-training baseline. Exact per-method step counts are not reported in the visible text.

• Better reward-oracle agreement. At the end of training, reward-oracle agreement is 48.9% under HarnessRL versus 38.4% under GRPO.

• Both sub-components matter. Ablating the just-in-time critic drops Qwen3.5-4B performance to 59.5 in domain and 45.9 OOD (below AMARIS++ at 47.0 OOD); removing memory as well drops it to 58.1 and 44.1, within 0.2 points of GRPO.

• Single-interface harnesses are weaker than the joint harness. Adapting only the rollout interface gives 60.6 in domain / 46.3 OOD; only the rubric interface gives 61.2 / 47.2, against 64.5 / 49.8 for the full harness.

• Published single-interface rules capture most, but not all, of the gain. Replacing one interface with an existing method — SEC for sampling, RuscaRL for guidance, EvoRubric for rubrics — yields 63.9/49.2, 63.6/48.1, and 63.3/48.4 respectively.

• Uneven margins across domains. At Qwen3.5-4B the in-domain margin over the best baseline is widest on writing and role-play (+3.7 points each) and narrowest on instruction following (+0.9), where GRPO already reaches 68.1. The authors attribute the wide margin to many defensible responses saturating a frozen rubric, and the narrow one to near-verifiable criteria.

• Guidance-side and rubric-side adaptation transfer differently OOD. At Qwen3.5-4B, RuscaRL and SkillRL gain 3.0 and 2.6 points over GRPO in domain but barely transfer OOD (44.3 and 44.6 against 43.9), while EvoRubric and AMARIS transfer further (46.2 and 46.5).

• Statistically supported and stable. Paired bootstrap 95% intervals for the AMARIS++ margins (10,000 item-and-seed resamples) are [2.4, 3.6] in domain and [2.0, 3.5] OOD at 4B, and [2.2, 3.6] and [2.3, 3.9] at 9B, with permutation tests giving p < 0.001. Seed standard deviations are below one point for all arms and smallest for HarnessRL (0.3 and 0.4 at 4B; 0.2 and 0.3 at 9B).

Methodology in Plain English

The authors keep a standard GRPO training loop but wrap it in a harness with three adjustable dials, all decided at training time.

The sampling dial decides which tasks to draw. If a task's criteria are all being met by every rollout, it stops teaching anything, so its weight is lowered one level (from 1 through γ₁ = 0.61, γ₂ = 0.38, γ₃ = 0.17, set by a pre-training "cold start"). Each draw carries an importance-sampling correction u_t = 1/(N·p_t^i) so reweighting changes only how often a task appears, not its share of the objective.

The guidance dial mounts up to three fields after the prompt — <criterion> (a paraphrase of the target criterion), <reference> (positive and negative example responses), and <critique> (the previous group's shortfall). The seed harness mounts none, and a schedule removes remaining fields by training's end so guidance is empty at evaluation.

The rubric dial scores each response criterion-by-criterion with a frozen judge (gemma-4-31B-it) and combines binary scores with signed weights; the authors set Λ = 100 so reward changes are read in "points." Criteria can be created, rewritten, deleted, reweighted, merged, or split.

Every step, an attributor labels each criterion blind, saturated, spurious, or agree, asking three questions: can the policy reach this property (reach), has the policy already mastered it (training stage), and does meeting it actually reflect quality or just surface compliance (reward hacking). Each failure label routes to one dial. A proposer then drafts candidate updates, and a just-in-time critic measures each on the current frozen policy: it regrades the recorded group for a rubric change, re-samples a group under candidate guidance for a rollout change, and makes no policy query for a sampling change. An update is accepted only if its measured effect is positive and, for rollout and rubric updates, falls inside a calibrated band ([10.2, 28.0] and [3.1, 28.0] reward points). Rejected candidates return to the proposer, with up to B = 3 rounds. Everything is logged to memory, which the evolver reads back through the time and semantics indexes (top-k = 4 criteria with cosine similarity ≥ 0.82 under bge-base-en-v1.5).

Evaluation separates the reward under test from the reported metric: in-domain held-out tasks are scored under their unchanged seed rubrics, while OOD uses official protocols — MedQA and GPQA-Diamond accuracy, RoleBench ROUGE-L, IFBench programmatic constraint checks, and WritingBench's own rubric. Only WritingBench uses the training judge, so the other four test whether gains survive beyond the adapted reward. For analysis only, Claude Sonnet 5 serves as an oracle judge, comparing all 28 pairs within each group of J = 8 rollouts in both orders; no harness role sees oracle output.

Why This Matters

Research impact. The paper reframes rubric reward design as a diagnosis-and-repair problem rather than a one-shot prompt-engineering problem. Its failure taxonomy, and the finding that 61.6% of GRPO's response pairs carry no usable or misleading signal, gives the field a measurable target; the oracle-agreement metric (48.9% vs 38.4%) offers a way to audit reward quality rather than just end-task score.

Real-world applications.

• Healthcare communication training and patient-facing answer generation, where rubrics encode clinical correctness and safety and where spurious "confident-sounding" compliance is dangerous.
• Long-form content and marketing writing assistants, where the paper's largest gains appear (+3.7 points at 4B).
• Role-play and character agents in entertainment and customer simulation, also +3.7 points at 4B.
• Instruction-following and constraint-satisfaction systems, where gains are smallest (+0.9) because rule-based checks already work well.

Industry relevance. The method is a drop-in wrapper around an existing GRPO pipeline with a frozen judge, and it reports better final quality with roughly half the steps to best checkpoint — a direct compute argument. Its stability result (seed standard deviations of 0.2–0.4 points) is attractive for production training runs, though the paper notes that costs are reported in cluster hours on a 16-node cluster with eight NVIDIA H100 80GB GPUs per node without listing the actual figures in the visible text.

Future Directions

• Scaling the evolver's backbone. All three evolver roles share the same frozen gemma-4-31B-it that serves as judge. Whether a stronger or separately trained evolver widens the margins, and whether judge and evolver should be decoupled, is unexplored.

• Extending beyond five domains and one guidance vocabulary. The guidance field set is fixed at three fields, and the acceptance bands come from a single cold-start pass on Qwen3.5-4B reused for the 9B model. Whether these transfer to other model families, longer training, or domains outside Rubric-Mix-10k is not reported.

• Verifying the failure taxonomy at scale. The visible text describes annotation of 200 attributor labels (50 per class) and 200 accepted rubric updates by three annotators, but the resulting agreement figures are cut off; a fuller human validation of the four-way label scheme is a natural next step.

• Cheapening the critic. The just-in-time critic requires re-sampling or regrading at every candidate update and is the component whose removal costs the most. Reducing that cost while keeping the measured-acceptance guarantee is an open engineering question.

Target Audience

Researchers and engineers working on LLM post-training who already use or are considering rubric-based or LLM-judge rewards for open-ended tasks — particularly those who have seen rubric scores rise while actual output quality plateaus. It will also interest practitioners building evaluation harnesses, since the oracle-agreement framing and the Rubric-Mix-10k suite are useful independent of the training method. Readers without a background in policy-gradient RL and LLM-as-judge pipelines will find the method sections dense.

Authors’ abstract

Rubric-based reinforcement learning (RL) trains policies for open-ended generation with natural-language criteria as rewards, but these rewards can become ineffective as training progresses. We identify two failure modes. Under non-discriminative rewards, scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability. Under spurious rewards, higher scores do not correspond to better responses. Both reflect a training configuration whose task sampling, exploration, and evaluation stay fixed as the policy evolves. We introduce HarnessRL, a training-time harness that co-evolves with the policy through three interfaces for task sampling, rollout guidance, and rubric criteria. HarnessRL maintains a training-time memory from which a harness evolver retrieves within-task history and cross-task evidence. Using this evidence, the evolver attributes each reward failure to the interface whose update can address it. It then runs a proposer and just-in-time critic loop in which the critic accepts a candidate update only after measuring its effect on the current frozen policy, returning rejected candidates for revision. We also curate Rubric-Mix-10k, a five-domain training and evaluation suite for open-ended RL across medical, science, writing, role-play, and instruction-following tasks. On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by 2.8 to 3.1 points across both 4B and 9B model scales and nearly halves the training steps to its best checkpoint. Code and data are available at https://github.com/Tongyi-ConvAI/Qwen-Character/tree/main/HarnessRL.