J-Zero: Unified Challenger--Solver--Judge Self-Evolution from Zero Data

AI summary

Overview

Research area: Self-evolving large language models — specifically zero-data self-play / self-improvement, spanning verifiable domains (math, general reasoning, instruction following) and unverifiable domains (open-ended writing and helpfulness).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (group relative policy optimization), reward models, and Bradley–Terry preference learning, though the high-level idea is explained in accessible terms here.

Scope in one sentence: J-Zero is a framework in which a Challenger, a Solver, and a Judge all co-evolve from zero external data, with the Judge trained on preference pairs whose ordering is known in advance from how each response was produced, enabling self-improvement in both verifiable and unverifiable domains.

What This Paper Is About

Self-evolving models that generate their own training tasks work in verifiable domains because a checker can supply objective feedback, but they stall in unverifiable domains where a learned Judge must score responses in place of a verifier. A frozen Judge can only push the Solver toward preferences it already knows, so improvement is capped by the Judge's own evaluation ability.

J-Zero's goal is to lift that ceiling by making the Judge co-adapt inside the self-play loop, using preference labels derived from the structural roles of the Challenger and Solver rather than from the Judge's own scores — and to do so without any external data or human feedback.

Key Contributions

1. A unified three-role self-evolution framework (J-Zero). The Challenger and Solver co-evolve adversarially under a frozen Judge, while the Judge itself is updated in the same loop, supporting self-improvement in both verifiable and unverifiable domains from zero data.

2. Two label-free sources of Judge supervision. Role-asymmetry pairs label the Solver's response as preferred over the Challenger's, because the Solver is trained to answer well while the Challenger is trained to make tasks difficult rather than to answer them. Subtask-amplification pairs label the Solver's decomposed-and-recombined answer as preferred over its one-shot answer, exploiting the principle of iterated amplification.

3. An adversarial training scheme with an information-theoretic task filter. The Challenger maximizes task difficulty (measured as 1 minus the mean Judge score) with a BLEU-based repetition penalty and a format check requiring well-formed tasks inside <question> tags, while the Solver is trained on the top-K tasks selected by response-level score dispersion.

4. Empirical evidence that Judge co-adaptation is the key driver. J-Zero outperforms the baselines by an average of 4.2 points on verifiable and 8.0 points on unverifiable domains and keeps improving through at least ten iterations, while the baselines degrade after two.

Main Findings

• Verifiable-domain gains: On the eleven verifiable benchmarks, J-Zero improves average performance by 9.47 and 7.88 points over Qwen3-4B-Base and Qwen3-8B-Base respectively, and outperforms R-Zero by 4.74 and 3.56 points — despite R-Zero being designed specifically for verifiable self-evolution. Overall averages: 54.38 for Qwen3-4B-Base and 58.55 for Qwen3-8B-Base, versus base-model scores of 44.91 and 50.67.

• Unverifiable-domain gains (widest gap): J-Zero improves the unverifiable average by 11.23 and 10.18 points over the base models, with the largest gains on AlpacaEval 2.0 (6.22 → 28.56 and 12.93 → 33.53). By contrast, R-Zero improves by only 3.08 and 2.31 points, and G-Zero by only 1.31 and 2.08 points.

• Large jumps on hard math benchmarks: On AIME24, J-Zero reaches 16.15 (Qwen3-4B-Base) and 19.58 (Qwen3-8B-Base), versus base scores of 8.96 and 10.52. On AIME25 it reaches 15.83 and 15.94, versus base scores of 6.67 and 8.96.

• Role-asymmetry labels are reliable from the start: Evaluated by an external LLM judge (Claude Opus 4.8) in both presentation orders with ties dropped, the Solver's response wins more than 60% of comparisons at every iteration. The win rate declines from 87.9% to approximately 66%, which the authors attribute to the increasingly difficult adversarial curriculum rather than to unreliable labels.

• Subtask-amplification labels become reliable once the Solver matures: The divide-and-conquer response wins fewer than half the comparisons in the first three iterations (21.1% at iteration 1), exceeds 50% from iteration 4, and later reaches roughly 70 to 80%. The two reliability curves cross mid-training, so the Judge is never without a usable signal.

• Both pair types matter, amplification more: Ablation (Table 3) shows removing subtask-amplification pairs causes a −1.64 drop and removing role-asymmetry pairs a −0.97 drop, against an overall J-Zero score of 37.59.

• Frozen Judge plateaus: The frozen-Judge variant tracks J-Zero closely for the first three iterations, then plateaus, ending 1.66 and 4.44 points below the full run on the verifiable and unverifiable domains respectively (52.72 vs. 54.38 verifiable; 16.37 vs. 20.81 unverifiable).

• Sustained rather than early improvement: R-Zero and G-Zero peak at iteration 2 and decline, whereas J-Zero improves monotonically through iteration 10 in both domains.

• Judge co-evolution generalizes beyond the loop: Co-evolution also improves performance on RM-Bench, an independent reward-model benchmark unrelated to the in-loop preference pairs (Section C).

• Scale generality checks: Besides Qwen3-4B-Base and Qwen3-8B-Base, J-Zero is evaluated with Llama-3.2-3B-Instruct as both Challenger and Solver (same Judge), and with a smaller Skywork-Reward-V2-Llama-3.2-3B Judge paired with Qwen3-4B-Base policies (Section B). The Judge model for the main experiments is Skywork-Reward-V2-Llama-3.1-8B.

Methodology in Plain English

Start from one base model family and no external data, then repeat a three-stage cycle.

Stage 1 — Challenger and Solver play against each other. The Challenger writes a batch of N tasks. For each task the Solver produces M responses, and the Judge scores each task–response pair between 0 and 1. The Challenger is rewarded when the Solver scores poorly (difficulty = 1 minus the mean score), but is penalized for near-duplicate tasks, measured with pairwise BLEU distances and a threshold, and is heavily penalized (a −1 term) for malformed output that fails the <question> format check. The Solver is rewarded for high Judge scores. Both are updated with GRPO.

Stage 2 — Pick the most teachable tasks. The Challenger is frozen, a larger pool of candidate tasks is sampled, and each task is scored by the standard deviation of the Solver's responses to it. The top-K highest-dispersion tasks are kept, on the theory (citing Bae et al., 2026) that expected policy improvement is lower-bounded by reward variance. This mirrors R-Zero's "informative band" selection but uses continuous Judge scores rather than binary verifier accuracy.

Stage 3 — Update the Judge on self-labeled pairs. Two kinds of pairs are built on held-out tasks. Role-asymmetry pairs take the Solver's answer as chosen and the Challenger's answer to its own task as rejected. Subtask-amplification pairs have the Challenger decompose a task, the Solver answer each subtask in the context of the original task, and the Challenger compose the partial solutions; the composed answer is chosen and a one-shot Solver answer is rejected. The Judge is trained with the Bradley–Terry loss on the union of these sets. Crucially, the labels come from how the responses were produced, not from the Judge's scores, which avoids the circularity of self-rewarding with the Judge's own rankings.

Each iteration uses 5 Challenger steps, 15 Solver steps, and 8 Judge steps, with equal proportions of the two pair types in Judge training, implemented on the verl framework across four NVIDIA B200 GPUs and four NVIDIA H200 GPUs. Training stops when the average score in either domain begins to drop, and the checkpoint before that decline is selected as best.

Why This Matters

The paper reframes the evaluator as a trainable component of self-evolution rather than a fixed referee: a self-evolving model, in the authors' framing, can improve only as far as its evaluator can see. This matters because most human supervision is expensive, and most real tasks — writing, advice, planning — have no checkable ground truth.

Potential real-world applications:

• Open-ended writing and content assistants, where quality is defined by human preference rather than a correct answer, and where the paper's AlpacaEval 2.0, Arena-Hard-v2.0, and EQ-Bench Creative Writing v3 evaluations are proxies.
• Instruction-following systems for business communication, planning, personal advice, and recommendations — the task categories the paper lists for AlpacaEval 2.0.
• Mathematical and analytical reasoning tools for domains such as competition mathematics and general knowledge QA, covered by the GSM8K, MATH500, Minerva, OlympiadBench, AMC23, AIME24, AIME25, MMLU-Pro, SuperGPQA, and BBH evaluations.
• Continual or lifelong learning pipelines where a deployed model must keep improving from its own interaction data without waiting for new human-labeled datasets.

Industry relevance: The method requires no external seed corpus, no ground-truth labels, and no human annotators, which lowers the data-acquisition cost of post-training. The reliance on a learned reward model is already standard practice for LLM post-training in unverifiable domains, so the contribution is an extension of an established pipeline rather than a replacement of it. The paper also notes that all experiments ran on eight GPUs total (four B200 and four H200), and that its largest policies are 8B parameters — a scale within reach of well-resourced labs rather than only frontier-scale training runs.

Future Directions

• Scaling beyond 8B. The limitations section notes that compute constraints restricted Challenger and Solver policies to at most 8B parameters with an 8B Judge, leaving larger scales untested.

• Post-trained reasoning models. Only base models were used; models that emit long chains of thought remain untested.

• Generative Judges. The current Judge is a classifier-based discriminative reward model initialized from an off-the-shelf checkpoint and trained with the Bradley–Terry loss, while the Challenger and Solver share a single generative initialization. A generative Judge (for example, LLM-as-a-judge) would let one base model instantiate all three roles, and its critiques could serve as richer in-loop supervision — the open question is how to make such a Judge co-adapt inside the self-play loop.

• Understanding why the baselines collapse. The paper shows R-Zero and G-Zero peak at iteration 2 while J-Zero improves through iteration 10, and locates the divergence point where the Solver reaches a fixed Judge's evaluation ceiling. Extending the same analysis to other zero-data self-play frameworks and reward-model designs is a natural follow-up.

Target Audience

Researchers and engineers working on self-improving or self-play language models, reinforcement learning from AI feedback, and reward-model training, especially those focused on unverifiable or open-ended domains where no verifier exists. It is also relevant to practitioners building post-training pipelines who need to reduce dependence on human-annotated preference data, and to readers tracking the debate over whether a model's own evaluator can serve as a durable learning signal.

Authors’ abstract

Self-evolving language models have recently emerged as a promising path toward superintelligence, with the advantage of reducing the cost of human supervision. While considerable progress has been made in verifiable domains, self-evolution in unverifiable domains remains less explored. We propose Judge co-adaptation from Zero data (J-Zero), a unified Challenger--Solver--Judge self-evolution framework that supports self-improvement across both domains. The Challenger and Solver co-evolve through an adversarial interaction: the Challenger generates increasingly difficult tasks, while the Solver learns to produce higher-quality responses to them. In parallel, the Judge co-adapts using preference pairs whose ordering is known in advance from how each response was produced, i.e., the Solver's answer over the Challenger's, and the Solver's decomposed-and-recombined answer over its one-shot answer, rather than from the Judge's own scores. J-Zero outperforms the baselines by an average of 4.2 points on verifiable and 8.0 points on unverifiable domains, and continues to improve through at least ten iterations, whereas the baselines degrade after two. Further analysis identifies Judge co-adaptation as the key driver of this sustained improvement.