RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

AI summary

Overview

Research area: Reinforcement learning (RL) for large language models, specifically RL with verifiable rewards (RLVR) using procedurally generated training problems.

Technical level: Intermediate — the paper assumes familiarity with RL post-training for language models, reward-based policy optimization, and standard reasoning benchmarks.

Scope: The paper introduces RLVE (Reinforcement Learning with Adaptive Verifiable Environments), a method that lets each training environment automatically raise or lower problem difficulty as the model improves, and releases RLVE-Gym, a manually engineered suite of 400 such environments, to show that scaling the number of environments (rather than the number of problems) drives generalizable reasoning gains.

What This Paper Is About

RL training for language models tends to stall because the training data distribution is fixed: problems that were once hard become trivially easy, and problems that are too hard give no useful learning signal, so the model runs out of things to learn. Collecting enough problems with verified answers at the right difficulty is also expensive. RLVE addresses both problems by having each environment procedurally generate unbounded problems at a configurable difficulty level and automatically shift that difficulty upward as the policy model's success rate rises, then pairing this with a large suite of 400 such environments so the model learns broadly transferable reasoning rather than a narrow skill.

Key Contributions

1. The RLVE method for adaptive verifiable environments. Each environment is defined as a tuple of an input template, a problem generator, and an algorithmic verifier, plus an integer difficulty level. RLVE tracks how many rollouts are correct at the current hardest difficulty level and increments that level once accuracy crosses a threshold, using a sliding window (d_Δ > 1) to keep easy problems from dominating the mix.

2. RLVE-Gym, a suite of 400 verifiable environments. Built through manual "environment engineering," drawing on six representative sources: programming competition problems, mathematical operations, optimization problems, classical algorithmic problems, logical puzzles, and NP-complete problems. Each environment both generates problems and verifies outputs algorithmically.

3. Evidence for environment scaling. Systematic experiments showing that expanding the collection of training environments (1 → 4 → 16 → 256 environments) consistently improves accuracy on 50 held-out environments never seen in training.

4. Demonstration that RLVE scales RL beyond saturation and under compute constraints. Two experiments: one continuing training from the already-saturated 1.5B RL model ProRL-1.5B-v2, and one from the SFT model OpenThinker3-1.5B compared against the RLVR dataset DeepMath-103K.

Main Findings

• Adaptive difficulty beats static difficulty. Training Qwen2.5-7B-Base on Sorting and Multiplication showed that with a low static difficulty upper bound, the effective prompt ratio (percentage of prompts retained after dynamic sampling whose rollouts yield non-identical rewards) eventually drops to zero, meaning learning stalls. With a high static upper bound, the ratio stays nonzero but is substantially below the adaptive case, indicating most problems are inappropriately hard. Adaptive difficulty maintained the highest effective prompt ratio and better in-distribution and out-of-distribution performance.

• Static difficulty at [0,20] had an oracle advantage but still did not win. The static distribution d ~ [0,20] matched the in-distribution evaluation distribution, yet RLVE achieved comparable or superior in-distribution performance without that advantage.

• Adaptive environments reach a wide spread of difficulty levels. At step 400 of joint training on 256 environments, the upper-bound difficulty levels h_π^(i) reached by adaptive environments ranged from 0 to 12. Even when a static range of [0,20] was chosen to cover all of these distributions, training on static environments was consistently outperformed by RLVE.

• Environment scaling improves generalization. Training the same four models on collections of 1, 4, 16, and 256 environments, with each larger collection strictly containing the smaller ones, produced consistently better accuracy on the 50 held-out environments as the collection grew.

• RLVE scales past data saturation. Starting from ProRL-1.5B-v2 (originally trained from R1-Distill-Qwen-1.5B using over 20,000 H100 GPU hours of RLVR to saturation on roughly 136,000 problems), RLVE improved the average across six reasoning benchmarks by an absolute 3.37% within approximately 1,100 H100 GPU hours. Continuing the original RLVR training on the same ProRL dataset yielded only a 0.49% absolute gain — 7× smaller — even after more than three times the compute (3,600 H100 GPU hours).

• RLVE beats a strong curated RLVR dataset under identical compute. Both starting from OpenThinker3-1.5B, RLVE outperformed DeepMath-103K by about 2% absolute improvement. RLVE was stronger on non-mathematical benchmarks (LiveCodeBench and BBEH) and most mathematical benchmarks (OMEGA-500 and OlympiadBench), and comparable on AIME 2024/2025 with peak performance exceeding DeepMath-103K by roughly one point.

• Cost efficiency. Constructing RLVE-Gym was substantially cheaper than DeepMath-103K, which required roughly $138,000 USD and 127,000 GPU hours to build. RLVE also requires no benchmark-specific data, whereas DeepMath-103K was explicitly designed for mathematical reasoning.

• Environment advantages over imitation. Verifiers can be far cheaper than solvers: an environment can execute programs while the model cannot, and for problems with asymmetric complexity (Sudoku, SAT, Hamiltonian path), verification is easy even when solving is intractable. Environments were also designed as pedagogical tools — for example, training the model to sort an array by hand rather than calling a sorting program — so the model learns the reasoning process, not the outcome.

Methodology in Plain English

The researchers built environments that each follow the same recipe: a template describing the problem, a generator that fills in random parameters, and a checker that grades the model's output programmatically. Every environment has a dial — an integer difficulty level — that controls how hard the generated problems are. Raising the dial always makes problems at least as hard as before; for example, a higher level means longer arrays to sort or larger expression trees to integrate.

During training, the system samples a difficulty level uniformly from a window and generates a problem at that level. It counts how many rollouts got the right answer at the window's hardest level. Once enough rollouts have been collected, if accuracy meets a threshold (the paper's figure illustration uses 90%), the upper bound of the window goes up by one, the counters reset, and training continues at harder problems. A sliding window caps how many levels stay in play so the model keeps seeing the frontier rather than replaying easy problems.

For joint training across many environments, the system picks an environment uniformly at random for each new problem and keeps a separate difficulty window and accuracy counter for each one. Any RL algorithm that works with verifiable rewards can be plugged in; the authors used DAPO, a variant of GRPO, with dynamic sampling that discards prompts where all rollouts receive identical rewards. Training problems came from a randomly chosen set of 256 environments, while evaluation used 50 held-out environments (50 problems each, 2,500 problems total) never seen during training.

Why This Matters

Impact on research. The paper reframes RL data scaling: because a single environment can generate unlimited problems, adding more problems stops helping, but adding more environments keeps improving performance on unseen tasks. It also shows that RLVE can push a model past the point where a large, expensive, curated RLVR dataset has saturated it, which challenges the assumption that RL gains are bounded by data collection.

Real-world applications:

• Post-training pipelines for small reasoning models, where the paper shows substantial gains at the 1.5B scale from SFT and RL checkpoints.
• Code and mathematical reasoning assistants, since RLVE-Gym draws heavily on programming competition and mathematical operation environments.
• Domain-specific training where labeled ground truth is scarce, because environments supply their own algorithmic verifiers instead of requiring human-annotated answers.
• Cost-sensitive training runs, given that RLVE requires no benchmark-specific data and RLVE-Gym was far cheaper to build than DeepMath-103K's roughly $138,000 and 127,000 GPU hours.

Industry relevance. The approach lowers the data-construction barrier for RL post-training and offers a reusable environment suite, which matters for teams that want continued improvement after their curated datasets stop yielding gains. It also raises the profile of "environment engineering" as a discipline alongside feature, data, and prompt engineering.

Future Directions

• Adaptive non-verifiable environments. Extending the approach to domains such as creative writing or deep research, where rewards cannot be defined algorithmically, is described as an open problem because these tasks lack clear structure for controlling difficulty.
• Model-based automatic environment engineering. The authors tried using frontier LMs to build environments automatically and found it nontrivial to preserve quality — specifically unambiguity of input templates, reliability and efficiency of generators in producing valid and diverse problems, and robustness of verifiers against diverse model outputs. RLVE-Gym is offered as a prototype seed for such work.
• Per-environment difficulty calibration. The results suggest each environment's difficulty must be tuned to the policy individually, since each defines its own notion of difficulty; how to do this at scale without the adaptive mechanism remains a design question.
• Understanding why environment diversity helps. The paper draws an analogy to classical RL and to SFT and embedding learning, where expanding the set of tasks matters more than increasing data volume, but does not provide a mechanistic explanation.

Target Audience

Researchers and engineers working on RL post-training for language models, especially those hitting saturation on curated RLVR datasets; practitioners building verifiable-reward or procedural task-generation pipelines; and readers interested in curriculum learning and adaptive difficulty. The paper is best suited to those with some background in RL for language models, since it assumes familiarity with terms like rollout, policy, reward signal, and dynamic sampling.

Authors’ abstract

We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model's capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM's original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We release our code publicly.