Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms

AI summary

Overview

• Research area: Agentic reinforcement learning for language-model agents, specifically the automatic construction of interactive training environments with verifiable reward signals, drawing the underlying mechanisms from operations research and mathematical optimization.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning (policy optimization, group-relative advantages, KL divergence), mathematical programming families (dynamic, linear, and quadratic programming, routing, knapsack, scheduling), and partially observable decision processes.
• Scope: The paper proposes VHD-Play, a mechanism-first pipeline that samples and solves a mathematical model first, then renders the solved mechanism into a stateful, tool-based agentic environment whose reward is inherited from the same solution, and it evaluates whether training agents in 3,300 such environments transfers to unseen mechanisms and external benchmarks.

What This Paper Is About

Language-model agents need to operate inside processes where state changes as they act, but existing pipelines usually build an environment before defining how it will be scored, leaving dynamics and evaluation to be reconciled afterward. VHD-Play reverses this order: it samples a mathematical model, solves it to obtain reference outcomes, and only then realizes the environment and its outcome rule from that same solved model. The goal is to produce many diverse, verifiable agentic environments cheaply while giving training a dependable, non-learned reward signal.

Key Contributions

1. Mechanism-first construction. The paper introduces an ordering in which a sampled mathematical model M(θ) is solved to yield reference outcomes z_θ, which in turn fix the outcome rule ℛ_θ before the executable dynamics D(θ) are realized and wrapped into the agent-facing environment E(θ). This makes the environment and its evaluator descend from a single source rather than being aligned post hoc.
2. A corpus-grounded generation and replay pipeline. A frozen language-model setter receives the complete parameter draw and one independently sampled passage from a corpus of real-world documents, and generates the scenario, relational database, domain-specific tool schemas and bodies, and player instruction. With automatic admission, the pipeline produces 3,300 admitted environments, each with at least 10 tools, at approximately $0.01–$0.03 per admitted record.
3. Training with solver-derived rewards. Using group-relative policy optimization (GRPO) on groups of rollouts, training Qwen3.6-35B-A3B raises its mean agentic score across five optimization families from 0.204 to 0.815, with gains on held-out instances from all three training families and on eight unseen mechanism families, plus transfer to external benchmarks.
4. A diagnostic separating problem solving from stateful interaction. The same problem is presented in written-out form (F), informed-agentic form with parameters revealed (I), and agentic form where parameters must be acquired through interaction (A), showing that most of the learnable gap lies in stateful interaction rather than in solving the underlying problem.

Main Findings

• Large gains on generated mechanisms. Held-out training families gain +0.56 on average at the agentic level; near-OOD families (unseen optimization families) gain +0.63; the two far-OOD groups, which change decision structure or restrict probing, gain +0.24 and +0.16. Training improves every evaluated family.
• Family macro scores move across all three presentations. The family macro goes from 0.962 to 0.992 in written-out form (F), 0.231 to 0.875 in informed-agentic form (I), and 0.204 to 0.815 in agentic form (A). Training changes (F, I, A) by (+0.030, +0.644, +0.611), closing 84% of the parameter-revealed stateful gap and 77% of the full written-out-to-agentic gap.
• The base checkpoint is near ceiling in written-out form but not in stateful form. It scores 0.962 in F but falls to 0.231 in I even though parameters remain available, and to 0.204 in A where parameters must be acquired through the interface.
• The gain is not merely more code-tool use. A fixed-stratum analysis attributes +0.030 of a +0.443 agentic gain to increased adoption across the eight families with both tool-use strata, leaving +0.413 within fixed strata. The policy shift is D_KL(π_trained ∥ π_Base) = 0.089 nats per assistant token on held-out trajectory states.
• Transfer to general function calling. The unweighted mean over ten interaction-focused BFCL V4 cells (Multi-Turn, Agentic, and Hallucination) rises from 61.25 to 64.08, a gain of 2.84 points.
• Transfer to constrained planning. TravelBench plan quality increases from 0.700 to 0.794, which is below the Qwen3.7-Max reference of 0.891.
• Transfer to long-horizon business simulation. On E-Commerce Bench, across five reported storefront runs per arm, the base checkpoint completes four and goes bankrupt once, while the trained 35B-A3B checkpoint completes all five without bankruptcy over 1,147–1,825 assistant turns. Mean ending balance rises from 54,294 to 182,844 (3.4 times) and exceeds the Qwen3.7-Max reference of 165,224.
• Environment–policy co-scaling. Over four knapsack configurations scaling from 6 items over 5 periods to 11 items over 11 periods, gains are +0.412, +0.551, +0.675, and +0.861. At 6×5 the base and trained policies score 0.432 and 0.843; at 11×11 they score 0.085 and 0.946. The two larger configurations average +0.768 versus +0.482 for the two smaller ones.
• Trajectory-level failure modes are identified. In a seven-period allocation task, the base policy writes "For each period, find best combination," spends 27 of 29 shared-budget units in the first three periods, and ends with score 0, while the trained policy plans across all seven periods and reaches score 1. In an observation case, the base policy probes only four of eight items before acting, while the trained policy reveals all eight before planning.
• Admission and coverage. Every sampled environment in the held-out audit executed successfully, repeated outcomes agreed, and realized outcomes remained within their stored references. Scenario seeds draw from 28 topical domains with normalized entropy 0.997, with no admitted domain above 4.7%. Episodes average 59.7 assistant turns.
• Reward comes from arithmetic, not a learned judge. The reward is r(π; θ) = clip_[0,1]((u(π; θ) − u_0(θ)) / (u*(θ) − u_0(θ))), where u*(θ) is the optimal value and u_0(θ) is the value of a fixed default policy, both computed by closed-form arithmetic, enumeration, dynamic programming, or a numerical solver before the environment existed.

Methodology in Plain English

The pipeline starts by drawing parameters θ from a mechanism family (for example, an inventory control, routing, knapsack, linear programming, quadratic programming, or scheduling model). A family-specific solver computes two reference numbers without any language model: the optimal value of the model and the value of a fixed default policy. Those two numbers define the outcome scale, mapping the default policy to a reward of 0 and the full-information optimum to a reward of 1.

Only after those references are fixed does construction begin. A frozen language-model setter receives the complete parameter draw plus one sampled passage from a corpus of real-world documents. The passage supplies entities, relations, and domain language; the mathematical model supplies the decision process; the solved references supply the evaluation. The setter renders the mechanism as a scenario with a relational database, domain-specific tools, and a player instruction.

The environment wraps the model's executable dynamics with an interface that controls what the agent can see. The agent never reads the latent parameters or the reference values directly; it must use catalogue, probe, decision, or clock operations to reveal state. Information-gathering actions (probes) and decision actions share the same turn budget T. Each turn is one answered interface call.

Candidates are admitted only if they execute and if their realized behavior agrees with the precomputed construction; failures are regenerated or rejected. Training then uses the fixed episode-level reward with GRPO over groups of rollouts from the same instance. The paper then compares the same problems in written-out form, informed-agentic form, and agentic form to isolate where the difficulty lies, and tests transfer to benchmarks that share neither instances nor interfaces with the generated environments.

Why This Matters

Impact on research. The paper reframes agentic environment construction as a problem of ordering: if the outcome rule is fixed before the environment is built, the difficulty of validating a generated evaluator disappears, because no evaluator is learned or judged. It connects the scalable verifiability of written-out mathematics to stateful, trajectory-level interaction, and it provides a diagnostic that separates mathematical problem-solving ability from agentic policy ability — a distinction that matters when interpreting benchmark scores.

Real-world applications (as referenced in the paper):

• Inventory and replenishment control. The paper's worked example involves per-product demand, shared capacity, and joint ordering fees, where stock bought early occupies capacity later and no action recovers a lost sale; generated settings include hardware replenishment.
• Autonomous e-commerce operations. E-Commerce Bench runs a 365-day business with multiple storefronts, customer orders, returns, market events, and delayed cash flow.
• Travel planning. TravelBench requires agents to assemble multi-day itineraries satisfying coupled timing, budget, commonsense, and personalized constraints.
• General tool use in production workflows. BFCL V4 multi-turn and agentic cells target stateful function calling, and the generated domains include education logistics and adaptive-sports planning.
• Software repair and customer-service workflows. The introduction cites agents repairing software through repeated iterations and resolving customer requests across tools.

Industry relevance. The cost figures matter commercially: 3,300 admitted environments at roughly $0.01–$0.03 each means environment supply can scale with parameter resampling rather than with specialist authoring. The finding that a frozen 35B setter can realize new draws, and that a scale-matched training setup retains gains as mechanism size and horizon grow, suggests a training substrate that can expand alongside policy capability rather than being rebuilt.

Future Directions

• Whether the policy explicitly reconstructs the underlying model. The paper states that the experiments establish feasibility and learnability of the construction over the evaluated families, not that the policy explicitly reconstructs the underlying model.
• Whether operations research is sufficient or merely convenient. The authors note that any formal mechanism supporting parameterized variation, executable realization, controlled information asymmetry, and an outcome standard fixed before interaction could play a similar role, and they do not claim operations research is the only substrate for this transition.
• Co-scaling to larger mechanisms and longer horizons. The setter and frontier results suggest the substrate can expand with policy competence, but the paper reports the sweep over four knapsack configurations, and larger or structurally different families are left open.
• Validation beyond the evaluated families and benchmarks. Eleven mechanism families are evaluated, three are used for training, and external transfer is measured on BFCL V4, TravelBench, and E-Commerce Bench; broader task settings remain untested.

Target Audience

Researchers and engineers working on reinforcement learning for language-model agents, especially those building or scaling agentic training environments, reward design, or tool-use benchmarks. It also suits practitioners in operations research or decision science who want to see established optimization models used as a source of agentic training data, and evaluation researchers interested in separating problem-solving competence from stateful policy competence. Readers need comfort with reinforcement learning terminology and mathematical programming families.

Authors’ abstract

Language-model agents increasingly face long-horizon tasks with evolving state, interdependent decisions, and delayed outcomes. Scaling their training requires diverse agentic environments, dependable outcome signals, and low extension cost. Existing generation pipelines commonly construct an environment before defining its outcome rule or annotating its trajectories, leaving dynamics and evaluation to be aligned post hoc. VHD-Play reverses this dependency by sampling and solving a mathematical model before a corpus-grounded setter renders its decision process as stateful tools. The executable dynamics and trajectory-scoring reference are inherited from the same solved model. The pipeline produces 3,300 diverse agentic environments at a cost of a few cents each. Training Qwen3.6-35B-A3B on three families raises its mean agentic score from 0.204 to 0.815 in a five-family diagnostic. Gains also appear on held-out instances from all three training families and eight unseen mechanism families, then extend beyond the generated substrate to external benchmarks for general function calling, travel planning, and 365-day e-commerce. On E-Commerce Bench, the trained checkpoint completes every run without bankruptcy and exceeds Qwen3.7-Max. We compare written-out problems with stateful versions that reveal or hide their parameters. The comparison shows that most of the learnable gap lies in stateful interaction rather than underlying problem solving. A frozen 35B setter realizes larger environments, and scale-matched training retains gains as mechanism size and horizon grow, indicating the potential for an evolving training substrate.