Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents

AI summary

Overview

Research area: Natural Language Processing / tool-use agents and training-data construction for LLM agents (supervised fine-tuning, trajectory synthesis, agentic benchmarks).

Technical level: Advanced. The paper formalizes supervision as a decision-level mask over agent-visible histories, defines support/deficit objectives, and reports propositions and matched ablations; the prose is readable, but the method section assumes familiarity with SFT, imitation learning, and agent evaluation.

Scope: Trajectory2Gym is a three-stage data-construction and training recipe that synthesizes under-supported compositional tool-use interactions, certifies supervision only for decisions justified by the agent-visible history, and allocates that experience across learner scales — evaluated on Tau1, Tau2, and BFCL with five Qwen3 model sizes.

What This Paper Is About

Most tool-use agents are trained by collecting successful trajectories and imitating them, but the authors argue that success only certifies that an episode ended well — not that it teaches useful cross-step dependencies, and not that each imitated action was justified by what the agent could actually see at decision time. They identify two coupled requirements, compositional coverage (the data must exercise observation–action dependencies with repeated support) and decision grounding (each supervised action must follow from the learner-visible interaction history), and build a pipeline that constructs training interactions around both. The goal is to reframe trajectory generation from "collecting successes" into deliberately building learnable interaction experience.

Key Contributions

1. Problem diagnosis and formalization. The authors identify compositional coverage and decision grounding as coupled requirements and formalize them as a decision-level supervision mask over agent-visible histories (a mask m_t selecting which assistant decisions are trained on, while the full visible interaction remains context).
2. Compositional interaction synthesis. A generator and teacher construct and execute tasks in native environments around under-supported capability paths (ordered sequences of semantic actions, observation outcomes, recovery, branching, handoff, termination, and horizon), with only certified structures updating the support inventory.
3. Grounded supervision. A per-decision certification procedure separating native execution validity e_t from decision-time grounding v_t, combined with workflow-order checking, so that rejected calls and their authentic errors stay in context but receive no imitation loss.
4. Structural-support learning with parent-normalized SFT. A selector that allocates repeated support per canonical path at a scale-dependent depth, plus a loss that weights each semantic trajectory equally regardless of how many rendered training examples it produces.

Main Findings

• Every scale improves on all three benchmarks. With plain supervised fine-tuning from each public initialization, Trajectory2Gym raises the equal-weight three-benchmark mean by 16.9–29.8 percentage points across Qwen3 1.7B, 4B, 8B, 14B, and 30B-A3B.
• Tau1/Tau2 gains are large and paired. The Tau1+Tau2 mean gain is 20.04 (1.7B), 35.00 (4B), 36.21 (8B), 40.92 (14B), and 39.11 (30B-A3B) — the smallest at 1.7B, and between 35 and 41 points from 4B upward.
• Smaller post-trained models beat much larger initializations. The post-trained 4B model reaches 69.17 on Tau2 versus 28.33 for the public 30B-A3B initialization; the post-trained 30B-A3B reaches 84.17 on Tau2, within 1.2 points of closed-source reference averages of 84.76 and 85.31.
• Broad rather than narrow gains. Across five domains and five scales, 24 of 25 paired comparisons improve; the exception is Tau2 Airline at 1.7B, unchanged at 20.00. The largest movement is Tau2 Telecom at 30B-A3B, from 12.50 to 95.00.
• BFCL gains concentrate in multi-turn use. Overall improves at every scale, driven by Multi-turn Base, which rises by 7 to 41 points; every post-trained model from 4B upward exceeds the reported GPT-5.5 Multi-turn Base score of 60.00, and the 30B-A3B model posts 80.93 Overall against 79.97 reported for Gemini-3.0-Pro under its source protocol.
• One consistent regression. Live Simple drops 6 to 13 points at four of five scales, which the authors hypothesize reflects a shift toward information gathering on queries expecting one immediate call.
• Function calling does not transfer to stateful tasks. EnvFactory-8B nearly matches the 8B Trajectory2Gym model on BFCL Overall (77.55 versus 78.62) and ties on Multi-turn Base (68.00), yet trails it by 47.5 points on Tau2.
• Best evaluated checkpoint mean at 8B. The 8B model's three-benchmark equal-weight mean is 67.97, against at most 57.17 for any other evaluated checkpoint.
• All four design choices help in matched 8B ablations. Targeted expansion +3.55, decision grounding +2.75, structural selection +2.32, and parent normalization +8.04 (the largest isolated gain).
• Structural selection covers more structures at fixed budget. At 14,819 parents with equal target tokens, structural selection covers 942 more canonical paths than random selection.
• More support depth is not automatically better. In the depth sweep, the three-benchmark mean is highest at depth 2 (72.28) versus depth 3 (70.55) and depth 4 (69.44), with the decline concentrated in Tau2 (82.50 to 75.00) while BFCL stays comparatively stable.
• The starting pool is badly skewed. Of 34,772 certified candidates, 40.9% contain at most two supervised decisions and only 10.0% contain a certified decision after an authentic error.

Methodology in Plain English

The pipeline has three coupled stages.

1. Synthesize the interactions that are missing. Rather than sampling whatever environments happen to produce, the authors define a shared vocabulary of capability "atoms" (semantic actions and arguments, observation outcomes, recovery, branching, handoff, termination, horizon) and compose them into ordered paths. For a target path, a generator creates a user request and a compatible initial state, and a teacher model executes the task in the real environment. The construction deliberately sets up prerequisites and information sources (for instance, a task that identifies the customer but requires a lookup for a transaction identifier, or a variant that creates a discoverable native rejection). A path only counts toward the support inventory if its action and observation dependencies actually appear in the executed log, so a bad generator or teacher proposal surfaces as an uncertified structure instead of a training target.

2. Decide which decisions are actually learnable. Each executed log is converted into what a student agent would see through the native interface — teacher hints and private events are stripped. Each assistant decision is then checked twice: whether it executed natively, and whether it is grounded, meaning the tools were available and identifier-like arguments trace back to information visible earlier (a discoverable tool must first appear in a visible retrieval result). An additional check verifies that the task's ordered requirements appear as an ordered subsequence of the relevant operations. A rejected call and its genuine error message stay in the visible context but get no loss; the grounded recovery that follows does.

3. Allocate support and weight it fairly. For each model size, the selector starts from a fixed certified core, adds one completing exemplar per training task, closes uncovered atoms and paths, and then adds trajectories that reduce the remaining per-path deficit toward a target depth (1, 2, 3, 4, and 8 distinct trajectories per feasible path from 1.7B to 30B-A3B). Because one semantic interaction can render into several training examples, standard per-example averaging would let long multi-turn interactions dominate the loss; parent-normalized SFT instead gives each semantic trajectory equal weight (the paper's illustration: two interactions rendering into three and one example get 3/4 and 1/4 under row averaging, but 1/2 each under parent normalization).

Training setup. One epoch of full-parameter SFT on each Qwen3 initialization, global batch size of eight rendered examples, bfloat16, AdamW with weight decay 0.01, cosine schedule with ten warmup steps, max gradient norm 1.0, and no preference learning, process rewards, or reinforcement learning. Only the learning rate varies by scale: 1.1e-5 at 1.7B, 7e-6 at 4B and 8B, and 3.7e-6 at 14B and 30B-A3B. The 1.7B–14B learners draw from a 34,772-interaction, 2,515-task corpus; 30B-A3B draws from an expanded 35,819-interaction, 3,602-task corpus. Selected views contain 11,583, 13,986, 14,819, 15,805, and 19,937 interactions, corresponding to roughly 13.0M, 18.0M, 19.4M, 20.6M, and 23.9M supervised target tokens. Benchmarks are Tau1 (115 Retail, 50 Airline), Tau2 (40 Retail, 20 Airline, 40 Telecom), and BFCL (1,718 examples).

Why This Matters

Impact on research. The paper argues that outcome filtering is a weak proxy for supervision quality and supplies an alternative: certify targets one decision at a time against the learner-visible history. Its diagnostic statistics (40.9% of certified candidates with at most two supervised decisions, 10.0% with a grounded decision after an authentic error) give a concrete way to audit trajectory corpora. The depth sweep further cautions that "more data" and "better-structured data" are different interventions, and that support is an allocation problem rather than a quantity to maximize.

Real-world applications.

• Customer-service agents that must cancel, exchange, or update multiple orders in a required order while respecting state changes (the paper's Tau2 Retail scenario).
• Travel and booking agents where returned identifiers gate downstream insurance, invoice, or support operations.
• Telecom and device-support agents where the user's reported device state and connection speed steer the recovery path.
• Enterprise tool/MCP ecosystem agents that must gather information before acting and recover from failed lookups (for example, an email lookup failing and the agent re-identifying the user by name and ZIP code).

Industry relevance. Every organization building agents on internal APIs faces the same data question: what to imitate. This recipe is a data-construction and SFT-only procedure, so it does not require an RL infrastructure or an environment farm to apply, and the reported result that a post-trained 4B model beats a public 30B-A3B initialization on Tau2 (69.17 versus 28.33) speaks directly to cost-per-capability decisions. The paper is explicit that its external checkpoint comparisons are references rather than matched or equal-compute controls, so cross-method claims should be read with that caveat.

Future Directions

• Extending decision-level certification to on-policy rollouts. The authors name this as the natural next step, which would let the same eligibility test apply to the model's own trajectories rather than teacher executions.
• Porting the recipe to web, software, and terminal agents, beyond the retail, airline, telecom, and function-calling settings studied here.
• Separating path support from supervision volume. The depth sweep confounds repetition with token count, so the paper leaves open how to isolate the effect of structural support from raw data scaling.
• Turning the sensitivity result into a principled schedule. Depth was set by a predeclared per-scale schedule and the shallowest depth tested scored highest on the three-benchmark mean, so how to choose depth per scale — and whether shallow depth holds under other budgets — remains an open question.

Target Audience

Researchers and engineers working on tool-use and agentic LLM systems, especially those responsible for building or curating supervised fine-tuning data; practitioners comparing data-centric fixes against reinforcement learning for agent training; and readers interested in evaluation methodology for agents, given the paper's emphasis on paired comparisons and its explicit distinction between matched ablations and cross-method reference checkpoints. Readers wanting a gentle introduction may find the formal problem formulation and propositions demanding, but the three-stage intuition and the qualitative trace comparisons are accessible without it.

Authors’ abstract

Successful tool-use trajectories are not necessarily useful supervision. Even successful interactions may underrepresent multi-step dependencies or contain actions that are unjustified by the information visible to the agent at decision time. We identify two coupled requirements for effective trajectory supervision: compositional coverage, so that important observation-action dependencies are represented, and decision grounding, so that each supervised action is supported by the learner-visible interaction history. We introduce Trajectory2Gym, which constructs executable interactions around under-supported compositional structures and derives supervision only from grounded decisions in their realized executions. Structural-support selection allocates the resulting experience, while parent-normalized supervised fine-tuning preserves the semantic interaction as the unit of training weight. This reframes trajectory generation as the deliberate construction of learnable interaction experience, rather than the collection of successful trajectories. Across five Qwen3 scales, Trajectory2Gym improves Tau1, Tau2, and BFCL at every tested scale, increasing the equal-weight three-benchmark mean by 16.9-29.8 percentage points. The gains are strongest in stateful and multi-turn interaction, where later decisions depend on earlier observations and feedback. Our results suggest that scaling tool-use agents depends not only on collecting more successful trajectories, but on constructing interactions whose dependencies are represented and whose decisions are learnable from the agent's visible history.