ActionOPD: Distilling Decisions, Not Tokens

AI summary

Overview

• Research area: Knowledge distillation and policy optimization for large-language-model agents that act through external tools and user-facing messages.
• Technical level: Advanced. The method is derived as a REINFORCE-form policy gradient with a PPO-style clipped surrogate, and the paper assumes familiarity with on-policy distillation, token-level KL/divergence objectives, and group-relative advantage estimation.
• Scope: The paper proposes ActionOPD, an action-level (rather than token-level) on-policy distillation method that trains small agent students to agree with a black-box teacher's decisions at student-encountered states, and evaluates it on four out-of-distribution agent benchmarks plus a τ²-bench comparison and ablations.

What This Paper Is About

Distilling a strong but expensive frontier agent into a smaller model requires deciding where the teacher should supervise the student and at what granularity. Offline distillation fits teacher-written sequences and can suffer state-distribution mismatch, while token-level on-policy distillation (OPD) fixes that mismatch but typically needs teacher token probabilities and a compatible tokenizer. ActionOPD instead treats the semantic action (a tool call or a communicative response) as the unit of supervision, rewarding student responses for agreeing with teacher actions sampled at the same student-visited state, with only generation access to the teacher required.

Key Contributions

1. ActionOPD method: An action-level on-policy distillation method that decouples teacher decision supervision from student token realization, enabling distillation from black-box teachers without teacher token probabilities or tokenizer alignment.
2. Cross-family and scaling evidence: Applicability across teacher families plus empirical scaling with student size and teacher capability; all tested configurations improve the four-benchmark average over their corresponding base models by 3.3–6.9 percentage points.
3. Competitive baseline comparison: On τ²-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD while requiring only generation access to the teacher.
4. Self-ActionOPD exploration: A same-model teacher given privileged task context provides useful action-level supervision, suggesting a route toward agent self-improvement through action-level self-distillation.

Main Findings

• Cross-benchmark transfer: Training on Agent World Model (AWM) and EnvScaler (6,697 curated tasks in total) improved the four-benchmark average over base models by 3.3–6.9 percentage points for all four trained configurations, on benchmarks excluded from training.
• Per-student results (Table 1): Qwen3-4B base average 30.8 rose to 35.9 with Qwen3.7-Flash (+5.1) and 37.7 with DeepSeek-V4-Flash (+6.9). Qwen3-8B base average 35.6 rose to 39.0 (+3.3) and 40.0 (+4.4) respectively.
• Teacher reference scores: Direct evaluation of the teachers gave averages of 65.6 (Qwen3.7-Flash) and 71.2 (DeepSeek-V4-Flash), so distilled students remained far below their teachers.
• Scaling with student size: With either teacher, Qwen3-8B reached a higher final average than Qwen3-4B under the same task-start and training-step budgets, using the same candidate count, teacher sample count, and reward formulation.
• Scaling with teacher capability: DeepSeek-V4-Flash had the higher direct-evaluation average and yielded higher student averages at both sizes, and the DeepSeek-to-Qwen pairing demonstrates cross-family distillation without a shared model family or tokenizer.
• Tau distillation comparison (Table 2, Qwen3-4B student, Qwen3-32B teacher, 178 Tau training tasks, evaluated at step 50): KD achieved the highest test average pass⁴ (19.2), while OPD and ActionOPD achieved higher test average pass¹. ActionOPD exceeded OPD's test average pass¹ by 0.4 percentage points but had a lower test average pass⁴. ActionOPD's training-set averages were higher than OPD's on both metrics.
• Ablations (Table 3): ActionOPD outperformed Validity Reward, Binary Reward, and Random Commit variants in average single-trial success on both the training split and 100 held-out tasks (train 37.9/13.7 and test 40.0/14.2 for ActionOPD, versus 30.5/13.2 and 30.8/10.0 for Validity Reward, 33.3/10.5 and 36.9/14.2 for Binary Reward, and 33.2/9.2 and 37.5/12.5 for Random Commit). The largest drop occurred with Validity Reward, which gives identical rewards to all valid candidates and thus no learning signal in all-valid groups.
• Behavioral pattern analysis: Selected cases with at least two additional successes after training showed the distilled student respecting policy constraints, distinguishing information sources from operation targets, and following through on promised actions (for example, Airline 2/4 → 4/4 and Retail 0/4 → 3/4 for the Qwen3-4B student with DeepSeek-V4-Flash).
• Self-ActionOPD (Table 5): After 50 steps with shared Qwen3-4B weights and a teacher given privileged context, test average pass¹/pass⁴ improved over base by 4.8/2.5 percentage points, and single-trial success improved in all three domains, with the student evaluated using public context only.
• Not reported: The paper does not report compute cost, wall-clock or token budgets for training, results at student sizes beyond Qwen3-4B and Qwen3-8B, or measured benefits from increasing K and N in the dispersed-decision regime.

Methodology in Plain English

The method starts from a simple objective: at the states the student actually reaches during a rollout, maximize the probability that the student produces the same semantic action the teacher would produce. A semantic action is the meaning of a tool call (tool name plus canonicalized arguments) or the meaning of a user-facing message, not the exact wording or reasoning text.

To estimate this, the system samples several teacher responses at a state (K = 3) and several student responses at the same state (a group of N = 4), then checks whether each student response semantically matches the teacher samples. Matching is done programmatically for tool names and canonicalized arguments, with a frozen LLM judging remaining argument differences and comparing the intent and factual content of communicative responses; invalid outputs never match. The count of matches produces a reward: −1 for an invalid action, 0 for a valid but unmatched action, and 1 plus a frequency bonus for matched responses, with λ = 0.5 and [A, A, B] earning 1.25 for A and 1 for B.

Rollouts are "state groups": all N candidates at a state are scored against the same teacher multiset, but only the highest-reward action is committed to the environment so the trajectory is teacher-guided, while all retained group candidates are used for training. Learning then uses group-relative advantages (each group's mean reward as a local baseline, normalized by a shared batch-wide standard deviation) inside a PPO-style clipped surrogate over the student's own tokens. Because the derivation lands on a REINFORCE-style policy gradient with semantic agreement as the reward, the student's reasoning tokens are trained but not forced to imitate teacher-written reasoning.

Experiments pair Qwen3-4B and Qwen3-8B students with Qwen3.7-Flash and DeepSeek-V4-Flash teachers in a 2×2 design, training for 100 steps with 64 task starts per step (59 AWM, 5 EnvScaler). Evaluation uses greedy decoding without teacher-guided action selection on BFCL Multi-turn, ToolSandbox's verified-local-only subset, AppWorld Normal and Challenge, and τ²-bench Airline/Retail/Telecom, with the reported average weighting the four benchmarks equally.

Why This Matters

Research impact. The paper reframes agent distillation as matching decisions rather than tokens, which removes the usual requirements of teacher token log-probabilities, tokenizer compatibility, and a shared model family. It connects this objective to existing lines of work on black-box/semantic distillation, privileged self-distillation, and interactive imitation such as DAgger, and it argues that the semantic-action objective is identical to the action-likelihood objective under its definitions.

Real-world applications.

• Customer-service agents, the setting of the τ²-bench Airline, Retail, and Telecom domains, where policy compliance and correct tool execution matter more than wording.
• Enterprise and workflow automation agents, as represented by AppWorld's Normal and Challenge task/scenario goal completion.
• Multi-turn tool-using assistants, as represented by BFCL Multi-turn and ToolSandbox.
• Self-improvement pipelines, where a deployed model with privileged access to user goals or reference actions supervises an earlier copy of itself using only public context.

Industry relevance. Because only generation access to the teacher is required, teams can distill from proprietary API models without obtaining logits, and can choose teachers on decision quality rather than tokenizer compatibility. The gains were measured on benchmarks excluded from training, which is the setting practitioners care about when swapping in a smaller, cheaper model.

Future Directions

• Correctness beyond agreement: Agreement with teacher actions does not guarantee correctness, so teacher errors or suboptimal decisions can become learning targets; the paper flags this as an open problem.
• Better semantic matching: LLM matching of communicative responses and tool-argument differences can produce false positives or negatives and misassign rewards, which motivates more reliable matchers.
• Coverage under dispersed decisions: With fixed-size teacher (K) and student (N) sample sets, long-horizon interactions may contain few semantically matching actions; the paper states that increasing K and N could improve coverage but that the benefit in this regime remains untested.
• Extending scaling analysis: The reported scaling evidence covers two student sizes (Qwen3-4B, Qwen3-8B) and two teachers, leaving wider scaling trends, compute trade-offs, and non-Tau baseline comparisons unexplored.

Target Audience

Researchers and engineers working on LLM agent training, on-policy distillation, and post-training with reinforcement learning, particularly those who need to transfer behavior from closed or black-box teacher models into smaller agents. It is most useful to readers already comfortable with policy-gradient objectives and agent benchmarks; readers seeking an introductory treatment of distillation will find the derivations in Appendices A.1 and A.2 dense.

Authors’ abstract

Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On $τ^2$-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.