DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents

AI summary

Overview

Research area: Natural Language Processing / Large Language Model agents — specifically multi-turn tool-calling agent training, agent distillation, and structured policy optimization.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, behavior cloning, reinforcement learning for LLMs (GRPO-style policy optimization), credit assignment, and graph-based state abstractions.

Scope: The paper introduces DART-SD, a framework that replaces full-trajectory imitation of multi-turn tool-calling agents with topology-aware, localized correction of only the steps where a student model first departs from a success-reachable region.

What This Paper Is About

Training LLM agents to call tools across many turns is usually done by imitating entire expert trajectories, which forces many equally valid interaction orders into one linear sequence and penalizes legitimate exploration. The authors argue that for tasks with order-independent sub-goals, the space of valid solutions is a combinatorial "diamond" lattice, so global imitation causes what they call topological collapse. DART-SD instead builds a graph of interaction states from teacher rollouts, finds the exact point where a student rollout first leaves the region from which success is still empirically reachable, and trains the model only on the recovery steps after that point.

Key Contributions

1. Identification of topological collapse and the Interaction-State Transition Graph (ISTG). The authors diagnose full-trajectory imitation and standard RL as misrepresenting multi-turn tool use as isolated linear paths, and represent execution instead as a directed multigraph whose nodes are cumulative interaction states, capturing the diamond topology created by order-independent exploration.

2. The Critical Topological Breakpoint (CTB) and CTB-guided localized supervision. Student interaction states are projected onto an empirical, budget-filtered success-reachable region; the first transition from projectable to non-projectable defines the CTB. Supervision is then applied only to assistant response steps generated after the CTB and before the final-answer step, with the retained student prefix zero-weighted.

3. A progressive self-distillation paradigm. The student is repeatedly rolled out, its evolving capability boundary is re-detected, and CTB-guided localized supervision is re-applied across iterations, forming a self-paced curriculum that extends valid interaction prefixes.

4. Empirical validation across five benchmarks and two model scales. Experiments with Qwen3-4B and Qwen3-8B backbones on FTRL (in-domain) plus BFCL, ToolHop, τ-bench, and RoTBench (out-of-domain) show consistent gains over distillation and RL baselines, alongside reduced redundant tool calls.

Main Findings

• DART-SD leads on overall average across both backbones. With Qwen3-4B, DART-SD reaches an average of 39.17 versus 37.62 for SFT, 33.66 for FTRL-GRPO, 29.91 for ToolRL, 29.87 for MatchTIR (KM), 28.35 for OPSD, 26.29 for SCoRe-SFT, and 25.19 for the base model. With Qwen3-8B, DART-SD reaches 45.58 versus 41.64 for SFT, 40.33 for FTRL-GRPO, 39.94 for ToolRL, 38.31 for MatchTIR (KM), 34.95 for OPSD, 33.38 for SCoRe-SFT, and 29.60 for the base model.

• In-domain FTRL results with Qwen3-8B: DART-SD obtains Solve-P 42.00, Solve-R 54.13, and Solve-F1 45.66; SFT obtains 38.08, 50.95, and 41.89 respectively.

• Large gains on multi-hop tool dependency. On ToolHop Answer Correctness with Qwen3-4B, DART-SD scores 42.11, compared with 26.23 for MatchTIR (OT), 25.83 for SCoRe-SFT, 20.60 for ToolRL, 20.20 for OPSD and the base model, and 40.50 for SFT.

• The student surpasses its teacher on three benchmarks. Figure 3 reports that DART-SD, initialized from Qwen3-8B, improves on the base model on all five benchmarks and surpasses the teacher on FTRL, ToolHop, and τ-bench, despite having substantially fewer parameters.

• Trajectories get shorter while performance rises. On the FTRL test set, Solve-F1 improves from 40.37 at Iter1 to 45.66 at Iter5, while average tool calls in successful trajectories drop from 4.23 at Iter1 to 3.55 at Iter5. The final model's 3.55 is shorter than the golden references' 4.02.

• The capability boundary shifts later each iteration. The average CTB position in failed training trajectories rises from 0.348 at Iter1 to 1.452 at Iter5 (a change of +1.104 relative to Iter1), meaning the model stays within the recoverable region for longer prefixes. The Multi category moves from 0.500 to 1.953 and Para-Multi from 0.395 to 1.816.

• Thinking mode. Under the thinking setting (Table 4), DART-SD scores FTRL 41.03, BFCL 49.75, and ToolHop AC 46.43, versus MatchTIR (KM) at 37.33, 47.13, and 46.16, FTRL-GRPO at 32.85, 41.50, and 36.72, ToolRL at 26.72, 34.25, and 32.93, and the Qwen3-8B base at 29.74, 40.00, and 42.21.

• General capabilities are preserved and improved. Under a thinking-enabled setting, DART-SD scores 45.29 on IFEval, 50.00 on AIME24, 30.00 on AIME25, and 74.27 on MMLU, an average of 49.89 versus 44.18 for SFT and 43.92 for Qwen3-8B.

• Ablation confirms each component contributes. On FTRL with Qwen3-8B, Solve-F1 rises from 23.48 (base) to 38.10 with self-distillation added, 39.51 with CTB added, 43.93 with progressive SFT added, and 45.66 with the full ISTG-based method.

Methodology in Plain English

The approach has four moving parts.

First, the authors build a map of what "getting work done" looks like for each task. Instead of treating a trajectory as a fixed sequence of calls, they compress tool responses into task-specific "information atoms" — canonical facts. A deterministic stage discards responses that carry no usable data (status signals, empty or placeholder fields, errors), and a semantic stage assigns atoms to the remaining responses, judging all candidate calls of a task jointly. Semantically equivalent responses from different tools share the same atom.

Second, they construct the ISTG: a directed multigraph whose nodes are cumulative interaction states, consisting of the set of information atoms acquired so far plus a record of useless operations since the last information-gaining step. Information-gaining nodes are "main," useless-operation nodes are "aux." Parallel edges are kept. Because information is modeled as a set, two different orderings of independent sub-goals can diverge and reconverge at the same main node — this is the diamond structure.

Third, they define where success is still empirically possible. Using only successful teacher rollouts, they compute a per-task reachability budget from the shortest successful depth plus a task-dependent allowance, capped by a maximum. Nodes on successful rollouts whose remaining distance to the success terminal is within that budget form the success-reachable region. Student states are projected onto this region with type-specific rules: a student main node matches any reachable teacher main node whose information set is contained in the student's; a student auxiliary node matches reachable auxiliary nodes attached to the largest compatible main node and having the same count of useless operations. The Critical Topological Breakpoint is the first step where projection goes from possible to impossible — the earliest point where the student leaves teacher-supported territory. If the student stays projectable yet fails, the terminal state is used as the correction boundary.

Fourth, they train on the recovery. An augmented generator is conditioned on the task, the student prefix retained before the CTB, and privileged teacher references (randomly sampled successful and failed traces, since those traces cannot simply be concatenated with the student prefix). The generated continuation is appended, and a masked causal language modeling loss is applied only to assistant tokens after the CTB and before the final-answer step. Prefix tokens, user messages, tool observations, and the final answer receive zero weight. This loop — rollout, detect CTB, generate recovery, localized SFT — repeats for five iterations.

Training details reported: the FTRL training set has 2,215 tasks; the base dataset comprises over 2,000 automatically constructed tool-use environments with verifiable feedback, covering Single, Para-Single, Multi, and Para-Multi structures. Teacher trajectories come from a mixed pool of Qwen3.6-27B and GLM-5.2. Each iteration generates eight trajectories per task at temperature 0.7, maximum generation length 4,096 tokens, and at most nine interaction turns, then fine-tunes for one epoch with a learning rate of 5×10⁻⁷ and batch size 32. Each augmented training context contains two positive references and one negative reference, each positive accompanied by a teacher-generated analysis.

Why This Matters

Impact on research. The paper reframes agent distillation as a topology problem rather than a sequence-imitation problem. It offers a concrete alternative to two dominant paradigms — global loss SFT and uniformly credited RL such as GRPO — and argues that credit assignment should be anchored to a structural notion of where a policy becomes unrecoverable. The ISTG abstraction also provides a shared state space in which teacher and student executions can be compared without requiring action-wise correspondence.

Real-world applications:

• Enterprise API orchestration agents that must complete several independent sub-goals in any order, such as provisioning resources or reconciling records.
• Customer-service agents that combine knowledge-base lookups, order status checks, and account updates across multiple turns.
• Coding or data-analysis assistants that query repositories, run scripts, and inspect outputs in a self-chosen order.
• Long-horizon research or retrieval agents that chain multiple tools to answer multi-hop questions, the setting probed by ToolHop.

Industry relevance. The method targets compact open-source models (Qwen3-4B and Qwen3-8B here) that can be distilled from expensive frontier teachers and deployed more cheaply. Two reported properties are directly relevant to production: the final student's successful trajectories are shorter than the golden references (3.55 versus 4.02 tool calls), reducing redundant tool calls and latency, and general capabilities measured on IFEval, AIME24, AIME25, and MMLU are not degraded relative to the base model, which is a common failure mode of narrow fine-tuning.

Future Directions

• Scaling and architecture generality. The paper tests only two backbones from one model family (Qwen3-4B, Qwen3-8B). Whether the ISTG projection and CTB detection transfer to other architectures, mixture-of-experts models, or much larger scales is not reported.

• Cost and complexity of graph construction. ISTG construction requires a semantic stage that assigns information atoms by examining all candidates of a task jointly, plus a mixed teacher pool (Qwen3.6-27B and GLM-5.2). The paper does not report the compute, token, or human-judgment cost of building and maintaining these graphs, nor how sensitive results are to atom-judgment errors — it only argues that errors tend to omit an atom rather than invent one.

• Interaction with reinforcement learning. The baselines include both distillation and RL methods, but DART-SD is not combined with RL. Whether CTB-localized supervision could serve as a reward-shaping signal or be interleaved with GRPO-style optimization remains an open question.

• Robustness and verification. The paper reports single scores per benchmark without error bars, variance, or significance tests, and σ-bench is measured only via Pass^1. Repeated-seed evaluations, sensitivity to the reachability budget parameters (the allowance and cap), and behavior under more open-ended tasks without verifiable feedback are not reported.

Target Audience

Researchers and engineers working on LLM agents, tool use, and model distillation will get the most from this paper, particularly those who have hit the limits of full-trajectory SFT or struggled with credit assignment in outcome-reward RL for long-horizon tasks. It is also relevant to practitioners building production tool-calling agents who need to close the gap between large frontier teachers and smaller deployable students. Readers should already be comfortable with supervised fine-tuning, policy optimization terminology, and graph-based state representations; the paper's methodology section is equation-heavy and assumes that background.

Authors’ abstract

Equipping Large Language Models (LLMs) with multi-turn tool-calling capabilities is essential for building autonomous agents. However, progress is fundamentally limited by the reliance on full-length trajectory imitation. For tasks involving multiple order-independent sub-goals, the optimal solution space forms a vast combinatorial diamond lattice. Forcing this rich topology into monolithic trajectories causes a severe topological collapse, indiscriminately penalizing valid alternative explorations and severely degrading policy diversity. To address this, we propose DART-SD (Diamond-topology Aware Retrieval and Tuning for Self-Distillation), a novel framework that shifts the paradigm from global forcing to topology-guided localized correction. DART-SD first models the execution process as a converging Interaction-State Transition Graph (ISTG), faithfully capturing the inherent diamond topology of successful and failed exploratory paths. During autonomous rollouts, the framework identifies the Critical Topological Breakpoint (CTB) and retrieves success-supported recovery references. Finally, we introduce a progressive self-distillation paradigm through CTB-guided localized supervision, ensuring that the training loss is calculated exclusively on the generated recovery steps while strictly protecting the valid reasoning prefix from destructive gradient updates. Experiments on complex multi-turn tool-calling benchmarks demonstrate that DART-SD significantly outperforms traditional full-trajectory baselines.