STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Models

AI summary

Overview

Research area: Efficient AI / small language models — specifically transferring function-calling (tool-use) ability from large language models into "super-tiny" models (0.6B–4B parameters) via knowledge distillation combined with reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with KL-divergence-based knowledge distillation, logits and softmax gradients, GRPO-style reinforcement learning, reward shaping, and benchmark evaluation for tool calling.

Scope: A single-paper summary of the STAR framework, its two components (Constrained Knowledge Distillation and Similarity-guided RL), the training curriculum that combines them, and the reported results on BFCLv3 and ACEBench for Qwen3-0.6B, 1.7B and 4B students.

What This Paper Is About

Large language models are good at function calling — deciding which external tool or API to invoke and with what arguments — but their size makes them expensive to run on devices or at large scale. The standard recipe for shrinking them (supervised fine-tuning followed by reinforcement learning) breaks down for very small models: they overfit and memorize tool-call formats, RL is unstable on them, and binary success/failure rewards unfairly punish valid alternative answers on multi-solution tasks. STAR's goal is a training framework that stably transfers a large teacher model's function-calling ability into super-tiny students and then refines them with a continuous, similarity-based reward.

Key Contributions

1. Constrained Knowledge Distillation (CKD). A distillation objective that augments top-k forward KL divergence with a targeted L1 penalty on tokens the student finds probable (its top-m set) but the teacher has dismissed (outside the teacher's top-k set). It suppresses "confidently incorrect" predictions without forcing the long tail to zero, preserving the exploration capacity needed for later RL.
2. Sim-RL, a similarity-guided reinforcement learning mechanism. Instead of a binary pass/fail signal, Sim-RL computes a continuous reward from the similarity between generated outputs and ground truth — argument-level similarity for tool calls (ROUGE-L F1 for strings, exact match for numeric/boolean and other types) and ROUGE-L F1 for text-only responses — combined with a strict binary format reward. Total reward is bounded in [-1, 1].
3. A unified training curriculum. Teacher adaptation with Sim-RL, then CKD distillation from the refined teacher into the student, then a final Sim-RL refinement pass on the student — a sequence the authors argue makes CKF and Sim-RL synergistic rather than interfering.
4. Empirical demonstration that a 0.6B model can reach state of the art in its size class. The paper reports 0.6B STAR results on BFCLv3 and ACEBench that outperform all baselines and, per the authors, all open models under 1B.

Main Findings

• Combined method sets a new mark on the 0.6B scale: CKD+Sim-RL reaches 51.70 overall accuracy on BFCLv3 and 53.00 on ACEBench Normal, versus 47.33 and 27.20 for the untuned Qwen3-0.6B base model.
• Components work alone but combine best: On BFCLv3, CKD alone scores 49.84 and Sim-RL alone (on SFT-think) 50.41; on ACEBench, CKD scores 39.00 and Sim-RL 38.90. The paper states the combination adds over 2 points on BFCLv3 and 14 points on ACEBench relative to the individual components.
• Relative gains: The introduction reports relative gains of 9.2% on BFCL and over 50% on ACEBench for the 0.6B STAR model against baselines.
• SFT collapses on ACEBench: Standard SFT drops to 2.10 ACEBench Normal (from 27.20 for the base model), which the authors attribute to overfitting to the training data's JSON format and failure to adapt to ACEBench's Python-style call syntax. STAR, trained on the same data, generalizes to the unseen format.
• Top-k RKL/AKL is unstable in distillation: Combining top-k truncation with mode-seeking reverse KL (or AKL) leads to catastrophic training collapse, while top-k FKL stays stable because it simply ignores the tail.
• RKL costs exploration: Even stabilized RKL/AKL variants reduce the student's output entropy relative to FKL, yielding policies that underperform a simple top-k FKL initialization after downstream RL.
• KD beats cross-entropy as an RL initializer: In the KD-strategy ablation, CKD with RL gives 51.70 BFCLv3 / 53.00 ACEBench, compared with CE 50.41 / 38.90, FKL 51.46 / 50.00, RSKD 50.65 / 49.80, stabilized RKL 50.49 / 41.30, and stabilized AKL 50.29 / 49.00.
• Similarity reward beats binary and PRM-style rewards: On ACEBench Normal, CKD+Sim-RL scores 53.00 versus CKD+Binary Reward 35.70, CKD+ToolRL 48.59 on BFCL but 40.50 on ACEBench, and CKD+SwiRL 51.10 / 40.30.
• Cross-scale comparisons: STAR-0.6B (53.00 on ACEBench) exceeds Llama3.1-8B (46.60); STAR-4B (74.10) exceeds Qwen3-8B (72.90); STAR-1.7B scores 56.05 BFCLv3 / 60.90 ACEBench against Qwen3-1.7B's 54.70 / 51.60. The teacher (Teacher-8B) scores 67.74 / 72.70.
• Pass@k and entropy evidence: The paper reports in figures (no numeric values given in the text) that CKD-initialized policies show the highest Pass@k among initializers and higher policy entropy at the start of RL.

Methodology in Plain English

The authors start from the observation that a very small model cannot simply be fine-tuned on tool-call examples — it memorizes formats instead of learning the underlying decision process. Their alternative is to let a bigger model teach it, but through probability distributions rather than hard answer keys.

For the distillation step, they keep the stable, well-behaved forward KL divergence but only over the teacher's top-k tokens (for computational efficiency, with k set to 100). They then add one extra penalty term: if the student assigns meaningful probability to a token that the teacher put outside its top-k list, that probability is penalized directly. The penalty weight (lambda_tail) is set to 10, and the student's monitored top-m set is also 100. The intent is to squash the student's most dangerous mistakes while leaving the rest of the tail alive for later exploration.

For the reinforcement learning step, they use GRPO, which samples multiple rollouts per prompt and computes advantages by standardizing rewards within the group. The reward has three parts. A binary format reward checks that the output obeys the required structure (one pair of think tags, tool calls wrapped in tool-call tags, a JSON object with "name" and "arguments" keys, a name drawn from the available function set, and argument keys that are a subset of that function's defined keys). A tool-call reward compares predicted calls to ground-truth calls using greedy one-to-one matching and an argument-level similarity score, normalized in an IoU-like way. A response reward uses ROUGE-L F1 for text-only answers. Format errors produce a -1 penalty; otherwise the answer term applies. Groups in which all rollouts get 0 or all get 1 (zero advantage) are filtered out, following DAPO, to avoid wasted computation.

The full curriculum runs three stages: adapt the teacher (Qwen3-8B) with Sim-RL on the training data, distill the adapted teacher into the student with CKD, then refine the student with Sim-RL one more time.

Training data merges four sources: ToolACE (11.3k instances), xLAM (60k samples), xLAM-irrelevance (6.7k filtered samples for irrelevant-function detection, with answers synthesized using Qwen3-32B), and tool-use-synthetic (50k sampled instances of multi-step and multi-turn interactions). The teacher generates rollouts over this data to create an augmented set used for distillation. Baseline comparisons include Base-model, SFT, SFT-think, top-k FKL (k=100), ToolRL, LUFFY, and GKD. Evaluation uses BFCL and ACEBench. Experiments ran on a single server with 8 NVIDIA H20 GPUs using OpenRLHF; RL used a learning rate of 3e-7, batch sizes of 128, a k2 KL approximation with initial coefficient 1e-3, and 8 rollouts per prompt; KD used a learning rate of 3e-6 and batch size 128; SFT used a learning rate of 2e-5 and batch size 128. Reporting says "extensive experiments"; no repeated-run variance or error bars are reported in the available content.

Why This Matters

Impact on research. The paper argues that the dominant SFT-then-RL recipe is the wrong starting point for low-capacity models, and offers distillation as the initializer instead. It also contributes an empirical claim about divergence choice — that reverse KL and AKL, including stabilized variants, suppress the student's entropy and therefore handicap later RL — plus a reward-design argument that continuous similarity signals outperform binary and process-reward-model-style signals on tasks with many valid answers.

Real-world applications:

• On-device and edge assistants that need tool calling without cloud inference.
• Automated personal assistants that route user requests to the right API with the right arguments.
• Complex data analysis systems that chain multiple tool invocations across turns.
• Large-scale services where per-query inference cost is the binding constraint.

Industry relevance. The work comes from the Algorithm Platform Team, AI Hardware Division at Alibaba, which frames the motivation in terms of hardware division and deployment economics. Low-latency, low-cost inference for agentic workloads is a direct commercial concern, and the authors release code and pre-trained models at https://github.com/Qwen-Applications/STAR.

Future Directions

• Extend beyond function calling. The limitation section states the framework is validated only on function calling, and names SQL generation and mathematical reasoning as promising target tasks.
• Design better similarity measures. The authors describe their similarity-based reward as an initial approach and leave a more comprehensive study of alternative or more sophisticated similarity measures to future work, noting that the potential gains from more granular feedback "remain to be quantified."
• Multi-teacher strategies and richer reward designs. The conclusion explicitly lists these as directions, alongside deployment-aware constraints in the training objective.
• Interpret and control the exploration benefit. The entropy argument (RKL/AKL prune the tail and hurt RL; CKD preserves it) is supported empirically here; a fuller theory of how much entropy a tiny student needs, and how to tune lambda_tail, k and m for it, is left open.

Target Audience

Researchers and engineers working on small-model specialization, knowledge distillation for language models, and RL fine-tuning with verifiable or similarity-based rewards. It is also relevant to practitioners deploying agentic function-calling models on constrained hardware, and to anyone evaluating whether distillation beats supervised fine-tuning as an initialization for reinforcement learning on low-capacity models.

Authors’ abstract

The proliferation of Large Language Models (LLMs) in function calling is pivotal for creating advanced AI agents, yet their large scale hinders widespread adoption, necessitating transferring their capabilities into smaller ones. However, existing paradigms are often plagued by overfitting, training instability, ineffective binary rewards for multi-solution tasks, and the difficulty of synergizing techniques. We introduce STAR: Similarity-guided Teacher-Assisted Refinement, a novel holistic framework that effectively transfers LLMs' capabilities to super-tiny models. STAR consists of two core technical innovations: (1) Constrained Knowledge Distillation (CKD), a training objective that augments top-k forward KL divergence to suppress confidently incorrect predictions, ensuring training stability while preserving exploration capacity for downstream RL. STAR holistically synergizes these strategies within a cohesive training curriculum, enabling super-tiny models to achieve exceptional performance on complex function calling tasks; (2) Similarity-guided RL (Sim-RL), a RL mechanism that introduces a fine-grained, similarity-based reward. This provides a robust, continuous, and rich signal for better policy optimization by evaluating the similarity between generated outputs and the ground truth. Extensive experiments on challenging and renowned benchmarks demonstrate the effectiveness of our method. Our STAR models establish SOTA in their size classes, significantly outperforming baselines. Remarkably, our 0.6B STAR model achieves the best performance among all open models under 1B, surpassing even several well-known open models at a larger scale. STAR demonstrates a training framework that distills capabilities of LLMs into super-tiny models, paving the way for powerful, accessible, and efficient AI agents.