Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

AI summary

Overview

Research area: LLM agents / tool-calling (function calling), benchmark construction, and synthetic training-data generation for open-source language models.

Technical level: Intermediate. The framing (multi-step API chaining, pagination, dependency graphs) is accessible, but the method involves Bayesian posterior sampling (Thompson sampling, Beta priors), GRPO reinforcement learning, and evaluation protocols that assume familiarity with agent benchmarks like BFCL and τ²-Bench.

Scope: The paper introduces KOPA-Bench, a 145-task benchmark for multi-step tool-calling over live Korean public APIs, and EDGE, a synthesis pipeline that builds an execution-verified tool dependency graph to generate executable multi-step training trajectories.

What This Paper Is About

Government agencies subject to data-sovereignty rules must run open-source LLMs on local infrastructure, yet these models perform poorly when asked to chain several tool calls across live public APIs. The authors build a benchmark that exposes this gap and a data-synthesis method that closes it by verifying tool-to-tool dependencies against the real APIs rather than guessing at them.

Key Contributions

1. KOPA-Bench, a 145-task multi-step function-calling benchmark grounded in live Korean public APIs across 10 platforms and six domains (traffic, finance, education, law, politics, district administration), where every task requires chaining dependent calls through code lookups and high-cardinality intermediate results.

2. EDGE (Execution-grounded Dynamic Graph for tool-calling data synthEsis), a two-phase pipeline that proposes candidate tool dependencies with an LLM, then validates them by actually calling the live APIs, pruning edges that repeatedly fail.

3. A cardinality-aware trajectory synthesis scheme that types each output-to-input junction as Pure Sequential, Fan-out, or Derived, converting many-record API responses into valid sequential training trajectories through bounded enumeration or deterministic reduction.

4. Empirical demonstration that GRPO fine-tuning on the resulting 1,781-task dataset lifts small Qwen3.5 models (4B, 9B) substantially on KOPA-Bench and generalizes to the out-of-distribution BFCL benchmark.

Main Findings

• Large gains for small models on KOPA-Bench: pass@1 rose from 0.18 to 0.31 (+13 pp) for Qwen3.5-4B and from 0.33 to 0.43 (+10 pp) for Qwen3.5-9B. The fine-tuned 9B model nearly matches the untuned 27B model (0.45) at one-third the parameter count.

• Results are statistically robust: Re-evaluated across eight independent seeds using Student's t-distribution, the 95% confidence intervals of fine-tuned models and base models do not overlap (e.g., [0.2762, 0.3100] vs. [0.1400, 0.1807] for the 4B pair).

• Gains transfer out of distribution: On BFCL, multi-turn scores improved by +4.04 pp (4B) and +5.87 pp (9B), consistently exceeding single-turn gains despite training data being predominantly single-turn.

• Improvement generalizes to unseen platforms: On the 31 tasks drawn entirely from platforms withheld from synthesis, the 4B model improved by +22.6 pp in pass@4, larger than its +15.9 pp gain across the full benchmark.

• The data matters more than the objective: SFT on the identical 1,781 tasks (without GRPO) already raised pass@1 by +9.7 pp and Action by +9.0 pp, so most of the gain comes from the corpus. GRPO adds a further +11.0 pp on pass@4 by sampling multiple rollouts per prompt and learning from relative rewards.

• Execution grounding meaningfully separates viable edges from dead ones: Only 50.2% of LLM-proposed skeleton edges actually execute successfully, whereas the refined graph G⋆ reaches 62.7%. The pruned edges execute only 14.8% of the time, and 70.5% of pruned edges never succeed in any trial versus 27.7% of retained edges.

• Trajectory diversity helps: Training only on purely sequential trajectories scores 0.2327 pass@1, versus 0.3080 for parallel + mixed and 0.3094 for the full mixture. Sequential data teaches precise handoffs; parallel data supplies comparative and conditional patterns.

• Filtering matters: Removing anomalous synthesized tasks (malformed structures, stale labels from time-varying APIs) lifted pass@1 from 0.242 to 0.309.

• Even frontier models struggle: Claude Sonnet 4.6 leads at 0.4655 pass@1 on KOPA-Bench, with GPT-5.1 at 0.3706 and Qwen3.5-27B at 0.4482, confirming the benchmark is far from saturated.

Methodology in Plain English

The authors start by turning official Korean public API documentation into 2,318 callable functions exposed through Model Context Protocol servers, complete with typed signatures, Korean descriptions, authentication handling, and retry logic.

To create training data, they treat the tools as nodes in a directed graph where an edge means "an output field of tool A can fill an input parameter of tool B." Because testing all ~5 million possible tool pairs is infeasible, a dense retriever narrows candidates to the top 15 same-domain and top 10 cross-domain neighbors, and a single LLM call scores each candidate's feasibility and proposes bindings.

The key step is Phase A: instead of trusting the LLM's judgment, the system samples paths through the graph using Thompson sampling — a Bayesian method that balances exploiting edges with high estimated success rates against exploring uncertain ones — and actually executes them against live APIs. Edge posteriors update on every run. Notably, structural failures (wrong binding) penalize an edge far more than environmental failures (transient server errors), so random server hiccups don't wrongly kill valid edges. Edges whose posterior success probability drops too low get pruned, yielding a refined graph G⋆.

Phase B walks G⋆ to build trajectories. The problem is that Korean public APIs often return thousands of records, so a single output field doesn't map to one downstream argument. The authors classify each junction by how many distinct values it produces: single-value junctions pass through directly (Seq); two to five values trigger one downstream call each (Fan); more than five values trigger an inserted reduction step that filters, ranks, or selects a bounded subset (Drv). They also generate three non-sequential patterns from templates: parallel calls (Sem), same-tool comparisons (Cmp), and conditionals (Cond). An LLM then writes a Korean natural-language query for each executable trajectory, and a three-stage validation pipeline checks query, answer, and execution trace. Only open-source model outputs become training labels; proprietary models only verify.

Fine-tuning uses GRPO with a binary reward over RESPONSE and ENVIRONMENT dimensions, trained on 8×H100 GPUs with verl.

Why This Matters

Impact on research. The paper directly attacks a blind spot in existing tool-calling data synthesis: prior systems (ToolACE, Magnet, BUTTON, APIGen-MT, AWM) fix links between calls using signature matching or LLM plans without ever checking whether those links execute against real endpoints, and they assume each call returns a single result. EDGE shows that only about half of plausible-looking edges survive contact with live APIs, and that multi-record responses — the normal case for public-sector data — require explicit cardinality handling. It also contributes a benchmark where frontier proprietary models still score under 0.47.

Real-world applications:

• Government chatbot and citizen-service agents that need to chain lookups across corporate disclosure, law, traffic, and education portals to answer a single user question.
• Enterprise compliance and deal-diligence tools that must resolve entity codes and aggregate paginated financial filings before computing an answer.
• Any on-premise deployment where privacy or sovereignty rules forbid sending queries to commercial cloud models, making small open-source models the only option.
• General-purpose agent frameworks that need robust handling of paginated or high-cardinality API responses, a failure mode that appears well beyond Korean public APIs.

Industry relevance. The work is funded by South Korea's Ministry of Science and ICT as part of its Sovereign AI Foundation Model Project, and the authors are at LG CNS. This reflects a concrete commercial demand: regulated public institutions that need capable agents but cannot use cloud LLMs, and who therefore need small models that punch above their weight class.

Future Directions

• Endpoint drift. Because both benchmark and training data are grounded in live APIs, schemas, availability, and returned records change over time. The authors filter out tasks tied to real-time data and hash-compare environment state, but exact reproduction depends on services they do not control.

• Transfer beyond Korean public APIs. Whether EDGE's execution-grounded approach generalizes to other languages, commercial or private APIs, and foreign administrative systems remains open.

• System-level comparison. The ablations compare EDGE against controlled variants of itself, including a schema-only skeleton. Running an existing synthesis pipeline end-to-end on this tool inventory and training on its output was left out of scope.

• Scaling and trajectory length. With tasks averaging 4.92 tool calls (up to 14), the question of how the approach behaves on substantially longer horizons, or how fan-out budget φ and pruning thresholds should scale, is unexplored.

Target Audience

Researchers and engineers working on LLM agents, function-calling, and synthetic data generation; practitioners building on-premise or sovereign AI deployments for government and regulated industries; and benchmark designers interested in how to evaluate agentic behavior when outcomes, not just action sequences, must be checked. Readers familiar with BFCL or τ²-Bench will get the most out of the evaluation sections, while the graph-based synthesis approach is legible to anyone comfortable with basic machine learning and API design.

Authors’ abstract

Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution. EDGE builds a graph of how each tool's output can feed another's input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuned via GRPO on the resulting dataset, our 9B model nearly matches the untuned 27B model from the same family, improving substantially not only on KOPA-Bench but also on the BFCL benchmark.