EDGEGEN: Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation

AI summary

Overview

Research area: synthetic training-data generation for tool-calling LLM agents, with an emphasis on policy compliance, edge cases, and automated agent optimization.

Technical level: Intermediate. The pipeline uses LLM-driven generation, SQL-grounded database sampling, and supervised finetuning, but the core ideas are explained in accessible terms.

Scope: The paper introduces EdgeGen, a framework that extracts business-compliance rules from an agent's specification, enumerates rule-violation scenarios, grounds them in executable database states, and uses the resulting tasks for finetuning or harness optimization — reporting gains on τ²-bench airline and retail and on ToolSandbox across six open-source models.

What This Paper Is About

Enterprises deploying tool-calling LLM agents need high-quality, diverse task data to evaluate and improve them, but production logs are often unavailable for privacy reasons and manually writing realistic tasks is expensive. Existing synthetic task generators tend to produce generic function-calling tasks that ignore the agent's database state and business rules, leaving rare edge cases — the situations where agents actually fail — underexplored. EdgeGen generates database-grounded tasks explicitly designed to violate an agent's compliance rules, forming a closed loop with no human annotation.

Key Contributions

1. EdgeGen, a violation-driven synthetic task generation framework that extracts compliance rules from an agent's specification, enumerates structured combinations of rule violations, and grounds each task in executable database states.
2. A fully automated closed-loop system combining synthetic database generation and EdgeGen task construction, producing training and evaluation data without human annotations or production logs.
3. Empirical evidence that explicitly covering edge cases improves tool-using agent performance across benchmarks and model scales, both through supervised finetuning and through agent harness optimization.
4. A complexity ablation analyzing how the number of embedded policy violations (happy path, 1, 2, 3 violations) affects downstream performance, plus a human study validating the evaluation pipeline.

Main Findings

• Consistent finetuning gains from EdgeGen data: Across all 18 model–benchmark pairs in Table 1, EdgeGen improves mean progress over the base model. The paper reports a consistent mean progress improvement of +2% to +42% on the τ²-bench airline domain.

• Large gains for small models on airline: On τ²-bench airline, EdgeGen achieves a +42.2% relative improvement on Qwen 2.5-3b (mean progress 0.258 vs 0.366, as reported in the text; Table 1 lists 0.26 → 0.37).

• Baselines can degrade performance: Finetuning on human-curated data regresses on 3 of 6 models (up to −0.05 points, or −13.16%, on gemma-4-e2b), and TaskBench regresses on 4 of 6 models in the airline domain. The authors interpret this as evidence that tasks with less coverage (no violation scenarios) are insufficient and can be actively harmful.

• ToolSandbox results: Relative improvement over base ranges from +10.9% (0.605 vs 0.671) to +23.4% (0.528 vs 0.652). ToolSandbox focuses more on tool-calling mechanics with fewer turns, so most baselines also improve there. FuncBenchGen produced only a single valid rollout for ToolSandbox.

• Harness optimization with Gemma-4-e4b: The EdgeGen-optimized harness improves mean progress from 0.4301 to 0.5608, a 30% relative improvement over the base harness (0.6583 max progress, 5.8 tools, 81.0k tokens) and a 10% relative improvement over the harness optimized on human-curated data (0.8167 max progress, 0.5083 mean progress, 8.8 tools, 121.0k tokens). The EdgeGen harness also uses fewer tool calls (6.9) and tokens (85.0k).

• Harness optimization with gpt-5.4: For gpt-5.4, the EdgeGen harness (0.9542 max, 0.7979 mean, 7.9 tools, 66.5k tokens) is within a 2–4% relative difference of the human-curated-optimized harness (0.9750 max, 0.8307 mean, 8.9 tools, 78.8k tokens) across both metrics, while being more efficient in tool calls and tokens.

• Different prompt structures emerge: Harness optimization on human-curated data produces long procedural checklists that hard-code business rules case by case. EdgeGen-optimized prompts collapse into a short discover → verify → confirm → write protocol that references the policy document and tool descriptions instead of enumerating them inline, which the authors say is more robust to edge cases and suits smaller models with shorter context windows.

• Complexity ablation is non-monotonic: For Qwen2.5-3B, maximum progress is 0.35 at base, 0.39 with happy-path training, 0.67 with 1 violation, 0.24 with 2 violations, and 0.38 with 3 violations. For Qwen3.5-4B, base is 0.89, happy path 0.88, 1 violation 0.92, 2 violations 0.90, 3 violations 0.90. One violation is the strongest condition; additional violations do not improve results.

• Data scaling: Experiments varying the synthetic training budget over 10, 30, and 60 tasks show human-curated test mean progress is highest at 30 tasks for both finetuned models. For harness optimization, improved synthetic validation progress does not consistently transfer to the human-curated test set.

• Human study: Among runs marked successful, 6 of 10 airline runs and 4 of 10 ToolSandbox runs were verified correct; remaining cases involved assertion errors, missing task details, or judge errors. Most sampled failures reflected genuine agent limitations.

Methodology in Plain English

The framework treats an agent deployment as three artifacts: a set of tools with typed inputs and outputs, a natural-language policy document, and a relational database the agent reads and writes.

1. Extract rules. An LLM extracts atomic compliance rules from the policy document. The paper limits the rule set to the five most important extracted rules, giving at most 2^5 = 32 subsets before feasibility filtering.

2. Build and sample workflows. Tools are arranged in a directed graph where an edge exists between two tools if one's output types overlap the other's input types. Workflows are sampled conditioned on the policy, then shaped as a Node (single tool), Chain (linear), or DAG (branching).

3. Enumerate violations. The space of compliance conditions is the power set of the rule set. Subsets that are not jointly satisfiable are removed by an LLM-based consistency checker. A subset size of 0 means a happy-path task; larger sets require the agent to handle multiple interacting rules at once.

4. Ground in the database. A scenario-aware SQL agent iteratively looks for database rows that instantiate the scenario, resampling the workflow and violation set if no valid grounding is found. The synthetic database itself is produced by a Generalist Populator LLM agent that invokes API write operations rather than editing rows directly; EdgeGen is described as agnostic to the database source.

5. Generate and verify. An LLM writes the task summary using concrete database entities and the natural-language assertions that encode the expected response. A verifier then runs two checks — data grounding (facts and identifiers match the database, including user claims that contradict stored state) and scenario feasibility (the expected response is achievable with the available tools and state) — using deterministic SQL queries. A test case is accepted only if both hold; otherwise the pipeline restarts from workflow sampling.

For experiments, EdgeGen generates 15 diverse tasks per benchmark. Expert demonstrations come from gpt-5.4 over 8 trials per sample, giving 120 attempted rollouts per benchmark before filtering; only traces with progress rate 1 are retained for supervised finetuning. Evaluation applies the Talk, Evaluate, Diagnose (TED) framework with gpt-4.1 as the expert persona user simulator and LLM-as-a-judge, over 8 trials per scenario, reporting mean and maximum progress rate.

Benchmarks: τ²-bench airline (14 tools, 10 human-curated samples used), ToolSandbox (33 tools, 15 human-curated samples), and τ²-bench retail (15 tools, 10 human-curated samples). Models: Qwen 2.5-3b, Qwen 3.5-4b, Qwen 3.5-9b, Qwen 3.5-35b-A3b, Gemma 4-e2b, Gemma 4-e4b. Harness optimization uses 10 train scenarios, 8 synthetic EdgeGen validation scenarios, and 10 human-curated test samples, with 8 evolution iterations for Gemma 4-e4b and 4 for gpt-5.4, and three candidate harnesses proposed per iteration.

Why This Matters

Impact on research: The paper argues that coverage of behavioral policies — not just correct tool invocation — is a missing axis in synthetic data generation for agents. It provides a recipe for turning a written policy into structured violation scenarios, and it reports that training on narrow, human-curated or generic synthetic data can actively harm performance, which is a notable counterpoint to the assumption that more curated data is always better.

Real-world applications:

• Enterprise customer-service agents in domains like airlines or retail, where refunds, cancellations, and baggage rules must be denied or approved based on database state.
• Compliance-sensitive workflows where the agent must verify identity or eligibility before modifying records.
• Black-box or hosted models that cannot be finetuned, but whose surrounding harness can be optimized using the same generated tasks.
• Automated regression testing of deployed agents against rare boundary cases rather than only happy-path interactions.

Industry relevance: The pipeline requires no production logs, no human-annotated tasks, and no real customer data, which addresses the privacy and cost barriers that block evaluation in enterprise settings. It also demonstrates efficiency benefits: the EdgeGen-optimized harness for Gemma-4-e4b uses fewer tool calls and tokens than the human-curated-optimized harness, which matters for inference cost and for small models with limited context.

Future Directions

• Reducing dependence on structured policy documents, since sparse, ambiguous, or informal specifications may yield incomplete rule sets and weaker edge-case coverage.
• Extending the approach beyond enterprise tool-calling to coding agents and web-navigation agents, which the paper says remain untested.
• Improving verification quality, as the human study found loosely defined assertions and missing task details in some sampled runs, and the verifier does not guarantee that every extracted rule or assertion is semantically correct.
• Scaling evaluation: the main experiments use 10–15 scenarios per benchmark and the supplementary scaling uses 21 airline scenarios, so larger evaluation sets and broader benchmark suites are needed. The authors also note that gains are most pronounced for smaller models and that harder real tasks should be studied.

Target Audience

Researchers and engineers working on LLM agents, synthetic data generation, and agent evaluation will benefit most, particularly those building enterprise tool-calling systems with policy constraints. The paper is also relevant to practitioners who cannot access production logs or finetune their models and need an automated way to stress-test and optimize agent harnesses.

Authors’ abstract

Tool-calling LLM agents are increasingly deployed in enterprise applications. However, effective evaluation and optimization require high-quality, diverse task datasets that are often difficult to obtain due to privacy and other constraints. Existing synthetic task generation methods often produce generic tasks that ignore an agent's underlying state or database and fail to reflect real-world usage diversity. We propose EdgeGen, a synthetic task generation framework that extracts compliance rules from an agent's specification and uses them to generate database-grounded edge-case tasks designed to violate these rules. When combined with existing synthetic data generation techniques, EdgeGen enables agent improvement through finetuning and harness optimization. The resulting pipeline forms a fully automated closed-loop system that requires no human annotation. Finetuning on data generated by EdgeGen yields a consistent mean progress improvement of 2 percent to 42 percent on tau2bench airline domain, while other baseline methods show degradation for some models. On the other hand, for harness optimization, our method shows a mean progress improvement of 10 percent and 30 percent over the human-curated and base harnesses, respectively, for the Gemma-4-e4b model.