Bridging Tool Dependencies and Domain Knowledge: A Graph-Based Framework for In-Context Planning

AI summary

Overview

Research area: LLM agents and tool-augmented reasoning; specifically knowledge-graph construction, retrieval, and in-context planning (ICP) for domain-specific business toolchains.

Technical level: Intermediate. The paper assumes familiarity with LLM agents and tool calling, retrieval-augmented generation, knowledge graphs, personalized PageRank-style graph propagation, and benchmarks such as ToolBench. It does not require deep mathematical background; the emphasis is on system design and empirical comparison.

Scope: The paper proposes and evaluates a pipeline that builds a tool dependency graph from tool schemas, fuses it with a domain knowledge graph built from internal documents and SOPs, and uses the fused graph to generate exemplar plans for in-context planning.

What This Paper Is About

Most tool-use research targets general-purpose, single-function tools (for example, "send email" or "make calendar") whose descriptions are easy for LLMs such as GPT-4o and Claude 3.5 to interpret. Real business assistants instead operate over hundreds of complex, domain-specific tools for areas such as inventory tracking, performance monitoring, and financial reporting, which general-purpose models struggle to parse, especially at cold start. The paper's goal is to automatically discover dependencies among those tools, link them to the internal documents and SOPs that describe how to use them, and use the resulting fused graph to generate high-quality exemplar plan artifacts for in-context planning.

Key Contributions

1. A Deep Research approach to tool dependency extraction. The authors repurpose a Deep Research pipeline (originally web-search plus LLM refinement) to explore tool schemas — descriptions, arguments, and output payloads — in order to uncover dependencies among tools and build a tool knowledge graph. The node in the original Deep Research pipeline is modified into dependency extraction, followed by an LLM-as-a-judge step that removes dependencies that do not make sense under in-domain specific requirements. The pipeline is written in LangGraph.

2. Fusion of a tool graph with a domain knowledge graph. Internal documents and SOPs containing tool usage instructions are modeled as a knowledge graph using GraphRAG with its default setup, then enriched with the tool graph through graph fusion in Neptune, where the dependency relation is defined as can_use_this_tool_output.

3. A dense–sparse integration framework for exemplar plan generation. Following HippoRAG2, the fused graph is used to produce exemplar artifacts for in-context planning, with the tool graph acting as the sparse component and the domain knowledge as the dense component.

4. Empirical validation. Experiments on ToolBench-derived data validate the effectiveness of studying tool dependencies with deep research and demonstrate the usefulness of connecting tool graphs with document knowledge graphs.

Main Findings

• Dependency checking is accurate across several LLMs. On 1,500 valid tool dependencies, GPT-4o predicted 1,332 dependencies with 90.7% precision and 80.5% recall; Claude 4 predicted 1,652 with 79.9% precision and 88.1% recall; Claude 3.7 predicted 1,462 with 80.9% precision and 78.9% recall; DeepSeek R1 predicted 1,652 with 66.4% precision and 73.2% recall; Qwen3-8B predicted 1,453 with 83.2% precision and 80.7% recall.

• A smaller model is competitive on dependency checking. Qwen3-8B, despite its smaller size, performs well and attains slightly higher precision (83.2%) than Claude 4 (79.9%) and Claude 3.7 (80.9%), while DeepSeek R1 is the weakest on precision at 66.4%.

• Exemplar plan generation is stable across models. With binary match accuracy and an LLM-as-a-judge score on a 0–2 scale for plan coverage: GPT-4o scored 77% / 1.62; Claude 4 scored 69% / 1.47; Claude 3.7 scored 71% / 1.49; DeepSeek R1 scored 64% / 1.36; Qwen3-8B scored 72% / 1.58. The consistency across models suggests performance may be largely determined by the quality of the subgraph returned by Personalized PageRank and embedding match rather than solely by model capacity.

• Personalized PageRank contributes materially. The ablation with GPT-4o shows 77% binary match accuracy and 1.62 judge score with Personalized PageRank, versus 68% and 1.56 without it — a 9 percentage-point drop when it is removed. The authors attribute this to PPR recovering tool documents and procedural instruction pages that embedding-only retrieval overlooks; they give the example that pure embedding search can miss tools like BacklogCheck when they are dominated by other inventory-related tools, whereas PPR propagates importance through dependency links to surface them.

Methodology in Plain English

The authors start from the raw descriptions of the tools their business assistant must use — what each tool does, what arguments it takes, and what its output payload looks like. They feed tools to an LLM in pairs and ask it to judge whether one tool's output can meaningfully feed into another, an adaption of a Deep Research-style pipeline implemented in LangGraph. A separate LLM-as-a-judge pass then filters out dependencies that violate domain-specific requirements, yielding a tool knowledge graph.

In parallel, they build a domain knowledge graph from internal documents and SOPs using GraphRAG with its default configuration, since those documents contain the instructions for how tools are actually used in practice. The two graphs are merged in Neptune, with a dependency relation expressing that a tool's output can be used by another tool, producing one unified graph in which document nodes and tool nodes coexist.

To generate exemplar plans, they collect queries from production. For each query they use Jina Embedding (v3) to compute semantic similarity and retrieve the top-K tool triplets and relevant knowledge documents. Those retrieved tool triplets serve as seed nodes for Personalized PageRank, which produces a query-specific subgraph. An LLM then generates the exemplar artifacts from that subgraph, and those artifacts are stored in a vector database for future retrieval — supporting cold-start in-context planning.

For evaluation, they use ToolBench, which contains 16,464 APIs and provides three prompt levels: G1 for single-tool instructions, G2 for intra-category multi-tool instructions, and G3 for intra-collection multi-tool instructions, with queries and plans generated using depth-first search (DFS) planning. They randomly select 1,000 queries from each of G1, G2, and G3 (3,000 total) and use the dataset's ground-truth plans; after filtering invalid cases they identified 1,500 valid tool dependencies. To simulate external knowledge, they use the Tavily-Search API to retrieve online documents that can utilize each API output payload and add those document instructions into the ground-truth artifacts. Exemplar artifacts are judged by Nova pro as the LLM judge, using a 0–2 coverage score.

Why This Matters

Impact on research. The work connects two lines of prior research that are usually separate: graph-structured tool dependency modeling (LLM Compiler and follow-ups, which model tool interactions as directed acyclic graphs to enable parallel execution of independent tools) and domain-knowledge retrieval. It also extends a trend in planning-oriented work — CodeAct and CodePlan's pseudo-Python structure, ReWOO's separation of reasoning from tool observations, toolkit-level planning and replanning, Predictive-Decoding, ReasonFlux's template-slot reasoning with reinforcement learning, ToolRL's reward-design study, and Kimi K2's long-context training with multi-stage RL — by showing that the graph used to generate exemplars, not only the model generating them, drives plan quality.

Real-world applications:

• Business assistants that must operate over hundreds of in-domain tools across inventory tracking, performance monitoring, and financial reporting, where cold-start exemplar quality determines whether the assistant is usable.
• Automated onboarding of new tools: extracting tool dependencies from schemas reduces the manual effort of writing exemplar API executions for in-context planning.
• Grounding planning in internal SOPs and documents so that generated plans follow organizational procedure rather than generic tool-calling patterns.
• Dependency-aware orchestration, where knowing which tool outputs feed which tools allows more sensible sequencing and retrieval of the right procedural pages.

Industry relevance. All three authors are affiliated with Amazon, and the pipeline is built from production-oriented components — LangGraph, GraphRAG, Neptune, Tavily-Search, Jina Embedding v3, and Amazon Nova pro as the judge — with queries collected from production and documents simulated via web search. This places the contribution close to deployed enterprise agent infrastructure rather than purely academic settings.

Future Directions

• Building real benchmarks for tool dependency detection. The paper explicitly states that the absence of real benchmarks for detecting tool dependencies remains a limitation, which the authors plan to address in future work.
• Replacing or supplementing simulated external knowledge. Because domain documents were simulated with the Tavily-Search API, validating the fused-graph pipeline on genuine internal document and SOP corpora is a natural next step.
• Testing generalization beyond the repurposed ToolBench setting. The evaluation draws 3,000 randomly selected queries from ToolBench's G1, G2, and G3 splits; whether the dependency graph and PPR-based subgraph selection hold on other tool ecosystems and larger tool counts is not reported.
• Isolating what drives plan quality. The authors observe that results are relatively stable across models and may depend more on the retrieved subgraph than on model capacity; a fuller decomposition of embedding retrieval, PPR propagation, and graph fusion contributions would clarify this — only the PPR ablation (77% versus 68% for GPT-4o) is reported here.

Target Audience

Researchers and practitioners working on LLM agents, tool-augmented reasoning, and enterprise assistant deployment will benefit most — particularly those who need to move beyond general-purpose tools such as send email and make calendar into large, domain-specific tool catalogs. It is also relevant to engineers building knowledge-graph-backed retrieval systems who want a concrete example of fusing a structured dependency graph with document-derived knowledge, and to readers interested in how graph propagation (Personalized PageRank) compares with embedding-only retrieval for downstream plan generation.

Authors’ abstract

We present a framework for uncovering and exploiting dependencies among tools and documents to enhance exemplar artifact generation. Our method begins by constructing a tool knowledge graph from tool schemas,including descriptions, arguments, and output payloads, using a DeepResearch-inspired analysis. In parallel, we derive a complementary knowledge graph from internal documents and SOPs, which is then fused with the tool graph. To generate exemplar plans, we adopt a deep-sparse integration strategy that aligns structural tool dependencies with procedural knowledge. Experiments demonstrate that this unified framework effectively models tool interactions and improves plan generation, underscoring the benefits of linking tool graphs with domain knowledge graphs for tool-augmented reasoning and planning.