Efficient Tool-Calling Multi-Expert NPC Agent for Commonsense Persona-Grounded Dialogue

AI summary

Overview

Research area: Natural Language Processing — conversational agents, tool-calling large language models, and parameter-efficient fine-tuning for interactive NPCs (non-player characters).

Technical level: Intermediate. The paper assumes familiarity with large language models, LoRA adapters, and function/tool-calling APIs, but the architectural ideas are explained in accessible terms.

Scope: A single-author system description submitted to the Commonsense Persona-Grounded Dialogue Challenge 2025, describing a three-expert Qwen3-based agent that both converses and executes tools within strict latency and memory limits.

What This Paper Is About

Interactive NPCs need to do two things at once: hold a natural, persona-consistent conversation and take concrete actions in their environment (checking prices, equipping items, searching inventories). Most systems are good at one or the other, and doing both under a hard 7-second response budget on a fixed GPU is difficult. The paper's goal is to build a single NPC agent, under the constraints of the Commonsense Persona-Grounded Dialogue Challenge 2025, that decides when to call a tool, executes it, and folds the result back into fluent dialogue — without exceeding the compute budget.

Key Contributions

1. A multi-expert architecture that splits the NPC's job into three specialized modules sharing one base model: ToolLoRA (decides whether a tool is needed and emits tool calls), PersonaLoRA (interprets tool responses and writes the final reply), and NoLoRA (the unmodified Qwen3 instruct model for direct replies when no tool is needed). LoRA made this feasible because several lightweight adapters can be loaded against one base model far more cheaply than deploying multiple full models.

2. A three-tier data augmentation strategy using GPT-o4-mini-high as a generation tool, which expanded the official challenge datasets by nearly 300 percent while preserving 1:1 conversation structure and tool-calling positions. The tiers ranged from low aggressiveness (paraphrase only) through medium (renaming tools and arguments, e.g. check_amount to check_quantity) to high aggressiveness (changing roles and worldviews, e.g. a "Merchant in Medieval town" into an "Armor supplier on Mars secret military base").

3. A three-stage dataset validation suite (structural validation, schema compliance, semantic integrity) that acted as a quality gate before training on the enlarged dataset.

4. Inference-time token optimizations, including pre-filling the <tool_call>\n"name": " prefix to save 6 tokens per turn, early stopping when the dummy reply tool is detected to save an additional 9 tokens, hiding tool lists from the direct-reply model, and injecting hard-coded <think>...</think> reasoning content to steer response style.

Main Findings

• Competition placement: The system ranked 1st on the public leaderboard and 2nd overall after human evaluation and final automatic evaluation in the CPDC 2025 competition.

• Per-task scores: Task 1 scored 0.664 on the public leaderboard (2nd) and 0.632 in final evaluation (3rd). Task 2 scored 0.613 on the public leaderboard (1st); no final-evaluation score is reported for Task 2. Task 3 scored 0.639 on the public leaderboard (1st) and 0.626 in final evaluation (2nd).

• Latency well under budget: Average response time was 3 seconds, which the paper describes as 57 percent under the 7-second limit.

• Memory headroom: The system used under 30GB of the 48GB VRAM available on the L40S GPU, or 62.5 percent of available memory.

• Augmentation helps smaller models most: Qwen3-1.7B improved from approximately 0.5 to 0.6+ (an increase of 0.1+ points) on Task 1, a relative improvement of 20 percent or more. Qwen3-14B improved by approximately 0.030 to 0.040 points on Task 1.

• Model selection rationale: Qwen3 was chosen over alternatives because of native tool-calling support, a well-defined interaction format, and Unsloth support. The paper's compatibility table (as of August 14, 2025) lists Qwen3 at 8B rank #18 on the Berkeley Function-Calling Leaderboard, ToolACE2 at 8B #10 and watt-tool at 8B #15 (both lacking Unsloth support), Gemma3 at 12B #78 (prompt-based tool calling only, Unsloth-supported), and Llama3 at 8B #83 (function-calling, Unsloth-supported). Qwen3's reasoning mode was deliberately disabled because its long reasoning traces would have violated the 7-second constraint.

• Size–performance trade-off: Qwen3-14B was selected as the balance point between capability and resource use. Testing with Qwen3-1.7B achieved 0.6 on Task 1, and the architecture supports any Qwen3 variant from 0.6B up to the largest in the family.

• Design-driven failure modes were identified and fixed: Base Qwen3 occasionally omitted tool-returned information from replies (motivating PersonaLoRA), PersonaLoRA tended toward overly brief responses (addressed with forced thinking content), and ToolLoRA sometimes emitted spurious arguments for the reply tool, traced to a dataset issue that was corrected.

Methodology in Plain English

The researchers started from the official challenge datasets (Task 1 and Task 2 JSON files; Task 3 had no dedicated dataset because it combines the two). Those files were not self-contained — the tools and actions available at each turn had to be retrieved from separate Python files and converted into Qwen3-compatible schemas.

Because the original training data was small, they used GPT-o4-mini-high as a data generation tool to produce paraphrased and restructured variants at three levels of aggressiveness, keeping the conversation skeleton identical (the paper's phrase: "We kept the skeleton the same but changed the skin.") and deliberately avoiding fully independent conversation flows so as not to drift from the dataset's distribution. The enlarged dataset was then passed through three validation scripts checking message structure, JSON schema compliance, and argument-to-schema semantic consistency.

Training used LoRA rather than full fine-tuning. An early attempt with a single adapter handling both tool calling and dialogue degraded conversational quality, which prompted the split into separate experts. Each of the two LoRA adapters was trained separately on Google Colab using Unsloth, for 2 epochs, typically taking 1 to 2 hours on a single A100 GPU. Only training and validation losses were monitored; other metrics were skipped because they would have required processing logits and consumed extra time and VRAM.

At inference, a user message first goes to ToolLoRA to decide whether a tool is needed. If the dummy reply tool is emitted, the message is routed to unmodified Qwen3 (NoLoRA) for a direct answer. Otherwise the tools are executed and the responses are fed to PersonaLoRA, which writes the final reply. Only the most recent tool calls and responses are kept in conversation history during training, so the model does not learn to depend on tool outputs that would not be available at the current inference turn.

Why This Matters

Impact on research: The paper is a concrete demonstration that decomposing an agent into specialized adapters over a shared base model can beat a single monolithic fine-tune under tight latency and memory budgets. It also provides an unusually candid account of a data-augmentation strategy's size-dependent payoff — larger gains for small models, smaller gains for large ones — which is a useful empirical data point for anyone deciding whether augmentation is worth the effort for a given model scale.

Real-world applications:

• Game and virtual-world NPCs that must both talk in character and take real actions (trading, equipping, navigating) inside a strict frame budget.
• Customer service and commerce assistants that check prices, quantities, or inventory and fold the results into a natural reply rather than returning raw data.
• On-premise or edge deployments of conversational agents where 48GB of VRAM is the ceiling and response latency is user-visible.
• Energy-aware AI deployment, since the paper frames its sub-30GB footprint and token-level savings as reducing energy consumption and carbon footprint relative to less optimized approaches.

Industry relevance: The 7-second, 48GB constraint envelope is representative of production inference budgets rather than a lab setting, so the reported efficiency figures and the token-saving tricks (prefix pre-filling, early stopping) are directly transferable to deployed systems. The permissive-licensing and Unsloth-compatibility screening in the model selection section also reflects the practical constraints teams face when choosing a base model.

Future Directions

1. Dynamic knowledge graph integration. The authors propose using smaller Qwen3 models (such as the 1.7B or 4B variants) with batched parallel inference to pre-build structured knowledge graphs before injecting them into the NPC prompt. They note that because the current system uses under 30GB of the 48GB VRAM, roughly 18GB remains available for this parallel knowledge processing.

2. Constrained generation for tool calls. Integrating grammar-enforcement libraries such as Outlines or Guidance is proposed to guarantee 100 percent reliable tool calls by enforcing JSON schema compliance at the token level, eliminating malformed calls and removing the need for invalid-call filtering and post-processing.

3. Resolving the remaining quality issues. The paper lists information omission, overly brief PersonaLoRA responses, and spurious arguments for the reply tool as addressed but ongoing concerns; whether the fixes generalize beyond the competition setting is an open question.

4. The reply-name collision risk. The early-stopping optimization depends on detecting a dummy tool named reply; the authors acknowledge this introduces a brittleness risk if a real tool with that name existed, and that this is only acceptable in a controlled tool namespace.

Target Audience

This paper is most useful to applied NLP engineers and agent-system builders who need to fit multi-capability conversational agents into fixed latency and memory budgets, and to researchers interested in multi-adapter expert decomposition, synthetic data augmentation for tool-calling tasks, or competition-style benchmark engineering. It is also relevant to game developers evaluating tool-calling LLMs for NPC behavior, and to practitioners deciding between Qwen3, Gemma3, Llama3, ToolACE2, and watt-tool on the basis of tool-calling quality and fine-tuning toolchain support. Beginners will follow the architecture at a high level but may need background reading on LoRA and function calling to follow the optimization details.

Authors’ abstract

We present a multi-expert system for creating Non-Player Characters (NPCs) capable of both natural dialogue and contextual action execution in interactive environments. Using Qwen3 as the base model and Low-Rank Adaptation (LoRA) adapters, we instantiate three specialists: tool calling, tool-response interpretation, and direct dialogue. Our system comfortably meets the computational efficiency requirements, delivering fast responses and maintaining modest resource usage on L40S GPUs. In the Commonsense Persona-Grounded Dialogue Challenge 2025, our method ranked second overall. Code available at: https://github.com/MahammadNuriyev62/CPDC-challenge-2025-solution/