Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

AI summary

Overview

Research area: Natural language processing — specifically the robustness of LLM-based tool calling (function calling) when user queries are written in languages other than English, with an emphasis on agentic systems that invoke external APIs.

Technical level: Intermediate. Readers should be comfortable with the basics of function calling / tool calling, JSON-style parameter schemas, and benchmark evaluation protocols (exact-match scoring), but no specialized mathematics or training-method background is required.

One-sentence scope: The paper introduces a diagnostic benchmark, MLCL, built by extending the Berkeley Function Calling Leaderboard (BFCL) into Chinese, Hindi, and Igbo, and uses a fine-grained error taxonomy to show that multilingual tool-calling failures are dominated by execution-interface violations rather than misunderstanding of user intent.

What This Paper Is About

Large language models are increasingly used as agents that call external tools through structured function calls, and existing benchmarks report strong performance — but those benchmarks almost always assume the user query is in English. This paper asks what happens when the same user intent is expressed in Chinese, Hindi, or the low-resource language Igbo while the tool interface remains English-only and language-invariant.

The core problem the authors identify is that models often select the right function and produce semantically appropriate arguments, yet still generate tool calls that cannot be executed because parameter values are written in the user's language. They call this failure mode parameter value language mismatch, and the paper's goal is to isolate, quantify, and partially mitigate it.

Key Contributions

1. A diagnostic benchmark (MLCL) for tool-calling robustness under multilingual user queries, covering Chinese, Hindi, and Igbo, constructed so that only the natural-language query varies while the execution interface stays fixed and English-only.

2. Identification of parameter value language mismatch as a dominant failure mode in multilingual tool calling, occurring despite correct intent understanding and correct tool selection.

3. A fine-grained error taxonomy that separates execution-level violations from semantic errors, revealing that high-resource languages (Chinese and Hindi) are dominated by tool-calling convention issues and information loss during translation, whereas the low-resource language Igbo shows a larger share of errors from confusion in user-query semantics.

4. An empirical evaluation of inference-time mitigation strategies — prompt-level instruction (PT), pre-translation (PRE), and post-translation (POST) — showing that these substantially reduce language-induced errors but never fully restore English-level performance.

Main Findings

• Fully translated queries sharply increase execution-level errors. Moving from the English reference setting (NT) to fully translated (FT) queries produces a pronounced increase in execution failures across all three languages, dominated by parameter value language mismatch. Models frequently copy non-English tokens from the query directly into parameter values, violating the English-only execution interface; in most cases those values remain semantically correct.

• The error composition differs systematically by language. Parameter value language mismatch is most prevalent for Chinese, followed by Hindi, and least frequent for Igbo. The authors interpret this as models being more willing to reuse tokens from high-resource languages as parameter values and avoiding this for lower-resource languages — so Igbo shows fewer language-mismatch errors but a higher proportion of semantic misunderstanding errors.

• Partial translation isolates interface violations from comprehension errors. For several models, including GPT-5 and Llama 3.1–70B, the partially translated (PAR) setting shows fewer execution-level errors than FT and in some cases matches or outperforms the English reference. Because PAR preserves English parameter strings while translating surrounding context, it removes parameter value language mismatch without substantially changing query meaning — suggesting that many models interpret non-English queries about as reliably as English ones once interface violations are controlled.

• Semantic perturbations hurt most when strict surface-form matching is required. Paraphrasing (PARA) and synonym substitution (SYNO) substantially increase execution failures in the English (NT) setting, because altered surface forms no longer match expected parameter values. They have limited additional effect in fully translated settings, where language mismatch already dominates, and intermediate effects in partially translated settings, where perturbations can replace preserved English parameter strings with non-English equivalents.

• Inference-time mitigation helps but is insufficient. Prompt-level instruction (PT) reduces parameter value language mismatch relative to FT, but compliance is inconsistent and residual mismatches remain common. Pre-translation (PRE) and post-translation (POST) further reduce mismatch, with PRE generally more effective because it has access to the full query context. None eliminates execution failures, and translation introduces semantic drift and surface-form normalization that create new execution-level errors under strict matching.

• Mitigation behaves differently for the low-resource language. For Igbo, translation-based mitigations (PT, PRE, POST) provide limited benefit and can even increase errors, because parameter value language mismatch is rare for Igbo even without explicit prompting. Remaining Igbo failures are dominated by query understanding errors, including unstable semantic grounding and imprecise mapping between natural-language expressions and the target parameter schema.

• Reported metrics are compositional rather than a single accuracy number. The paper reports overall error rate together with a structured error breakdown, and the results described in the main text are presented through figures rather than numerical tables. Specific accuracy or error-rate percentages are not reported in the available content.

Methodology in Plain English

The authors start from an existing English tool-calling benchmark — BFCL V4, using the BFCL_v4_multiple.json subset of single-turn queries paired with multiple candidate functions — and deliberately keep the execution interface fixed so that any performance drop can only be attributed to the language of the query. Tool schemas, function names, parameter keys, and tool descriptions are never translated.

They then vary the input along two independent dimensions:

• Query language composition: NT (the original English query), PAR (the surrounding context is translated but ground-truth parameter values are kept in English, producing mixed-language queries), and FT (the entire query is translated).
• Semantic perturbation: NO (no perturbation), PARA (paraphrase with altered phrasing but unchanged meaning), and SYNO (synonym substitution on individual words).

Translations were generated by GPT-5 and manually verified by group members who speak Chinese, Hindi, and Igbo. Verification focused on intent preservation rather than literal word-level correspondence — for instance, the authors accept a translation of "queen-size bed" that is semantically equivalent to "a large bed." For the partially translated dataset, they verified both semantic invariance and that ground-truth parameter keywords stayed in English.

Twelve model configurations were evaluated across multiple families and scales: GPT-5, GPT-5 mini, and GPT-5 nano; DeepSeek V3.2; meta-llama/Llama-3.1-8B-Instruct and meta-llama/Llama-3.1-70B-Instruct; Qwen/Qwen3-8B, Qwen/Qwen3-14B, Qwen/Qwen3-30B-A3B, Qwen/Qwen3-32B, and Qwen/Qwen3-Next-80B-A3B-Instruct; and ibm-granite/granite-4.0-h-tiny and ibm-granite/granite-4.0-h-small. All were run through their officially supported tool-calling interfaces and recommended decoding configurations, with the team implementing its own input/output format conversion rather than using the official BFCL protocol implementation.

Evaluation follows the BFCL protocol, requiring exact matching of function names, parameter keys, and parameter values — a strict criterion that directly reflects whether a call can be executed without system intervention. The authors extended it with an error taxonomy ordered by severity: syntax errors (unparseable outputs), function-level errors (schema violations such as wrong function names or missing parameters), parameter value language mismatch (split into three subtypes by semantic correctness: mismatch plus wrong value, mismatch plus relevant-but-incorrect, and mismatch plus same meaning), and finally non-mismatch parameter errors (wrong value, relevant but incorrect, exactly same meaning).

Finally, they probe three lightweight inference-time fixes that require no retraining: PT (instruct the model in the prompt to output parameter values in English), PRE (translate the query into English before tool calling, using the same model for translation and tool calling), and POST (translate generated parameter values into English after generation but before execution).

Why This Matters

Impact on research: The paper argues that multilingual tool-calling degradation is an execution-interface problem, not a language-understanding problem, which reframes how robustness should be measured. Standard accuracy metrics on English benchmarks hide this failure entirely, and the proposed taxonomy gives the community a way to separate semantic errors from operational ones. Because the failure is reproduced consistently across model families, the authors present it as a shared system-level mechanism rather than a weakness of any particular model.

Real-world applications:

• Customer-facing assistants serving non-English users, where a semantically perfect but non-executable call causes a silent failure the user never understands.
• Travel, weather, and booking agents that resolve place names, dates, and identifiers into strict string formats — the exact scenario in the paper's motivating example, a get_weather() call with a "location" and "days" parameter.
• Enterprise automation and workflow orchestration pipelines where model output is fed directly into APIs with no human review.
• Any globally deployed agent using a shared, English-convention tool registry across many locales.

Industry relevance: The findings suggest that adding more language coverage to a model's training does not by itself fix tool calling, and that the fix is more likely to live in the interface layer — normalizing inputs, enforcing output conventions, or post-processing parameter values. The paper also notes the risk of silent non-executable calls or incorrect downstream actions, which the authors state may disproportionately affect non-English users. The work was partially supported by NSF award #2442477 and #2550203, with acknowledgment of Amazon Research Awards, Cisco Faculty Research Awards, and Toyota Faculty Research Awards, plus API credits from Google and OpenAI.

Future Directions

• Move beyond inference-time patching. The authors conclude that multilingual tool calling cannot be solved by inference-time interventions alone and call for execution-aware system design that aligns natural-language variability with programmatic interfaces — but do not propose what that design should be.

• Extend the diagnostic beyond single-turn settings. The paper explicitly limits itself to single-turn queries with predefined function interfaces (the BFCL_v4_multiple.json subset) and states that results should be interpreted within that setting; multi-turn dialogue, live execution, and dynamic tool registries remain open.

• Broaden language coverage. Only three non-English languages are studied, and the authors state the set does not cover all linguistic families or writing systems, framing the selection as illustrative rather than exhaustive.

• Address low-resource-language comprehension separately. For Igbo, translation-based mitigation had limited benefit and could degrade performance, with remaining failures dominated by query understanding errors — a problem distinct from the interface-mismatch problem the mitigation strategies target, and one not resolved here.

• Reconsider the interface assumption itself. The paper's notion of execution correctness assumes tool interfaces expect English parameter values; the authors frame this as a common design choice they analyze rather than endorse, leaving open whether multilingual-aware tool interfaces would be a better answer.

Target Audience

Researchers and practitioners working on LLM agents, function calling, and multilingual NLP will get the most from this paper, particularly those building or evaluating tool-using systems for non-English-speaking users. It is also relevant to benchmark designers who want to move past aggregate accuracy toward error taxonomies that distinguish semantic from execution-level failures, and to engineers responsible for the interface layer between a model's natural-language output and a programmatic API. The paper is written at an intermediate level: it assumes familiarity with tool-calling formats and strict-match evaluation, but its central insight — that correct intent can still produce a non-executable call — is accessible without deep technical background.

Availability: The authors state that all code and datasets are available at https://anonymous.4open.science/r/multilingual_robustness_tool_calling-CA44. The paper is distributed under a CC BY-NC-SA 4.0 license.

Authors’ abstract

Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls. While recent work reports strong tool-calling performance under standard English-centric evaluations, the robustness of tool calling under multilingual user interactions remains underexplored. In this work, we introduce MLCL, a diagnostic benchmark, and conduct a systematic evaluation of multilingual tool calling across Chinese, Hindi, and the low-resource language Igbo. Through fine-grained error analysis, we show that many failures occur despite correct intent understanding and tool selection. We identify parameter value language mismatch as a dominant failure mode, where models generate semantically appropriate parameter values in the user's language, violating language-invariant execution conventions. We further evaluate several inference-time system strategies and find that while these strategies substantially reduce language-induced execution errors, none of them can fully recover English-level performance.