QMUL_AK at AISA-ArabicFC: Schema-Constrained Function Calling with ALLaM

AI summary

Overview

Research area: Natural Language Processing — Arabic large language models, function calling (tool use), and parameter-efficient fine-tuning.

Technical level: Intermediate. Readers will get the most from it with some familiarity with instruction-tuned LLMs, LoRA fine-tuning, and the function-calling task format (selecting a function from a schema and extracting its arguments).

Scope: A system description of a competition submission that adapts an Arabic-English 7B instruction model to schema-constrained function calling for Arabic, reporting development and blind-test results across three shared-task tracks.

What This Paper Is About

Large language models are increasingly expected to call external tools by emitting a structured function name plus its arguments, and doing this reliably in Arabic is harder than in English because of dialect variation and rich morphology. This paper describes a competition system for the AISA-ArabicFC 2026 shared task that teaches an Arabic-centric model to produce valid function calls that conform to a supplied tool schema, and that also works when the call must be preceded by reasoning. The goal is accurate function selection and, more importantly, exact argument extraction under dialect variation.

Key Contributions

1. A LoRA-based adaptation of the ALLaM-7B-Instruct model to Arabic function calling, trained using only the official task data rather than external corpora.
2. An Arabic schema-aware prompt design that presents the candidate tools to the model in a form suited to Arabic inputs.
3. A training recipe combining dialect and no-call resampling with a token-weighted objective, aimed at under-represented dialect inputs and at the decision of when not to call a tool at all.
4. Deterministic validation that checks generated calls against the candidate tool schema, plus a single output format that serves both the plain function-calling track (Track A) and the reasoning-augmented track (Track B).

Main Findings

• Function selection is close to solved: On the development set the system reaches 0.9982 function accuracy, and the authors conclude from their results that function selection is nearly saturated as a source of difficulty.
• Argument extraction is the bottleneck: Development-set argument exact match is 0.8920, far below function accuracy, and the paper identifies argument extraction as the main remaining source of error.
• Strong development scores: Overall development scores are 0.9345 for Track A and 0.9454 for Track B, with the reasoning-augmented track scoring higher.
• Blind test performance: On the blind test set the system scores 0.9253 (Track A) and 0.9380 (Track B), ranking 3rd on both submission tracks.
• Best dialect robustness: The system ranks 1st of 21 teams on the Track C dialect-robustness diagnostic with a macro-averaged score of 0.9075, and reports the smallest dialect gap in the shared task at 0.077.
• One output, two tracks: The same generated output format is used to serve both the function-calling and reasoning-augmented tracks, so the reasoning track does not require a separate model.

Methodology in Plain English

The team started from an existing Arabic-English instruction-following model and taught it the function-calling task through low-rank adaptation, a fine-tuning method that updates only a small set of added parameters rather than the whole model. The training signal came entirely from the official task data.

To make the model behave well on the specific task, they shaped both the input and the training data. On the input side, they wrote the prompt so the model sees the candidate tools as a schema in a form appropriate for Arabic. On the data side, they rebalanced the examples so that dialectal inputs and cases where the correct behaviour is to make no call were not under-represented during training. They also weighted the training objective at the token level, which changes how much different parts of the output contribute to the loss.

Finally, after the model generates a call, a deterministic check validates it against the candidate tool schema, catching outputs that do not conform. The paper reports development and blind-test scores and a dialect-robustness diagnostic; the abstract does not give dataset sizes, baseline comparisons, or ablation analyses, so the contribution of each individual component cannot be assessed from it.

Why This Matters

Research impact. The results separate two sub-problems that are often lumped together in tool-use benchmarks: choosing the right function, which this system shows can be driven to near-ceiling performance, and filling in the arguments correctly, which remains difficult. That framing is useful for anyone building or benchmarking function calling, and the reported dialect gap gives the Arabic NLP community a concrete robustness measure to improve against.

Real-world applications:

• Arabic-language customer service assistants that must look up orders, check balances, or book appointments by calling backend APIs, including for users writing in dialect.
• Voice or chat agents for Arabic-speaking markets where the same assistant has to handle both Modern Standard Arabic and regional varieties.
• Enterprise workflow automation in Arabic, where an agent routes a request to the correct internal tool with correct parameters.
• Arabic information-access tools such as prayer-time, weather, or transit lookups, where knowing when not to call a tool matters as much as calling the right one.

Industry relevance. The work shows that a 7B Arabic-centric model with parameter-efficient fine-tuning and a modest amount of task-specific engineering can be competitive in a shared task and win a robustness diagnostic, which is relevant to teams who cannot afford to train or serve very large models. Schema validation as a post-processing step is a cheap and practical safeguard for deployment.

Future Directions

• Improving argument extraction, which the authors identify as the dominant error source; a targeted approach to argument spans, normalization, or entity copying could close much of the remaining gap.
• Reducing the dialect gap further. The system already reports the smallest gap in the task, but 0.077 is not zero, so the question is which resampling or data-augmentation strategies close it.
• Determining which components actually drive the gains. The abstract reports the combined system but no ablations, so the separate value of the schema-aware prompt, the resampling, the token weighting, and the schema validation is an open question.
• Testing whether the recipe transfers to other Arabic-centric base models or to other languages with strong dialect variation, and whether the near-ceiling function-selection result holds outside this shared task's tool sets.

Target Audience

Competition participants and practitioners building Arabic tool-using assistants; researchers working on low-resource or dialectal function calling and on parameter-efficient fine-tuning of Arabic LLMs; and engineers who want a practical template for prompt design, data resampling, and schema validation in a production function-calling pipeline. Beginners will find the high-level framing accessible, but the specifics assume comfort with LLM fine-tuning terminology.

Authors’ abstract

We describe the QMUL_AK submission to AISA-ArabicFC 2026. The system adapts the Arabic-English ALLaM-7B-Instruct model to function calling with low-rank adaptation (LoRA). It combines an Arabic schema-aware prompt, dialect and no-call resampling, a token-weighted training objective, and deterministic validation of generated calls against the candidate tool schema. The same output supports Track A function calling and Track B reasoning-augmented calling. Training uses only the official task data. On the development set, the system achieves 0.9982 function accuracy and 0.8920 argument exact match, corresponding to overall A and B scores of 0.9345 and 0.9454. On the blind test set, the system reaches overall A and B scores of 0.9253 and 0.9380, ranking 3rd on both submission tracks and 1st of 21 teams on the Track C dialect-robustness diagnostic with a macro-averaged score of 0.9075 and the smallest dialect gap in the shared task (0.077). The results show that function selection is nearly saturated, while argument extraction remains the main source of error.