Error-Driven Prompt Optimization for Arithmetic Reasoning

AI summary

Overview

Research area: Artificial Intelligence — arithmetic reasoning over tabular data using code-generating language-model agents, with a focus on privacy-preserving, on-premises small language models (SLMs).

Technical level: Intermediate. The individual components (prompt engineering, clustering, a statistical significance test) are described plainly, but the paper also formalizes a task-decomposition model and a theoretical optimum for rule count.

Scope: A single paper that proposes an error-driven prompt-optimization algorithm, evaluates it on the TAT-QA financial table benchmark with the Qwen3 4B small model, and argues that a compact model can match or exceed GPT-3.5 Turbo.

What This Paper Is About

Large language models are unreliable at arithmetic on structured/tabular data, and prior work improved things by having the model write Python code instead of computing answers directly. The remaining problem is that the best-performing setups relied on API-based large models, which is unacceptable in regulated sectors where data cannot leave a secure environment. This paper asks how far a small, locally runnable model can be pushed by systematically finding its recurring arithmetic mistakes and converting each mistake pattern into a targeted prompt rule.

Key Contributions

1. An error-driven prompt-extending algorithm. Errors from a base prompt are featurized, clustered with HDBSCAN, and the largest cluster is used to motivate a new domain-specific prompt rule, which is kept only if it passes acceptance criteria (ΔEM above a threshold and a significant exact binomial McNemar test). The paper presents the method in pseudo-code (Algorithm 1) and notes a human-in-the-loop step for actually phrasing the rule.

2. A feature and error taxonomy for tabular arithmetic failures. Errors are described by five extracted features: calc_pattern, code_calc_pattern, scale_mismatch, value_match, and error_code (the corresponding section is titled error_type). Six error types are defined: Selection, Calculation, Scale, Sign, Syntax, and Runtime errors, and value_match distinguishes equal values, sign errors, NaN predictions, and other value mismatches.

3. A theoretical account of an optimal rule set. The paper formalizes a marginal gain per added rule ε(K), arguing that small models are overloaded and that rule conflicts become more likely as the rule count K grows, so there exists an optimal number of rules K_opt = argmax_K P_accuracy(K) beyond which performance stagnates or falls.

4. A concrete, accepted rule set and a containerized agent. Three rules survive to the final prompt: percentage change results in a percent scale; change in percentage is a subtraction; and year average must be computed as the average of the given year and the previous one, e.g. 2015_average = (2015_value + 2014_value)/2. The authors report building a specialized, containerized agent based on the Qwen model.

Main Findings

• Base SLM performance is poor, and code generation is the main lever. In the earlier benchmark reported here, Qwen3 4B scored 4.83% exact match naively and 53.72% with CGA plus table restructuring (a change of 48.89). GPT-3.5 Turbo went from 29.38% naively to 52.62%, and Gemma 3 1B and Llama 3.2 1B were effectively at zero (0.00%) naively, improving only to 1.66% and 1.01% respectively.

• Simplified prompts help most small models. With a simple prompt instead of a complex one, Qwen3 4B rose from 53.72% to 59.96% (a change of 6.24), Gemma 3n E4B from 29.78% to 42.05%, Gemma3 4B from 34.81% to 42.66%, and Mistral 7B from 17.91% to 41.45%, while Llama 3.2 3B dropped from 30.58% to 22.33% (change of -8.25).

• The error-driven loop produced three accepted prompt versions. Starting from 59.96% (V1), the first rule (percentage change yields percent scale) had a cluster of 22 items, 20 improved, ΔEM 90.91%, p = 2.1E-05, raising global EM to 64.59% (Δ = 4.63%). The second accepted addition (change in percentage is a subtraction) had N = 8, 5 improved, ΔEM 62.50%, EM 67.61% (Δ = 3.02%). The third accepted addition (year average rule) had N = 6, 5 improved, ΔEM 83.33%, p = 0.0625, EM 70.82% (Δ = 3.22%).

• Three candidate rules were rejected by the acceptance criteria. A rule about the order of minuend and subtrahend scored p = 0.25 and ΔEM 30.50% (global EM 67.40%, Δ = -0.20%); a rule about using exact category/header values and the number_value property scored p = 0.5 and ΔEM 28.57% (EM 69.22%, Δ = -1.61%); and a rule to use all relevant years for averages scored p = 0.125 and ΔEM 57.14% (EM 70.02%, global Δ = -0.80%).

• The final SLM beats the reference large model on exact match but not on raw values. Qwen3 4B reached 70.82% exact match versus 66.27% for GPT-3.5 Turbo (a change of 4.55%), but on the numerical value alone without scale accuracy it scored 77.06% versus 78.92% (a change of -1.86%). The authors read this as the SLM relying on rule tuning for final formatting while its underlying multi-step numerical reasoning remains less robust.

• Clustering behavior. The first clustering run selected min_cluster_size = 2 and min_samples = 1, yielding 45 clusters with a noise ratio of 0.140704; the error set contained 199 items and 50 was the first-quartile reference. The largest clusters held 34, then 31 (noise label -1), 8, 7, and 5 items. The selected cluster of 34 consisted entirely of scale errors where the expected scale was "percent". In the second iteration the same parameters gave 43 clusters and a noise ratio of 0.164773.

• The paper's own top-10 grid search is reported. The lowest noise ratio listed was 0.130653 at min_cluster_size = 15 and min_samples = 3 (4 clusters), followed by 0.140704 for (2,1), (15,2) and (10,2). The authors nonetheless chose (2,1) because 45 clusters came closest to the quartile target of 50.

• Implementation specifics. Runs used Ollama version 0.11.4 accessed through its API from Jupyter Notebook, with the model tag qwen3:4b-q4_K_M, id 2bfd38a7daaf, size 2.6 GB, a 40k context window, and 4-bit K-series mixed quantization. Thinking behavior was controlled with the /no_think flag because thinking and non-thinking modes were not yet split in that version.

• A reported inconsistency worth noting. The narrative for the second accepted rule states a McNemar p-value of 0.0265 while Table 7 lists 0.0625 for that same step; the paper does not reconcile the two.

• Dataset filtering numbers. TAT-QA is described as containing financial reports in tabular format with related paragraphs; the filtering pipeline reduced tables from 278 to 215 and questions from 1668 to 497. WikiTableQuestions was rejected for having derivation annotations only for a limited set of questions and no answer categorization; OTT-QA was rejected for lacking both.

• Calculation pattern frequencies in the error analysis include #-# (134), (#-#)/# (100), (#+#)/# (89), #/# (49), (#+#+#)/# (26), #+# (21), #/#-# (9), -#-# (8), #%-#% (7), -(#+#)/# (5), (#%+#%)/# (5), #+#+# (4), [(#+#)/#]-[(#+#)/#] (4), and other (36).

Methodology in Plain English

The authors treat arithmetic question answering as a code-writing task. Instead of asking the model for a number, they restructure each table into a list of annotated values and ask the model to write a Python run(value_list) function that returns a tuple of the number and its scale (one of thousand, million, billion, percent, or an empty string). A deterministic executor runs that code, so arithmetic mistakes cannot come from the model's mental math.

They then attack the residual mistakes in a loop. On a test set of 497 filtered questions, they run the current prompt, collect every question it got wrong, and describe each error with features such as the required calculation shape, the calculation shape actually present in the generated code, whether the scale was wrong, whether the numeric value matched, and the error type. These feature vectors are one-hot encoded with a DictVectorizer into a dense binary matrix, and HDBSCAN clusters them using Hamming distance, with min_cluster_size and min_samples chosen by a grid search sorted by noise ratio. The single largest cluster is inspected to identify a shared root cause, a human writes a short rule addressing that cause, and the rule is tested on the cluster's questions.

A rule is kept only if it clears two bars: the change in exact match exceeds the threshold (the algorithm text uses ΔEM > 50% and the acceptance criteria state ΔEM >= 0.5), and the exact binomial McNemar test on paired old/new predictions yields p <= 0.0625. If the pair of top clusters both fail, the search stops. The authors also visualize error structure as a cross-tab heatmap of calculation pattern by error type across iterations.

Why This Matters

Impact on research. The paper reframes the arithmetic weakness of small models as a consequence of inference strategy rather than a hard capability limit, and offers a repeatable, auditable procedure for improving a fixed local model without fine-tuning. It also contributes a formal argument that more prompt rules is not better, which is directly testable and generalizes beyond arithmetic. The reported result that the SLM's value-only accuracy still trails GPT-3.5 Turbo (77.06% vs 78.92%) is a useful counterweight to the headline exact-match win.

Real-world applications.

• Financial statement analysis, where analysts ask arithmetic questions about balance sheets and income statements without data leaving the firm.
• Healthcare and other regulated tabular records where on-premises processing is a compliance requirement.
• Any setting that needs an auditable trail, since the generated Python code is inspectable and the prompt rules are human-readable.
• Deployment scenarios needing containerized, scalable agents, which the authors report building on the Qwen model.

Industry relevance. The core claim is economic as much as technical: a 4B-parameter model running on local hardware with model-specific prompt rules can reach 70.82% exact match where the API-based reference reaches 66.27%, removing a dependency on external API providers in privacy-constrained sectors.

Future Directions

• Generalization to unseen domains. The authors explicitly flag as a critical open question how a small model with a fine-tuned rule set can generalize to new domains without repeating the iterative error-analysis cycle.

• Automating the human-in-the-loop step. The algorithm requires an expert to formulate each domain-specific rule from an error cluster; the paper positions the method as automatable later, but does not automate it here.

• Reconciling the statistical reporting. The mismatch between the reported p = 0.0265 and the tabulated p = 0.0625 for the second accepted rule, plus a rejected rule whose global ΔEM is negative (-0.80%), invites closer scrutiny of the acceptance procedure and the ΔEM thresholds.

• Closing the value-accuracy gap. Since the SLM's numerical value accuracy (77.06%) still trails GPT-3.5 Turbo (78.92%), further work is needed on the core multi-step reasoning rather than format and scale selection.

• Extending the feature taxonomy. The feature list mixes a name used only in the bullet list (error_code) with a section definition (error_type), which leaves the exact feature set in need of clarification for replication.

Target Audience

This paper is most useful to applied machine learning engineers and data scientists building on-premises or privacy-constrained analytics agents, to researchers working on tabular question answering and code-generation agents, and to technical decision-makers in finance and healthcare evaluating whether a small local model can replace an API-based large model. Readers interested in prompt-engineering methodology and in the theory of rule-set size will find the algorithm and the K_opt argument directly relevant. It is not an introductory NLP paper; some familiarity with exact match metrics and clustering is assumed.

Authors’ abstract

Recent advancements in artificial intelligence have sparked interest in industrial agents capable of supporting analysts in regulated sectors, such as finance and healthcare, within tabular data workflows. A key capability for such systems is performing accurate arithmetic operations on structured data while ensuring sensitive information never leaves secure, on-premises environments. Here, we introduce an error-driven optimization framework for arithmetic reasoning that enhances a Code Generation Agent (CGA), specifically applied to on-premises small language models (SLMs). Through a systematic evaluation of a leading SLM (Qwen3 4B), we find that while the base model exhibits fundamental limitations in arithmetic tasks, our proposed error-driven method, which clusters erroneous predictions to refine prompt-rules iteratively, dramatically improves performance, elevating the model's accuracy to 70.8\%. Our results suggest that developing reliable, interpretable, and industrially deployable AI assistants can be achieved not only through costly fine-tuning but also via systematic, error-driven prompt optimization, enabling small models to surpass larger language models (GPT-3.5 Turbo) in a privacy-compliant manner.