From Abstract to Contextual: What LLMs Still Cannot Do in Mathematics

AI summary

Overview

Research area: Large language model evaluation and mathematical reasoning (cs.AI / cs.CL-adjacent).

Technical level: Intermediate. The framing is intuitive, but the paper assumes familiarity with benchmark evaluation, fine-tuning regimes, and conditional-probability metrics.

Scope: The paper introduces ContextMATH, a benchmark that rewrites AIME 2024, AIME 2025, and MATH-500 problems into narrative scenarios to test whether 61 LLMs can formulate and solve mathematics when it is not presented as an abstract problem.

What This Paper Is About

LLMs now score near the top on competition math benchmarks such as AIME, yet that success does not transfer to real-world settings where the mathematics is buried inside a descriptive scenario. The authors call this missing capability contextual mathematical reasoning: the ability to first extract the mathematical core from a narrative and then solve it. Their goal is to measure that gap precisely, identify where models fail, and test whether training can close it.

Key Contributions

1. A new task framing. The paper names and formalizes contextual mathematical reasoning — formulating and solving a mathematical problem when it is embedded in narrative scenarios with indirect or layered conditions — and argues this has been largely unexamined relative to abstract benchmark performance.

2. A new benchmark (ContextMATH). Each source problem (AIME 2024, AIME 2025, and MATH-500 filtered to difficulty ≥ 3) is instantiated in two variants: Scenario Grounding (SG), which embeds the problem in a realistic narrative without changing the reasoning core, and Complexity Scaling (CS), which hides explicit conditions inside simple sub-problems. CS is not built for MATH-500. Average problem lengths are 133 words (SG) and 176 words (CS).

3. Large-scale evaluation of 61 models. 46 open-source models (base, SFT, and RL-tuned) and 15 proprietary models are evaluated, including GPT-5, DeepSeek-R1, o3, Gemini 2.5 Pro, and QwQ-plus. Open-source models are sampled 16 times per problem; proprietary models get a single pass. A separate SG Avg@3 column reports the average over two additional generated and annotated AIME 2024-SG sets.

4. Diagnosis of failure modes and training experiments. Using a formulation-accuracy/necessity/sufficiency framework plus error categorization, the paper isolates formulation as the dominant failure mode, then tests end-to-end fine-tuning on scenario data and a dedicated formulation model.

Main Findings

• Contextual variants cause sharp, consistent accuracy drops. On average, open-source models decline by 13 points on SG and 34 points on CS, while proprietary models drop by 13 and 20 points respectively.

• Concrete examples of the drop. Qwen3-32B falls from 81.25% on AIME 2024 to 67.92% (SG) and 57.08% (CS). DeepSeek-R1 drops from 86.67% to 73.33% (SG) and 53.33% (CS) on AIME 2025. DeepSeek-R1-0528-Qwen3-8B falls from 75.0% on AIME 2024 to 39.6% on its CS variant, and QwQ-plus from 86.7% to 46.7%. Even GPT-5, with an estimated 1.8T parameters, drops 26% on AIME 2025-CS.

• Errors are dominated by problem formulation, not calculation. GPT-5 was used to categorize errors made by DeepSeek R1, Gemini 2.5 Pro, and Qwen3-32B on SG and CS sets, restricted to problems those models solved correctly in the original form. Formulation errors account for roughly 80% of failures across all three models, far exceeding calculation, logic, or other error types. One cited case: DeepSeek R1 fails to see that "the time for a gear to complete one rotation is adjustable, but it must not exceed six rotations per minute" implies x ≥ 10 when x is seconds per rotation.

• Formulation accuracy declines as the underlying problem gets harder. Qwen3-4B goes from 84.8% formulation accuracy on MATH to 64.6% on AIME24 to 47.1% on AIME25.

• Scale helps but does not solve formulation. Qwen3-0.6B averages 42.8% formulation accuracy versus 75.0% for Qwen3-32B, yet even GPT-5 averages only 81.4% formulation accuracy and remains below 75% on AIME25.

• Correct formulation is highly necessary for correct reasoning. Formulation Necessity, defined as P(F = True | R = True), is consistently above overall accuracy across all models. Qwen3-4B shows 61.6% accuracy versus 79.2% necessity; GPT-5 shows 81.4% versus 85.6%.

• Reasoning remains a second, separate bottleneck. Formulation Sufficiency, P(R = True | F = True), improves with scale but lags both accuracy and necessity. Qwen3-8B records 73.8% accuracy and 83.8% necessity but only 60.7% sufficiency, and GPT-5 reaches only 82.7% sufficiency.

• Scale mitigates but does not eliminate contextual failure. In the OpenMath-Nemotron series on AIME 2024-CS, the 1.5B model drops 77%, compared to 43% and 51% for the 14B and 32B models.

• Initial SFT helps; further specialization does not. Qwen2.5-Math-7B → R1-Distil-Qwen-7B boosts AIME accuracy and reduces the contextual drop, but later-stage models such as R1-Distil-Qwen-7B → AceMath-RL-Nemotron-7B and Qwen3-8B → DeepSeek-R1-0528-Qwen3-8B improve on AIME 2024/25 yet not on SG or CS, often with larger drops. The authors attribute this to over-specialization to canonical formats.

• Scenario fine-tuning improves results, especially when mixed with abstract data. Training data came from DeepMath-103K (3,000-character-scale heterogeneous math problems; each with three DeepSeek R1 solutions, one randomly sampled) plus 50k validated synthetic scenarios. Qwen3-14B-Base solves only 11.0% of AIME 2024-SG problems, rising to 52.5% with the mixed SFT regime (SFT_Mix). Average gains across all evaluation sets: 23.3 → 48.8 (+25.5%) for Qwen3-4B-Base, 27.3 → 55.6 (+28.3%) for Qwen3-8B-Base, and 29.4 → 61.3 (+31.9%) for Qwen3-14B-Base.

• Even after mixed training, the gap is far from closed. Models with SFT_Mix still solve under 40% of AIME 2024/25-CS problems.

• Training a dedicated formulation model is counterproductive. Using Qwen3-8B and Qwen3-14B as formulation models, best accuracy came from solving scenarios directly without a formulation stage (53.9 for the 8B reasoning model, 57.7 for the 14B). An untuned formulation stage produced a slight drop (48.9/53.4 and 51.8/56.2), and a fine-tuned formulation model caused performance to collapse (20.8/22.3 and 21.8/24.6). Training was stable with no signs of overfitting, leading the authors to conclude formulation is hard to learn from scenario–original pairs alone.

• Gains generalize beyond ContextMATH. Models also improved on AMC23 and on Math-Perturb's Simple and Hard variants, indicating that scenario training does not degrade abstract reasoning and can improve robustness to distribution shift.

Methodology in Plain English

The authors could not collect large numbers of genuine real-world applied math problems, so they took problems with known correct answers from AIME 2024, AIME 2025, and MATH-500 and rewrote each one in two controlled ways. In Scenario Grounding, the abstract elements are mapped onto real objects and interactions — a variable becomes "the initial number of barrels of oil" — while the underlying math is unchanged. In Complexity Scaling, explicit conditions are replaced by the outputs of small sub-problems: instead of being told there are 25 indicator lights, the problem states that the total number of unique pairs is exactly 300, which yields the same number.

Generation used structured prompts driving o1-mini through iterative scenario creation, self-verification, and revision. Human experts then reviewed everything. Three experts with advanced Computer Science degrees and competitive mathematics backgrounds independently judged each item on narrative plausibility, solvability, and mathematical equivalence to the original. Scenarios were also tested on Gemini and GPT-5, and any failure was diagnosed to see whether it came from genuine model weakness or from an ambiguous problem description.

For evaluation, models sampled 16 solutions per problem on the open-source side and single-pass on the proprietary side. To go beyond accuracy, the authors prompted each model to output only the bare mathematical formulation of a scenario, stripped of narrative, and had o1-mini act as an automated judge of equivalence. They manually annotated Qwen3-14B and Qwen3-32B outputs and found the judge agreed with human judgment in over 90% of cases. This let them define Formulation Accuracy, Formulation Necessity (P(F = True | R = True)), and Formulation Sufficiency (P(R = True | F = True)) to characterize the directional relationship between understanding a problem and solving it.

Finally, they ran training experiments on the Qwen3-Base series comparing three regimes with equal step counts: original DeepMath data only (SFT_Ori, 50k), synthetic scenario data only (SFT_Syn, 50k), and a balanced mixture (SFT_Mix, 100k). A separate experiment chained a fine-tuned formulation model into a reasoning model to see whether separating the two skills helps.

Why This Matters

The paper argues that near-perfect benchmark scores can mask a serious weakness: models may have learned to pattern-match canonical problem formats rather than to understand mathematics as it actually appears in practice. By showing that formulation errors — not arithmetic or logic — account for roughly 80% of failures, it redirects attention from "can models compute?" to "can models read a situation and know what the math is?"

Real-world applications the paper points toward:

• Financial analysis, where quantitative relationships must be inferred from business narratives rather than given as equations.
• Scientific research, where constraints and quantities are described in prose and indirect statements.
• Engineering design, where specifications arrive as descriptions of physical systems and interactions.
• Any domain that pairs LLM agents with descriptive task specifications, since the same abstract-to-contextual transformation approach is stated to apply beyond mathematics and beyond these datasets.

Industry relevance: The finding that advanced SFT and RL specialization improves canonical benchmark scores while failing to improve contextual performance is a direct challenge to current post-training pipelines. It suggests that progress measured on standard math benchmarks may not indicate a model is more useful in deployed, scenario-heavy workflows, and that scenario data should be part of the training mixture.

Future Directions

• Finding training signals that actually teach formulation. The paper shows that paired scenario–original supervision fails to produce a useful dedicated formulation model and explicitly leaves exploration of other approaches to future work. What kind of supervision could work remains open.

• Closing the remaining CS gap. Even the best mixed-SFT models solve under 40% of AIME 2024/25-CS problems, so the complexity-scaling format, which hides conditions inside sub-problems, is far from handled.

• Improving reasoning sufficiency. Formulation sufficiency trails necessity for every model measured, and even GPT-5 reaches only 82.7%, meaning correct understanding frequently does not convert into a correct answer. What happens between formulation and solution is unresolved.

• Extending the approach beyond math. The authors state that the same abstract-to-contextual transformation can be applied to other domains and datasets, enabling systematic evaluation of contextual reasoning outside mathematics.

Target Audience

Researchers and engineers working on LLM reasoning and evaluation, benchmark designers, and teams building post-training pipelines for mathematical or analytical tasks. It is also useful for practitioners in finance, science, and engineering who deploy LLMs on scenario-based problems and want evidence about where those models actually break down. Readers need some familiarity with benchmark evaluation and fine-tuning to get full value, but the central argument — that models understand abstract problems far better than narrated ones — is accessible without a deep technical background.

Authors’ abstract

Large language models now solve many benchmark math problems at near-expert levels, yet this progress has not fully translated into reliable performance in real-world applications. We study this gap through contextual mathematical reasoning, where the mathematical core must be formulated from descriptive scenarios. We introduce ContextMATH, a benchmark that repurposes AIME and MATH-500 problems into two contextual settings: Scenario Grounding (SG), which embeds abstract problems into realistic narratives without increasing reasoning complexity, and Complexity Scaling (CS), which transforms explicit conditions into sub-problems to capture how constraints often appear in practice. Evaluating 61 proprietary and open-source models, we observe sharp drops: on average, open-source models decline by 13 and 34 points on SG and CS, while proprietary models drop by 13 and 20. Error analysis shows that errors are dominated by incorrect problem formulation, with formulation accuracy declining as original problem difficulty increases. Correct formulation emerges as a prerequisite for success, and its sufficiency improves with model scale, indicating that larger models advance in both understanding and reasoning. Nevertheless, formulation and reasoning remain two complementary bottlenecks that limit contextual mathematical problem solving. Finally, we find that fine-tuning with scenario data improves performance, whereas formulation-only training is ineffective. However, performance gaps are only partially alleviated, highlighting contextual mathematical reasoning as a central unsolved challenge for LLMs.