I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Models

AI summary

Overview

• Research area: Benchmarking and evaluation of reasoning in large language models (LLMs) and large reasoning models (LRMs), specifically analogical reasoning (Raven's Progressive Matrices-style analogies) and arithmetic reasoning, plus robustness to simulated perceptual uncertainty.
• Technical level: Intermediate. The paper is readable without deep math, but assumes familiarity with prompting techniques (in-context learning, self-consistency, chain-of-thought) and benchmark concepts such as in-distribution vs. out-of-distribution generalization.
• Scope: The paper introduces I-RAVEN-X, a symbolic benchmark that stress-tests LLMs and LRMs along four axes — productivity, systematicity, confounding attributes, and non-degenerate attribute distributions — and reports that LRMs generalize better than LLMs on the first two axes while still failing badly on the last two.

What This Paper Is About

Existing Raven's Progressive Matrices benchmarks such as RAVEN and I-RAVEN have two problems for evaluating modern language models: their problems use very few operands and small attribute ranges, and their test items and answers are openly available online, creating a risk of data leakage from pre- and post-training. They also assume "oracle perception," meaning the reasoning model is handed clean symbolic attributes rather than having to cope with noisy or irrelevant perceptual input.

The goal of I-RAVEN-X is to remove those assumptions: it makes the reasoning chains longer, the attribute ranges wider, and it deliberately injects confounders and blurred attribute values so that the model must reason under uncertainty. The paper then measures how well LLMs and LRMs hold up as these stress factors increase.

Key Contributions

1. Introduces I-RAVEN-X, a fully symbolic, parametrizable benchmark that extends I-RAVEN along four dimensions: (i) productivity — more operands per reasoning relation, e.g., 3×10 matrices instead of 3×3; (ii) systematicity — larger dynamic ranges for operand values, e.g., 1000 attribute values instead of 10, described as up to 100× more values per attribute; (iii) robustness to confounding factors — randomly sampled irrelevant attributes added to each panel; and (iv) robustness to non-degenerate value distributions — smoothed attribute distributions that simulate an imperfect perception front-end.
2. Provides a systematic empirical comparison of state-of-the-art LLMs (GPT-4, Llama-3 70B) and LRMs (OpenAI o3-mini, DeepSeek R1, and a DeepSeek R1 distilled version based on Llama 70B) on both I-RAVEN and I-RAVEN-X.
3. Establishes the central empirical claim: LRMs generalize consistently better than LLMs in productivity and systematicity, but are still significantly challenged by reasoning under uncertainty and cannot effectively explore multiple probabilistic outcomes in superposition.
4. Releases the dataset and experiment code at https://github.com/IBM/raven-large-language-models, and notes that the paper summarizes contributions presented in two recent publications, Hersche et al. [28] and Camposampiero et al. [29].

Main Findings

• LRMs degrade far less than LLMs on arithmetic accuracy: the paper reports arithmetic degradation from 80.5% to 63.0% for LRMs versus 59.3% to 4.4% for LLMs. Elsewhere it states that LLMs show a massive drop in arithmetic accuracy on I-RAVEN-X, nearing 0% for comparable prompt complexity, while improvements reach up to 65.4% in certain settings for LRMs.
• Productivity and systematicity favor LRMs: on I-RAVEN (3×3) versus I-RAVEN-X (3×10) with larger attribute ranges, DeepSeek R1 scored 80.6 task / 74.8 arithmetic on I-RAVEN and 84.0 / 67.7 and 82.8 / 65.8 on I-RAVEN-X, while Llama-3 70B with in-context learning and 21 prompts scored 85.0 / 45.0 on I-RAVEN and 73.0 / 2.6 and 74.2 / 0.4 on I-RAVEN-X.
• LRMs need less prompt engineering: LRMs achieve reasoning accuracy on par with LLMs despite using significantly less engineered prompts. o3-mini shows no drops in accuracy on I-RAVEN-X and a 6% drop on I-RAVEN compared to GPT-4 while using only 1/21 of the prompts; when compared at similar prompt complexities (entangled prompting in both settings, retaining a 1:7 ratio due to self-consistency), o3-mini shows an average 6.5% increase in accuracy.
• Higher reasoning effort helps only so much: OpenAI o3-mini at "high" effort scored 92.6 task / 86.1 arithmetic on I-RAVEN, versus 86.6 / 74.4 at "medium," but the gap narrows on I-RAVEN-X, and token counts rise sharply (for example 19,041 average output tokens at high effort on the range-1000 setting versus 7,209 at medium).
• Confounders hurt LRMs measurably: with 10 additional confounding attributes, o3-mini's accuracy dropped by 11.2% (task) and 15.2% (arithmetic), while R1 was more robust to confounders with drops of 5.8% and 12.2%.
• Smoothed distributions hurt R1 much more than o3-mini: DeepSeek R1 lost up to 19.8% of task accuracy in the harshest smoothing scenario, while o3-mini showed a much smaller degradation of 5.4%.
• Combining both stressors is devastating: when confounders and distribution smoothening are applied at maximum level together, task accuracy falls for o3-mini to 17.0% and for DeepSeek R1 to 22.8%, bringing them close to random chance, which the paper identifies as 12.5%.
• Overall uncertainty penalty: the abstract and conclusion report a −61.8% change in task accuracy for LRMs under uncertainty, and state that LRMs cannot effectively explore multiple probabilistic outcomes in superposition.
• o1 versus o3-mini ablation: on a subset of 100 test examples, OpenAI o1 scored 88.0 task / 79.7 arithmetic on I-RAVEN and 86.0 / 68.2 on both I-RAVEN-X ranges tested, while o3-mini scored 86.6 / 81.4, 84.0 / 63.6, and 81.0 / 60.8 — roughly comparable, with o3-mini consistently slightly less accurate but approximately 14× less expensive ($1.1 and $4.4 per million input and output tokens versus $15 and $60 for o1).
• Confounders increase reasoning effort: output token counts rise with the number of confounders, for example o3-mini going from 7,209 tokens with no confounders to 13,709 tokens with 10 confounders.

Methodology in Plain English

The starting point is I-RAVEN, a symbolic version of Raven's Progressive Matrices. Each problem is a matrix of panels containing geometric objects; the bottom-right panel is blank and must be completed from eight candidate answers. Attributes (color, size, shape, number, position) are governed by four rules: constant (value does not change per row), arithmetic (third panel is the sum or difference of the first two), progression (monotonic increase or decrease by 1 or 2), and distribute three (three values stay constant across rows but shift positions row by row).

I-RAVEN-X generalizes this in a parametrized way. The number of columns g and the attribute value range m (values stay in [0, m−1]) become free parameters, so the researchers can generate 3×g matrices instead of 3×3, and ranges of 10, 100, or 1000 instead of 10. For the arithmetic rule, the generation procedurally samples the first g−1 panels with a shrinking sampling range and then computes the last panel; for progression, an increment δ is sampled from {−2, −1, +1, +2} and the matrix filled accordingly. Candidate answers are generated using I-RAVEN's attribute bisection tree, which avoids the original RAVEN flaw where the mode of the candidate answers revealed the correct one.

Uncertainty is injected in two ways. Confounders are extra attributes sampled uniformly from [0, m−1] that are irrelevant to the reasoning and therefore linearly reduce the signal-to-noise ratio (SNR), forcing the model to filter noise. Distribution smoothing replaces the originally degenerate (one-hot) attribute values with a three-bin strategy: the true value T is assigned probability p(T) ~ U(p_L, 1) with p_L > 0.5, and the two neighbouring values share the remainder. Separate experiments isolate confounders only, smoothing only, and both together.

For the experiments, LLMs (GPT-4, Llama-3 70B) were run with disentangled prompting (which multiplies queries by 3, going from 7 to 21 prompts), self-consistency (n = 7 samples with majority vote; temperatures of 0.5 for GPT-4 and 0.4 for Llama-3 70B found via grid search on 50 I-RAVEN problems), and optional in-context learning with 16 examples. LRMs (o3-mini with medium or high reasoning effort, DeepSeek R1 with 671B parameters, and its distilled version) were run with entangled prompts, no in-context examples, no self-consistency, and no attribute scaling, with 25,000 maximum reasoning tokens and, for R1, temperature 0.6 and top-p 0.7. Because of budget constraints, the evaluation uses a subset of 500 randomly sampled RPM tests for I-RAVEN and I-RAVEN-X.

Why This Matters

Impact on research. The paper argues that RAVEN-style benchmarks are close to saturated and potentially contaminated by data leakage, and that the standard practice of assuming oracle perception hides a large part of the difficulty in end-to-end analogical reasoning — namely filtering irrelevant attributes and handling perception uncertainty. I-RAVEN-X offers a controllable way to dial up difficulty along separate axes, which makes it possible to distinguish "the model cannot reason" from "the model cannot reason with noisy inputs." The paper's headline result — that LRMs largely survive longer reasoning chains and wider attribute ranges but collapse toward chance when uncertainty is added — is a specific, measurable gap rather than a general statement about model quality.

Real-world applications.

• Evaluating AI assistants that must extract structured data from noisy documents, forms, or scanned inputs, where irrelevant fields act as confounders.
• Assessing reliability of models used for spreadsheet-like or accounting-style arithmetic across long chains of operations, where errors compound.
• Screening models that consume outputs from imperfect perception front-ends (for example, attribute extractors or vision modules), since I-RAVEN-X emulates exactly that pipeline stage.
• Informing model selection and cost trade-offs: the appendix's o1 versus o3-mini comparison shows comparable accuracy at roughly 14× lower price, which is directly relevant to deployment budgets.

Industry relevance. Because the benchmark is parametrizable and symbolic, it can be regenerated at will, which reduces leakage risk and makes it usable as a recurring internal evaluation. The cost accounting (prompt counts, token counts, per-million-token prices) and the finding that LRMs need less prompt engineering than LLMs are directly relevant to teams deciding between prompt-engineering-heavy pipelines and reasoning models.

Future Directions

• Investigating the causal relationship between reasoning under uncertainty, prompt efficiency, and reasoning accuracy, which the paper explicitly names as a limitation and leaves to future work.
• Extending I-RAVEN-X from the purely symbolic domain to the visual domain, since the authors state the dataset "has not been extended yet to the visual domain" despite being illustrated visually for clarity.
• Understanding why performance collapses when confounders and distribution smoothing are combined at maximum level — the paper offers two candidate explanations (increased prompt complexity impairing pattern detection, or a more general limitation in maintaining coherence across multiple uncertain variables) without deciding between them.
• Exploring whether LRMs can be made to "explore multiple probabilistic outcomes in superposition," which the paper identifies as a capability they currently lack.
• Broadening the evaluation: the study is restricted to a handful of models and to a subset of 500 randomly sampled RPM tests due to budget constraints, and o3-mini was used rather than o1 for the full evaluation on cost grounds.

Target Audience

Researchers and practitioners working on reasoning evaluation for language models, especially those interested in out-of-distribution generalization, robustness to noisy inputs, and the distinction between LLMs and reasoning models. It is also useful for engineers who need a benchmark that can be regenerated to avoid data leakage, and for teams making cost-accuracy trade-offs between prompt-engineered LLM pipelines and LRMs. Readers unfamiliar with Raven's Progressive Matrices or prompting techniques will need some background, but the appendix and worked prompt examples make the setup concrete.

Authors’ abstract

We introduce I-RAVEN-X, a symbolic benchmark designed to evaluate generalization and robustness in analogical and mathematical reasoning for Large Language Models (LLMs) and Large Reasoning Models (LRMs). I-RAVEN-X extends I-RAVEN by increasing operand complexity, attribute range, and introducing perceptual uncertainty. Compared to LLMs, empirical results show that LRMs achieve improved productivity and systematicity on longer reasoning relations and wider attribute ranges, respectively. However, LRMs are still significantly challenged by reasoning under uncertainty and cannot effectively explore multiple probabilistic outcomes.