CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning

AI summary

Overview

Research area: AI evaluation benchmarks, multimodal mathematical reasoning, discrete mathematics.

Technical level: Intermediate. The paper is readable without deep mathematical background, though familiarity with LLM benchmarking (accuracy tracks, chain-of-thought prompting, multiple-choice evaluation) helps.

Scope: Introduces CombiGraph-Vis, a 1,135-problem curated benchmark drawn from Iranian National Olympiad in Informatics problems, designed to probe whether AI models can genuinely reason about discrete mathematical objects — particularly when those objects are presented visually.

What This Paper Is About

Existing math benchmarks are increasingly saturated, meaning top models score so highly that the benchmarks can no longer distinguish between them. Meanwhile, multimodal datasets tend to underrepresent discrete mathematics — combinatorics, graph theory, and logic — even though these areas produce visually rich problems with short, checkable answers. This paper builds a new benchmark that fills that gap: 1,135 discrete-math olympiad problems, about a third of which require interpreting an image to solve, each with a verified solution and fine-grained technique labels.

Key Contributions

1. A new benchmark dataset (CombiGraph-Vis) of 1,135 problems spanning 13 discrete-mathematics domains and three answer formats (short-answer, multiple-choice, yes/no), with 35.8% of problems containing images whose structure is necessary for solving.

2. An agentic curation pipeline with human oversight. The authors built a multi-stage workflow using specialized "critic" agents to detect and repair errors introduced by PDF conversion, translation, and annotation, plus errors present in the original source material.

3. A hierarchical technique-labeling taxonomy. Every problem is tagged along a three-level structure (Topic → Sub-topic → Sub-sub-topic), covering 13 major topics and 89 distinct technique labels derived from the official Olympiad curriculum.

4. A distractor-susceptibility evaluation setup. The authors separate multiple-choice problems into "standalone" and "choice-dependent" subsets and measure whether models land on any of the offered choices versus the correct one, exposing how often models are lured by trap answers.

Main Findings

• Wide performance range, no saturation. Across model families, single-sample accuracy spans roughly 16% to 78%. Top models (GPT-5 at 78.0%, Gemini 2.5 Pro at 75.8%) lead mid-tier and lightweight/open-weight models by 20–40 percentage points. Even the strongest models leave substantial headroom.

• A persistent modality gap. Models consistently score worse on image-tagged problems than on text-only ones. For top-tier models the drop is typically 14–16 points (e.g., GPT-5 falls from 83.5% to 68.2%); for mid-tier models it can approach 20 points. Reading and reasoning over graphs, grids, and geometric diagrams remains a central bottleneck.

• Susceptibility to deliberately crafted distractors. For standalone multiple-choice problems, models frequently produce an answer that matches some provided choice but not the correct one. The gap (Δ) between "answer lies among the choices" accuracy and "answer is correct" accuracy ranges from about 14 points for the best models to over 45 points for weaker ones — evidence that models often recognize plausible-looking options rather than deriving the answer.

• Uneven mastery across subfields. Top models are comparatively strong in combinatorics, number reasoning, and invariants/monovariants. Graph-theoretic subdomains (connectivity, matchings) and formal languages show much larger spread, with weaker models struggling markedly.

• A curated, validated dataset. The agentic pipeline flagged and resolved conversion artifacts, translation slips, and genuine source errors, with a repair loop requiring five consecutive successful validations (up to 20 attempts) before accepting a fix, and escalating major cases or image-related issues to human reviewers.

Methodology in Plain English

Collecting the problems. The authors gathered problems from the Iranian National Olympiad in Informatics, first and second rounds, across multiple years. Source PDFs were the primary material, cross-checked against the competition's official website. Where several problems shared a definition or setup, that shared text was stored separately as a "context" field. Figures with low resolution or Persian text were redrawn.

Cleaning the data with AI agents. Because converting PDFs and translating from Persian introduces errors, the authors ran a two-phase validation workflow. In phase one, three specialized critic agents — one for typos and clarity, one for logical soundness, one for checking that the solution's final answer matches the stored answer — each reviewed every problem three times independently. A majority-vote aggregator combined the nine reports into a single structured diagnosis with a severity score from 1 to 5.

Repairing the problems. In phase two, flagged cases were classified as either pipeline/conversion artifacts, suspected original-source errors, or image-understanding issues. The first category got targeted edits; the second went through a "solution engagement" pass that expanded brief official solutions and re-evaluated the problem. Automated fixes were only accepted after a validator agent confirmed success five times in a row. Major or image-related problems were sent to human reviewers.

Evaluating models. Each model generated eight chain-of-thought solutions per problem, wrapping its final answer in a boxed format. Answers were extracted with a regex, checked numerically with the Math-Verify library where possible, and otherwise parsed by a lightweight LLM. The authors report four tracks: average score across eight samples (avg@8), whether any of eight was correct (pass@8), majority-vote correctness (maj@8), and whether all eight were correct (all-pass@8). The distractor analysis added two more metrics: standalone accuracy, and the fraction of problems where the model's answer matched one of the offered choices regardless of correctness.

Why This Matters

Impact on research. The paper supplies a non-saturated benchmark targeted at a specific capability — discrete mathematical reasoning over visual structures — that existing datasets either don't cover deeply or don't probe at all. Because every problem carries a verified solution and hierarchical technique labels, researchers can attribute failures to particular techniques rather than reporting a single aggregate score. The distractor analysis also offers a concrete diagnostic for a failure mode that standard accuracy metrics hide.

Real-world applications:

• Automated tutoring and homework systems that must interpret diagrams, graphs, and grids alongside text, and that need to know when a student (or a model) has picked a plausible-looking wrong answer.
• Evaluation of multimodal AI assistants deployed in education, engineering design, and technical support, where reasoning about structured visual information is routine.
• Reinforcement-learning environments for reasoning. The authors explicitly propose the benchmark as an RL environment, since distractor avoidance is a learnable skill that transfers to competition-style problem solving.
• Assessment design. The finding that trap choices systematically fool models has implications for how human exams are constructed and how fairness is evaluated.

Industry relevance. Model developers and evaluation teams gain a benchmark that separates model tiers even at the frontier, plus a modality-gap metric that quantifies how much visual reasoning lags behind text reasoning in their systems. Companies building multimodal products can use the technique labels to pinpoint which subfields need targeted training data.

Future Directions

• Closing the modality gap. The 14–20 point drop on image-tagged problems points to a specific, measurable deficit in visual mathematical understanding that current architectures have not solved. Training or prompting methods that narrow this gap are an open problem.

• Reducing distractor susceptibility. The large Δ values suggest models learn to recognize plausible answers rather than construct them. Whether this can be trained away — for instance via RL against the benchmark — is left as a hypothesis rather than a result.

• Scaling the ontology. The three-level taxonomy with 89 technique labels enables per-technique error analysis, but the paper only reports top-level topic breakdowns. Deeper analysis of which specific techniques defeat which models is a natural extension.

• Expanding beyond one source. All problems come from a single national olympiad. Adding other competitions and language sources would test whether the observed capability gaps generalize.

Target Audience

Researchers building or evaluating multimodal reasoning systems; benchmark designers interested in curation methodology and agentic data pipelines; AI education practitioners who need reliability assessments for tutoring tools; and mathematically inclined engineers who want a concrete, non-saturated test of whether a model can actually reason about combinatorial and graph structures rather than pattern-match against answer choices.

Authors’ abstract

State-of-the-art (SOTA) LLMs have progressed from struggling on proof-based Olympiad problems to solving most of the IMO 2025 problems, with leading systems reportedly handling 5 of 6 problems. Given this progress, we assess how well these models can grade proofs: detecting errors, judging their severity, and assigning fair scores beyond binary correctness. We study proof-analysis capabilities using a corpus of 90 Gemini 2.5 Pro-generated solutions that we grade on a 1-4 scale with detailed error annotations, and on MathArena solution sets for IMO/USAMO 2025 scored on a 0-7 scale. Our analysis shows that models can reliably flag incorrect (including subtly incorrect) solutions but exhibit calibration gaps in how partial credit is assigned. To address this, we introduce agentic workflows that extract and analyze reference solutions and automatically derive problem-specific rubrics for a multi-step grading process. We instantiate and compare different design choices for the grading workflows, and evaluate their trade-offs. Across our annotated corpus and MathArena, our proposed workflows achieve higher agreement with human grades and more consistent handling of partial credit across metrics. We release all code, data, and prompts/logs to facilitate future research.