FractalBench: Diagnosing Visual-Mathematical Reasoning Through Recursive Program Synthesis

AI summary

Overview

Research area: Multimodal AI evaluation — specifically visual-mathematical reasoning and program synthesis from images, using fractal geometry as a diagnostic testbed.

Technical level: Intermediate. The paper is readable without specialist mathematics, but assumes familiarity with multimodal large language models (MLLMs), code generation, and basic notions of recursion and self-similarity.

Scope: The paper introduces and applies a 610-image benchmark of 12 canonical Iterated Function System fractals to test whether four leading MLLMs can infer recursive Python programs from images alone.

Authors: Jan Ondras (MIT) and Marek Šuppa (Comenius University in Bratislava, Cisco), with equal contribution and alphabetical author order. Published 2025-11-09 as arXiv:2511.06522v1 [cs.AI], in the MATH-AI workshop track.

What This Paper Is About

Mathematical reasoning often means inferring a general rule from a few concrete examples — seeing the infinite process inside finite observations. The authors ask whether multimodal AI systems can do this when the evidence is purely visual: given a picture of a fractal, can a model write code that regenerates it?

Fractals are used because each one is produced by a tiny set of recursive rules (an Iterated Function System with typically 2–8 contraction mappings) yet yields arbitrarily complex self-similar patterns. Correctly reproducing one therefore requires connecting visual perception to mathematical abstraction, not just pattern matching.

Key Contributions

1. A diagnostic benchmark (FractalBench). 12 canonical fractals — Cantor sets, Koch curves, Sierpiński structures, dragon curves, and tree fractals — are chosen to stress different reasoning capabilities, with a mapping from each fractal family to a specific mathematical skill.
2. Empirical evidence of a capability gap. Across four MLLMs, 76.1% of generated programs run successfully but only 4.2% reproduce the target fractal, showing syntactic competence without semantic understanding.
3. Contamination-resistant evaluation design. Fractals are rendered at variable recursion depths and in five colors (black, red, blue, green, purple), preventing models from relying on cached visual embeddings of canonical black fractal images. The authors describe this as parameterizable complexity.
4. A minimal-interface methodology. A four-command MinimalTurtle interface (move, turn, pen_up/pen_down, plus goto) is used deliberately so that models cannot bypass mathematical reasoning through library recall, L-system templates, or matplotlib shortcuts.

Main Findings

• Large execution-versus-correctness gap. 76.1% of generated code executes successfully, but only 4.2% produces a visually correct fractal, and only 5.5% of runnable samples were correct. Models write valid Python that draws some recursive pattern, not the correct generative rule.

• No model dominates overall. In the main results table (1,464 evaluations, 122 per condition), overall end-to-end success ranged from 0.0% (Qwen 2.5-VL with the Recursive Structure Focus prompt) to 11.5% (Gemini 2.5 Flash with Direct Code Generation).

• Success tracks mathematical challenge. Koch fractals reached the highest accuracy at 17–21%, attributed to reliance on iterative geometric transformations. Sierpiński fractals reached 3–18%. Tree fractals failed catastrophically at under 2%.

• Branching recursion is the specific bottleneck. Trees have the simplest IFS definitions (only 2 maps), yet fail worst. The authors report that models substitute iterative loops, single-branch recursion, or fixed-depth static approximations instead of true branching — suggesting branching, not recursion itself, is the limiting factor. The contrast with the better-performing Cantor set supports this.

• Geometric operations are learnable, recursive abstraction is not. Even the best-performing Koch results leave roughly an 80% failure rate, which the authors read as evidence that geometric intuition alone is insufficient without genuine recursive abstraction.

• Reasoning-first prompting hurts. Direct Code Generation outperformed both Reasoning Then Code and Recursive Structure Focus across Claude, Gemini, and GPT-4o (DCG 7.4–11.5% versus RTC 1.6–3.3% and RSF 0.8–3.3%), inverting the usual chain-of-thought advantage. The authors hypothesize that verbose intermediate reasoning anchors models on high-level verbal descriptions that are hard to translate into exact angles, ratios, and coordinates; that longer outputs dilute attention before code generation; and that direct image-to-code pathways preserve visual-geometric coupling better.

• Gemini's profile is unusual. Gemini 2.5 Flash had low execution rates (23.8%–31.1%) but, among runnable samples, notably high accuracy with the Direct Code Generation prompt (48.3%, 14 of 29 correct).

• Code complexity shows phase transitions. Analysis (Appendix J) found Gemini produces more verbose code, recursion-focused prompts reduce complexity, and some fractals show complexity initially growing with recursion depth before dropping sharply — a threshold where models shift from literal pixel-level description to compressed algorithmic representation. The authors connect code length to Kolmogorov complexity as a proxy for structural compressibility.

Methodology in Plain English

The researchers built a test set of 610 images, each rendered at 1,024 × 1,024 pixels, DPI 128, 0.5-pixel line width, on a white background, PNG format, spanning 4–12 recursive levels and five line colors. Recursion depth per fractal was derived by requiring the smallest substructure to remain at least 1 pixel, starting from a base block of 500 pixels, using the closed-form formula d_max = floor(ln(s_min/s_0) / ln(r)). For the Symmetric Binary Tree the ratio was set explicitly at 0.65 with a 60-degree angle and maximum depth 7. Depths ranged from 4 (Cantor Dust, Koch Curve, Sierpiński Carpet) to 12 (Lévy Dragon). Each color variant contains 122 images; all images were run across 12 model–prompt combinations (4 models × 3 prompting strategies), yielding 1,464 runs per color and 7,320 runs in total.

Four models were evaluated: GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Flash, and Qwen 2.5 VL 72B. Each was tested under three prompting strategies: Direct Code Generation (raw image-to-code), Reasoning Then Code (structured written analysis before code), and Recursive Structure Focus (explicit base cases, self-similarity, and parameter scaling). All prompts included the MinimalTurtle interface guide, examples, and formatting requirements.

Generated code was run in a sandbox with a 30-second timeout, catching syntax errors, runtime exceptions, and non-terminating code. Successfully executed code produced a 1,024 × 1,024 image, which was compared against ground truth using Intersection over Union (Jaccard Index) of binary masks, with a 95% similarity threshold defining correctness — a criterion the authors note follows comparable vision-to-code benchmarks such as TurtleBench.

The paper specifies a five-level reasoning hierarchy that synthesis requires: (1) scale invariance recognition, (2) geometric transformation inference, (3) recursive structure abstraction, (4) compositional reasoning, and (5) branching recursion.

Why This Matters

Impact on research. Prior benchmarks such as MathVista, MATH-Vision, MATHGLANCE, GeoGramBench, and TurtleBench largely test applying mathematical knowledge to visual problems. TurtleBench, for example, reaches only 19% accuracy on simple geometric shapes. FractalBench instead targets abstraction — inferring the recursive rule that generates what is seen. The authors note that the same difficulty patterns, such as failures on branching recursion, appear in GeoGramBench, MathVista, and MATHGLANCE, arguing that the limitations exposed are conceptual rather than API-specific.

Real-world applications:

• Education technology — systems that must generate worked examples and explanations rather than just answers.
• Formal verification and software engineering — synthesizing programs from specifications where the spec is a diagram or pattern rather than text.
• Scientific discovery pipelines — inferring compact generative rules from observational data, the same abstraction step fractals demand.
• Evaluation and red-teaming of multimodal models — a contamination-resistant diagnostic that resists memorized-image shortcuts.

Industry relevance. Any organization deploying multimodal models for diagram-to-code, CAD-style generation, or educational tooling gets a concrete warning: fluent, runnable output is not evidence of understanding. The counterintuitive prompting result also has direct engineering consequences — adding more reasoning scaffolding to spatial or numerical generation tasks may reduce output quality.

Future Directions

1. Extend beyond a single generation per image. The authors note the current methodology uses one generation per image and a binary IoU ≥ 95% criterion, which may not capture model stochasticity or how closely a model captures underlying generative structure.

2. Add structure-aware metrics. Complementary measures such as branch count accuracy or recursive depth detection could give finer-grained diagnosis of which geometric properties models capture versus miss.

3. Compare against specialized program synthesis and newer reasoning models. The evaluation omits traditional program synthesis baselines (e.g., Chaudhuri; Gulwani et al.) and reasoning-specialized systems such as OpenAI o1 or DeepSeek-R1. Testing these would show whether FractalBench exposes universal failure modes or distinguishes genuinely stronger capabilities.

4. Turn diagnostics into interventions. The prompting findings remain observational without stepwise ablations, and the authors have not shown how benchmark diagnostics could guide targeted model improvements. Controlled experiments isolating prompt complexity, reasoning depth, and instruction structure would enable causal interpretation.

Target Audience

Researchers and engineers working on multimodal LLM evaluation, visual program synthesis, and mathematical reasoning benchmarks; AI education and assessment teams who care about whether models genuinely abstract rules or merely match patterns; and model developers who need a contamination-resistant diagnostic that separates geometric competence from recursive abstraction. Readers seeking a general introduction to multimodal AI will find the framing accessible, but the paper's value is primarily for those already close to benchmarking or program synthesis.

Authors’ abstract

Mathematical reasoning requires abstracting symbolic rules from visual patterns -- inferring the infinite from the finite. We investigate whether multimodal AI systems possess this capability through FractalBench, a benchmark evaluating fractal program synthesis from images. Fractals provide ideal test cases: Iterated Function Systems with only a few contraction maps generate complex self-similar patterns through simple recursive rules, requiring models to bridge visual perception with mathematical abstraction. We evaluate four leading MLLMs -- GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Flash, and Qwen 2.5-VL -- on 12 canonical fractals. Models must generate executable Python code reproducing the fractal, enabling objective evaluation. Results reveal a striking disconnect: 76% generate syntactically valid code but only 4% capture mathematical structure. Success varies systematically -- models handle geometric transformations (Koch curves: 17-21%) but fail at branching recursion (trees: <2%), revealing fundamental gaps in mathematical abstraction. FractalBench provides a contamination-resistant diagnostic for visual-mathematical reasoning and is available at https://github.com/NaiveNeuron/FractalBench