AutoOpt: A Dataset and a Unified Framework for Automating Optimization Problem Solving

AI summary

Overview

Research area: Artificial intelligence applied to mathematical optimization — specifically Mathematical Expression Recognition (MER), deep learning for code generation, and automated optimization problem solving.

Technical level: Intermediate. The paper is readable for someone with basic familiarity with machine learning and optimization, but the three-module pipeline (computer vision, language models, bilevel optimization) spans several technical areas.

Scope: The paper introduces a dataset of 11,554 images of optimization formulations and an end-to-end framework that reads an image of a mathematical program and solves it.

What This Paper Is About

Optimization problems are frequently written down as mathematical formulations on paper, whiteboards, tablets, or in scanned book pages and research articles, but images lack the semantic structure machines need to process them. Existing optical character recognition and mathematical expression recognition tools handle one-dimensional or single-line expressions and do not cope with the multilined, interlinked structure of optimization models (variables, parameters, objectives, constraints). The paper's goal is to close this gap by releasing a domain-specific dataset of optimization formulations and building a pipeline that converts an image of a formulation into LaTeX, then into a PYOMO modeling script, and finally into a solved optimization problem with limited human intervention.

Key Contributions

1. AutoOpt-11k dataset: an image dataset of 11,554 mathematical programs that are a mix of handwritten (5,070) and typeset (6,484) images, with LaTeX labels for all 11,554 images and PYOMO labels for a subset of 1,018 unique mathematical programs. It is created by 25 experts and verified in two phases.

2. A deep learning architecture for MER (module M1): a hybrid vision encoder combining ResNet-101 and Swin Transformer with an mBART-based decoder, extending the NOUGAT architecture. It outperforms GPT 4o, Gemini 2.0 Flash, and Nougat on the BLEU score metric on this dataset.

3. A fine-tuned LaTeX-to-PYOMO model (module M2): a pre-trained DeepSeek-Coder 1.3B model fine-tuned on instruction-style data to generate PYOMO scripts from LaTeX code.

4. An automated optimization solver (module M3): a Bilevel Optimization based Decomposition (BOBD) method implemented in Python that builds on prior work by Sinha et al. and automates the decomposition task using machine learning, using a genetic algorithm at the upper level and a convex optimization solver at the lower level.

Main Findings

• Dataset scale and diversity: AutoOpt-11k contains 11,554 images (7,637 unique mathematical programs). Composition includes 10,838 single-objective, 159 multi-objective, 399 multi-level, and 158 uncertainty problems; 155 unconstrained and 11,399 constrained; 7,349 general-form and 4,205 fully defined; 608 vector-form, 10,246 scalar-form, and 804 scalable-form; 2,130 linear, 9,122 non-linear, 10,806 continuous, 424 discontinuous, 2,580 convex, 3,574 non-convex, 9,502 differentiable, and 502 non-differentiable problems.

• M1 outperforms large models on BLEU, and beats Nougat on most metrics: On Character Error Rate, AutoOpt-M1 (393.3M parameters) scores 0.0412 on handwritten, 0.0176 on printed, and 0.0286 on combined handwritten plus printed. Nougat (348.7M) scores 0.0752, 0.0168, and 0.0440 respectively. GPT 4o scores 0.1465, 0.0664, and 0.1017; Gemini 2.0 Flash scores 0.1607, 0.1047, and 0.1338. AutoOpt-M1 outperforms Nougat on all metrics except Character Error Rate for printed images.

• The hybrid encoder was chosen via ablation: DL1 (CNN, no Transformer) achieved BLEU 16.10 and CER 0.8812; DL2 (no CNN, with Transformer) achieved BLEU 95.51 and CER 0.0440; DL3 (CNN plus Transformer) achieved BLEU 96.70 and CER 0.0286, so DL3 was selected.

• M2 achieves strong LaTeX-to-PYOMO translation: The fine-tuned AutoOpt-M2 obtained a BLEU score of 88.25 and a Character Error Rate of 0.0825, trained on 80% of 1,018 mathematical models and tested on the remaining 20%.

• BOBD handles a wider variety of problems than common approaches: The paper reports that BOBD yields better results on complex test problems compared to common approaches such as the interior-point algorithm and genetic algorithm. The authors state they do not claim superiority over specialized optimization techniques on optimality guarantees and convergence rates.

• Annotator agreement: Each of 5 annotators (A1–A5) annotated 30% of images in the second phase, with an average 16.6% overlap between any pair. Pairwise BLEU scores ranged from 0.8185 to 0.8588, and CER from 0.1267 to 0.1791. Discrepancies were often due to syntactic differences in code for the same image.

• Framework reliability estimates: Module-level reliability was estimated as 97.14% for M1 (1 minus CER of 0.0286) and 91.75% for M2 (1 minus CER of 0.0825), giving an overall estimate of 89.12%, described as a lower bound. Framework-level evaluation on 500 sample problems outside AutoOpt-11k gave an overall success rate of 94.20%.

• Dataset statistics: Image width ranges from 159 to 3,611 px (mean 783.91, median 753.50); height from 24 to 2,670 px (mean 338.89, median 295.00); aspect ratio from 0.25 to 18.29 (mean 2.73, median 2.40); file size from 3.06 to 1,399.98 KB (mean 95.58, median 41.68); LaTeX length from 14 to 1,620 characters (mean 212.23, median 180.00); PYOMO length from 192 to 1,087 characters (mean 390.30, median 362.00).

Methodology in Plain English

The researchers first assembled a dataset. Twenty annotators sourced optimization problems from books, research papers, modeling software documentation (AMPL, GAMS, PYOMO, JuMP, LINDO), solver documentation (CPLEX, Gurobi, CBC, CLP, Ipopt), and online repositories (COIN-OR, MIPLIB, OR-Library, UCI, NEOS, Netlib), and produced handwritten versions of many of them. Five annotators with programming backgrounds then wrote LaTeX for every image and PYOMO scripts for a subset. As a check, they regenerated each image from its LaTeX code and compared it visually with the original to catch discrepancies.

The framework then works in three stages. In M1, an image is resized so its longer side fits a 768 × 1024 canvas, center-padded on a white background, contrast-enhanced, and sharpened with an unsharp mask. A hybrid encoder runs ResNet-101 (initialized with ImageNet weights) for local features alongside a Swin Transformer for long-range spatial relationships; the ResNet feature vector is projected, scaled by a learnable gating parameter initialized to zero, and prepended to the Transformer's patch embeddings. An mBART decoder initialized with NOUGAT weights generates LaTeX token by token. Training used an 80/10/10 split.

In M2, that LaTeX is fed to a fine-tuned DeepSeek-Coder 1.3B model that outputs a PYOMO script. In M3, the PYOMO script is solved using a bilevel decomposition: the problem is split so that a genetic algorithm handles the upper-level variables while a convex solver handles the lower-level ones for each sample, with machine learning used to classify variables into the two levels. The authors chose a two-stage design (image to LaTeX to PYOMO) rather than a single direct model because the intermediate LaTeX acts as a human-readable checkpoint for verification.

Why This Matters

Impact on research: The paper supplies a domain-specific resource that the authors state did not previously exist — an image dataset of mathematical programs for optimization, rather than single-line expressions from general mathematics. It also connects computer vision, natural language processing, and mathematical optimization in a single pipeline, and releases both the dataset and the framework code publicly.

Real-world applications:

• Classroom and academic settings where models are worked out on whiteboards or tablets and then need to be solved.
• Research workflows where formulations appear only as figures in articles or scanned pages in books.
• Industrial and business settings where engineering or operational problems are sketched by hand before being implemented.
• Rapid prototyping from notes taken on paper or electronic boards, including mixed handwritten and typeset content.

Industry relevance: The pipeline removes a mechanical, error-prone translation step between a written model and a solver-ready script, which matters wherever optimization is used but modeling expertise is a bottleneck. The module M2 and M3 components are also independent: the authors note any other method could be used to solve the PYOMO script.

Future Directions

• Handling ill-defined optimization problems effectively, which the authors explicitly name as a limitation.
• Handling optimization problem definitions that span multiple pages or multiple images.
• Building on the BOBD method's machine-learning-driven decomposition, since the paper notes optimization methods are evaluated on optimality guarantees and convergence rates, where it makes no claims of superiority.
• Extending the framework and dataset to encourage further work at the intersection of computer vision, natural language processing, and mathematical optimization, as the authors anticipate.

Target Audience

Researchers and practitioners in optimization, machine learning, and computer vision who are interested in automating the path from a written mathematical formulation to a solved problem. It is also relevant to educators and industry teams that frequently transcribe formulations by hand, and to anyone building datasets or models for mathematical expression recognition beyond single-line expressions.

Authors’ abstract

This study presents AutoOpt-11k, a unique image dataset of over 11,000 handwritten and printed mathematical optimization models corresponding to single-objective, multi-objective, multi-level, and stochastic optimization problems exhibiting various types of complexities such as non-linearity, non-convexity, non-differentiability, discontinuity, and high-dimensionality. The labels consist of the LaTeX representation for all the images and modeling language representation for a subset of images. The dataset is created by 25 experts following ethical data creation guidelines and verified in two-phases to avoid errors. Further, we develop AutoOpt framework, a machine learning based automated approach for solving optimization problems, where the user just needs to provide an image of the formulation and AutoOpt solves it efficiently without any further human intervention. AutoOpt framework consists of three Modules: (i) M1 (Image_to_Text)- a deep learning model performs the Mathematical Expression Recognition (MER) task to generate the LaTeX code corresponding to the optimization formulation in image; (ii) M2 (Text_to_Text)- a small-scale fine-tuned LLM generates the PYOMO script (optimization modeling language) from LaTeX code; (iii) M3 (Optimization)- a Bilevel Optimization based Decomposition (BOBD) method solves the optimization formulation described in the PYOMO script. We use AutoOpt-11k dataset for training and testing of deep learning models employed in AutoOpt. The deep learning model for MER task (M1) outperforms ChatGPT, Gemini and Nougat on BLEU score metric. BOBD method (M3), which is a hybrid approach, yields better results on complex test problems compared to common approaches, like interior-point algorithm and genetic algorithm.