Constructing Industrial-Scale Optimization Modeling Benchmark

AI summary

Overview

Research area: Machine learning for optimization — specifically natural-language-to-optimization (NL-to-Opt) modeling, where LLMs translate problem descriptions into mathematical optimization models and solver-executable code.

Technical level: Intermediate. The paper assumes familiarity with mixed-integer linear programming (MILP), solver-level formulations (MPS format), and standard LLM benchmarking practices.

Scope: The paper introduces MIPLIB-NL, a 223-instance benchmark of industrial-scale optimization problems reverse-generated from real MIPLIB 2017 models, and shows that current NL-to-Opt systems collapse on it relative to toy-scale benchmarks.

What This Paper Is About

LLMs are increasingly used to convert written problem descriptions into formal optimization models and working solver code. But the benchmarks used to evaluate them are almost entirely toy-sized or synthetic — typically 4 to 120 variables — while real industrial models routinely have 10³ to 10⁶ variables and constraints organized into indexed variable groups and repeated constraint families. The paper's goal is to build a benchmark grounded in genuine industrial MILPs so that NL-to-Opt capability can be measured at a scale and structural complexity that matches practice.

Key Contributions

1. Diagnosis of a data-centric bottleneck. The authors argue that the gap between benchmark performance and real modeling competence stems primarily from benchmark design, not model architecture. Existing datasets flatten the indexed, compositional structure that defines practical models.

2. A structure-aware reverse construction methodology. Rather than collecting natural-language problems and deriving models from them, the authors start from solver-level MPS files and reverse-generate natural-language specifications, recovering the latent loop-based structure through expert analysis.

3. The MIPLIB-NL benchmark. 223 one-to-one reconstructions from MIPLIB 2017, each a triply aligned specification consisting of a natural-language description with separated data files, a mathematical model, and solver-executable code. Variables, constraints, objectives, and numerical parameters are preserved exactly up to systematic renaming.

4. A comprehensive evaluation revealing hidden failure modes. Fourteen NL-to-Opt systems — fine-tuned and prompt-based — are evaluated across ten existing benchmarks plus MIPLIB-NL, exposing systematic degradation and a distinct error regime dominated by data/index coupling and incomplete model recovery.

Main Findings

• Severe performance collapse on MIPLIB-NL. Systems scoring 60–98% on existing benchmarks drop to roughly 0–39%. The best result on MIPLIB-NL is GPT-5.1 at 39.05% Pass@1; the average across all 14 evaluated systems is 17.85%. GPT-5.1 scores 93.82% on NLP4LP and 81.78% on NL4Opt by comparison.

• Fine-tuned models fare worst. OptMATH-Qwen2.5-7B achieves 0.00% on MIPLIB-NL despite 79.91% on NL4Opt. SIRL-Qwen2.5-32B reaches 98.90% on MAMO-E but only 1.43% on MIPLIB-NL. Fine-tuning on toy-scale data does not transfer.

• Scale matters, but not monotonically. Accuracy drops sharply from small (<500) to medium (500–999) instances, partially recovers for large (1000–9999), then declines again for very large (≥10000). This reflects interactions between scale, structure, and domain composition rather than a simple size effect.

• Failure modes shift qualitatively, not just quantitatively. Across failed instances, modeling-level errors account for 48.3%, execution-level errors 45.0%, and solver timeouts 6.7%. Prior benchmarks are dominated by variable-domain/type and constraint-logic mistakes; MIPLIB-NL failures concentrate on data/index coupling errors and incomplete-model recovery, with statistical significance in pairwise distributional tests.

• Natural language is far more compact than the flattened formulation. Instance scale inflates dramatically when loop-based structure is expanded into MPS rows, but the model–data separation format keeps the natural-language specification and auxiliary data markedly smaller across all scale buckets, increasingly so for larger instances.

• Validation numbers. Of 245 candidates entering full validation, 163 passed independent LLM reconstruction (Pass@8 existential criterion); of the remaining 82, 60 were retained after expert–LLM review and 22 were discarded. Eight infeasible and 15 open instances were retained but excluded from quantitative evaluation due to the absence of certified objective values.

Methodology in Plain English

The core idea is to build the benchmark backwards. MIPLIB 2017 already contains hundreds of real, solver-validated industrial MILPs, but they exist only as flat algebraic files (.mps) with no natural-language descriptions. The authors treat the reverse direction — Opt-to-NL — as the controllable one.

Stage 1 — Structural abstraction. Operations research experts inspect each MPS file, combining on-model signals (variable naming patterns, coefficient regularities, sparsity and incidence structure) with off-model evidence (MIPLIB metadata, source papers, domain hints). They recover a loop-based scaffold: indexed variable groups, constraint families that are repeated instantiations of the same rule over index sets, and a small number of atomic non-repeating constraints. A drafted scaffold can be expanded back into algebraic rows and checked for exact equivalence against the original, so the abstraction is a compact representation of solver algebra rather than a subjective annotation.

Stage 2 — Reverse generation. With the structure fixed, experts design deterministic blueprints mapping each variable group to a semantic role, the objective to an optimization intent, each constraint family to a loop-based natural-language clause, and data dependencies to explicit file references. The raw description produced this way is structurally complete. An LLM may optionally polish surface phrasing, but is forbidden from introducing variables, constraints, coefficients, or changing data references.

Stage 3 — Semantic validation. Each instance is tested by independent reconstruction: multiple strong LLMs receive only the instance specification and data files and must produce both a formulation and executable code. An instance passes if at least one of eight attempts matches the reference solver outcome. Instances failing all LLMs enter a secondary phase of expert review with interactive human–LLM dialogue, where experts refine prompts and correct any inconsistencies traced back to the reverse generation process.

The result is a benchmark where the natural-language description, the mathematical model, and the code are guaranteed to be mutually consistent, and where the underlying problems retain the scale and structure of their industrial sources.

Why This Matters

Impact on research. The paper reframes NL-to-Opt evaluation as a data-design problem rather than purely a modeling-capability problem. It provides evidence that reported progress on existing benchmarks substantially overestimates real capability, and supplies a benchmark with a unified schema and structural annotations that enables error categorization at a granularity toy datasets cannot support. It also documents a recurring taxonomy of industrial modeling constructs — nested loops, subset-indexed loops, temporal and recursive coupling, sliding-window aggregation, complete-graph pairwise instantiation, extra-dimension replication — that is largely absent from prior benchmarks.

Real-world applications:

• Logistics and supply chain — network flow, routing, and assignment models where constraint families are indexed over nodes, vehicles, and time periods.
• Manufacturing and production planning — capacity, sequencing, and bill-of-materials constraints instantiated across products, lines, and planning horizons.
• Energy systems — unit commitment and dispatch models with temporal coupling and recursive state constraints.
• Finance — portfolio and risk models with indexed asset and scenario dimensions.

Industry relevance. The work is a collaboration between Peking University, Great Bay University, and Huawei Technologies, reflecting direct industrial interest in whether LLM-based modeling assistants can be trusted on production-scale problems. The finding that fine-tuned small models score near zero on industrial instances has immediate implications for anyone considering deploying such systems in operations research workflows.

Future Directions

• Closing the supervision gap. The paper's family-held-out LoRA experiment on Qwen2.5-7B-Instruct shows that difficult in-domain supervision substantially improves a small model, particularly on executability, but leaves a large gap to frontier models. What training regimes would close this gap remains open.

• Diagnosing the non-monotonic scale trend. The partial accuracy recovery from medium to large instances is unexplained. Isolating the contributions of scale, structure, and domain would clarify what actually drives failure.

• Extending coverage beyond the selected subset. MIPLIB 2017 contains over 1,300 models; MIPLIB-NL covers 223 due to the manual effort required for structure recovery and expert validation. Reducing the human cost of the pipeline is a natural next step.

• Handling infeasible and open instances. Eight infeasible and fifteen open instances were retained but excluded from quantitative evaluation because standard metrics depend on objective-value comparison. Metrics that assess structural fidelity without a certified optimum would bring these into the evaluation.

Target Audience

Researchers and practitioners working on LLM-based code and model generation, particularly those in the optimization, operations research, and automated mathematical modeling communities. It is also directly relevant to industrial teams evaluating whether LLM modeling assistants are ready for production deployment, and to benchmark designers interested in constructing datasets from authoritative real-world sources rather than synthesizing them forward from templates.

Authors’ abstract

Optimization modeling underpins decision-making in logistics, manufacturing, energy, and finance, yet translating natural-language requirements into correct optimization formulations and solver-executable code remains labor-intensive. Although large language models (LLMs) have been explored for this task, evaluation is still dominated by toy-sized or synthetic benchmarks, masking the difficulty of industrial problems with $10^{3}$--$10^{6}$ (or more) variables and constraints. A key bottleneck is the lack of benchmarks that align natural-language specifications with reference formulations/solver code grounded in real optimization models. To fill in this gap, we introduce MIPLIB-NL, built via a structure-aware reverse construction methodology from real mixed-integer linear programs in MIPLIB~2017. Our pipeline (i) recovers compact, reusable model structure from flat solver formulations, (ii) reverse-generates natural-language specifications explicitly tied to this recovered structure under a unified model--data separation format, and (iii) performs iterative semantic validation through expert review and human--LLM interaction with independent reconstruction checks. This yields 223 one-to-one reconstructions that preserve the mathematical content of the original instances while enabling realistic natural-language-to-optimization evaluation. Experiments show substantial performance degradation on MIPLIB-NL for systems that perform strongly on existing benchmarks, exposing failure modes invisible at toy scale.