Large-Scale Optimization Model Auto-Formulation: Harnessing LLM Flexibility via Structured Workflow

AI summary

Overview

Research area: Applying large language models (LLMs) to the automated formulation of optimization models — the interface between operations research (OR) and AI agent design.

Technical level: Intermediate. The paper assumes familiarity with linear programming / mixed integer linear programming concepts (objectives, constraints, variables) and with LLM prompting and retrieval techniques, but its core idea — build a step-by-step plan, then follow it — is explained in plain terms.

Scope: The paper proposes LEAN-LLM-OPT, a three-agent LLM workflow that turns a natural-language problem description plus separate datasets into a large-scale optimization formulation and solver code, and evaluates it on new large-scale benchmarks and a Singapore Airlines use case.

What This Paper Is About

Building optimization models for real business problems is slow and expert-intensive, and existing LLM approaches mostly work on small problems where all data fits inside the prompt. The authors address the harder case — "large-scale" problems whose data must be stored separately in input files — where they show that state-of-the-art models degrade sharply as inputs grow, sometimes dropping below 50% accuracy once inputs exceed 800 tokens. Their goal is a lightweight, agentic workflow that exploits an LLM's flexibility for unstructured reasoning while offloading mechanical data handling to tools, without the cost of fine-tuning.

Key Contributions

1. An agentic workflow-construction framework (LEAN-LLM-OPT). For each query, a classification agent identifies the problem type; a workflow generation agent then dynamically constructs a step-by-step workflow using one or more reference problems from a reference dataset; finally a model generation agent follows that workflow, extracts relevant data with customized tools, and produces the formulation plus Python code.

2. Two new curated datasets/benchmarks. Ref-Data, a 96-entry reference dataset (78 small-scale and 18 large-scale instances) spanning problem types such as resource allocation (RA, 24 entries), mixture (24), others (16), facility location (FLP, 9), assignment (AP, 8), transportation (TP, 7), network revenue management (NRM, 5), and sales-based linear programming (SBLP, 3); and Large-Scale-OR, a 101-entry testing dataset emphasizing medium-size (20–99 variables, 26% of the dataset) and large-size (at least 100 variables, 50% of the dataset) instances.

3. Extensive simulations. Instantiated with GPT-4.1 and the open source gpt-oss-20B, LEAN-LLM-OPT achieves overall accuracy above 76% on Large-Scale-OR, substantially outperforming ORLM, Gemini 3 Pro, and GPT-5.2, and also performs well on the small-scale benchmarks NL4OPT, IndustryOR, and Mamo (Easy and Complex).

4. A real-world use case and ablation study. Applied to a Singapore Airlines fare type capacity allocation and flight network planning problem using the sales-based linear programming (SBLP) formulation on simulated datasets, with new benchmarks Air-NRM-CA (15 instances) and Air-NRM-NP (21 instances), plus an ablation showing that both the agentic workflow and the data-handling tools are important for accuracy.

Main Findings

• Accuracy collapses as inputs grow. Figure 1 shows most models' accuracies decreasing noticeably as input size grows. Gemini 3 Pro and GPT-5.2, the flagship models from Google and OpenAI, both drop below 50% modeling accuracy once the number of input tokens exceeds 800 — far below their advertised input limits of roughly 100,000 to 1,000,000 tokens.

• Strong performance on the new large-scale benchmark. On Large-Scale-OR, LEAN-LLM-OPT instantiated with GPT-4.1 and gpt-oss-20B both achieve overall accuracy above 76%, substantially outperforming ORLM, Gemini 3 Pro, and GPT-5.2. Exact per-model accuracy figures are not reported in the available content.

• Generalization to small-scale benchmarks. The same approach continues to achieve high modeling accuracies on NL4OPT, IndustryOR, and Mamo (Easy and Complex).

• Reference data is genuinely distinct from the test data. Average semantic similarity (cross-encoder score in [0,1], higher is more similar) between Ref-Data and the benchmarks is low: Large-Scale-OR 0.445, NL4OPT 0.476, MAMO Easy 0.479, MAMO Complex 0.358, IndustryOR 0.413.

• Input scale differs by orders of magnitude. Average token counts (o200k_base tokenizer from tiktoken): Ref-Data 422, Large-Scale-OR 47,269, NL4OPT 121, MAMO Easy 217, MAMO Complex 457, IndustryOR 266. Large-Scale-OR inputs are far larger than Ref-Data and all other benchmarks.

• Singapore Airlines use case. On Air-NRM-CA (15 instances), the approach achieves leading accuracies for fare type capacity allocation; on Air-NRM-NP (21 instances) for network planning (joint fare type capacity allocation and flight scheduling), it consistently achieves a small optimality gap. These outperform competing methods in almost all instances.

• Both components matter. The ablation study shows that the agentic workflow and the data-handling tools are each important to high modeling accuracy.

• Classification is helpful but not strictly required. The authors state the classification agent can positively impact modeling accuracy and can be bypassed, with performance reported in Section 4 and discussion in Remark 2 (details not in the available content).

Methodology in Plain English

The method takes a problem description plus its datasets as input and runs three LLM agents in sequence.

1. Classify. A classification agent is shown a demo of how to reason step by step, uses a retrieval tool called FileQA to pull the top 5 most similar problem instances from Ref-Data, and then labels the query as one of the predefined types (or as "Others" or "Mixture"). FileQA is a retrieval-augmented generation (RAG) tool that uses similarity search over Ref-Data.

2. Build a workflow. A workflow generation agent retrieves the most relevant reference instance and turns it into a step-by-step demonstrated procedure. Two flavors exist: a type-tailored workflow (used for all problems except those labeled "Others" or "Mixture") that prescribes a type-specific output structure ("abstract model + retrieved information") and type-aware formatting of the observations, and a type-agnostic workflow (for "Others" and "Mixture") that supplies only the demo query and label and instead builds an abstract model plan from the query and a CSV schema. The provided content is truncated partway through the description of the type-agnostic workflow.

3. Generate the model. A model generation agent follows the workflow, using a second custom RAG tool, CSVQA, which converts input datasets line by line into vector representations and uses similarity search — including fuzzy search — to fetch only the rows the problem actually needs. In the paper's running example, a demo query about Sony products causes CSVQA to retrieve Sony rows while excluding irrelevant Apple rows.

The effect is decomposition: a long, complicated reasoning task becomes a chain of small structured steps, with mechanical data handling offloaded to tools so the LLM's reasoning capacity is spent on the unstructured parts. The running example used throughout is a truck-scheduling problem with 10 trucks, 4 time periods, demand values of 1,500 / 2,000 / 1,800 / 1,000 kg, minimum runtime of 2 hours, cooldown of 1 hour, load fluctuation limit of 300 kg, and a 10% spare capacity buffer.

Why This Matters

Impact on research. The paper reframes large-scale optimization auto-formulation as a workflow-construction problem rather than a fine-tuning problem. It argues that because small changes to descriptions or datasets (splitting a file, adding or deleting rows or columns) can create a completely different instance requiring a new label, building a training set large enough for fine-tuning large-scale problems is "prohibitively large" — so fine-tuning, the dominant prior approach, does not scale here. It also contributes the first comprehensive benchmarks for large-scale optimization auto-formulation, giving the field a shared measuring stick.

Real-world applications (as discussed in the paper):

• Airline revenue management — fare type capacity allocation and flight network planning, in collaboration with Singapore Airlines.
• Supply chain and logistics — including the truck scheduling example and transportation-type problems.
• Retailing and assortment optimization — including the Nike shoe inventory example in Ref-Data used to illustrate network revenue management.
• Resource allocation in domains the authors say span medical, e-commerce, and supply chain management applications.

Industry relevance. Organizations rely on optimization but usually need highly skilled experts to interpret problems, process data, and write models; this is described as a major bottleneck and a significant cost. A plug-and-play framework that supports evolving LLM architectures and avoids GPU rental and large labeled-data generation costs offers a cheaper route to automating expert-level modeling, letting non-experts obtain optimization models through natural-language interaction.

Future Directions

• Closing the long-input gap. Since model accuracy falls below 50% for flagship models past 800 input tokens, an open question is whether workflow construction alone is enough, or whether better long-context processing is required for inputs far larger than those tested.

• Extending beyond linear and mixed integer linear programming. The authors state they currently focus on the prevalent linear programming and mixed integer linear programming but that their approach is general and can go beyond these; dynamic programs, nonlinear models, and other classes remain open.

• Robustness of the two-workflow design. The type-tailored workflow depends heavily on classification accuracy and "typically does not accommodate complex problems involving mixed types," which is why the type-agnostic workflow exists. How well each does, and whether classification can be reliably bypassed, is a live question (the paper points to Section 4 and Remark 2 for this).

• Tool and base-model generalization. The authors note the retrieval tool "can be different depending on our base model and the type of the problem," leaving open how to select or automatically construct tools for new problem families.

Target Audience

Researchers and practitioners at the intersection of operations research and LLM agents — particularly those working on automated optimization modeling, LLM tool use, and agentic workflows. It is also directly relevant to data scientists and OR practitioners in airlines, logistics, retail, and supply chain who want to automate model formulation from data files, and to benchmark builders looking for large-scale evaluation data with reported token-count and semantic-similarity statistics. Readers without any exposure to linear programming or LLM prompting should expect a moderate learning curve.

Authors’ abstract

Large-scale optimization is a key backbone of modern business decision-making. However, building these models is often labor-intensive and time-consuming. We address this by proposing LEAN-LLM-OPT, a LightwEight AgeNtic workflow construction framework for LLM-assisted large-scale OPTimization auto-formulation. LEAN-LLM-OPT takes as input a problem description together with associated datasets and orchestrates a team of LLM agents to produce an optimization formulation. Specifically, upon receiving a query, two upstream LLM agents dynamically construct a workflow that specifies, step-by-step, how optimization models for similar problems can be formulated. A downstream LLM agent then follows this workflow to generate the final output. The agentic workflow leverages common modeling practices to structure the modeling process into a sequence of sub-tasks, offloading mechanical data-handling operations to auxiliary tools. This reduces the LLM's burden in planning and data handling, allowing us to exploit its flexibility to address unstructured components. Extensive simulations show that LEAN-LLM-OPT, instantiated with GPT-4.1 and the open source gpt-oss-20B, achieves strong performance on large-scale optimization modeling tasks and is competitive with state-of-the-art approaches. In addition, in a Singapore Airlines choice-based revenue management use case, LEAN-LLM-OPT demonstrates practical value by achieving leading performance across a range of scenarios. Along the way, we introduce Large-Scale-OR and Air-NRM, the first comprehensive benchmarks for large-scale optimization auto-formulation. The code and data of this work is available at https://github.com/CoraLiang01/lean-llm-opt.