OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

AI summary

Overview

Research area: Benchmarking and evaluating large language model (LLM) agents on real-world operations research (OR) work — requirement elicitation, mathematical formulation, solver implementation, and delivery.

Technical level: Advanced. The paper assumes familiarity with optimization modeling, feasibility/objective metrics, and agentic coding harnesses.

Scope (one sentence): The paper introduces OR-License, a 31-instance, 10-industry benchmark built from paid OR consulting engagements ($3.68 million in project value), together with a client simulator (ORSim) and a feasibility-gated metric (Deployment-Adjusted Quality), and uses it to evaluate 11 frontier LLMs.

What This Paper Is About

Most OR benchmarks hand a model a self-contained optimization problem, or a problem already formulated by an expert. Real OR practice starts instead with messy client materials: incomplete requirements, ambiguous terminology, undefined parameters, and business facts that only the client holds. This paper asks whether LLM agents can perform that full translation — from original client materials to a validated, executable solution — and builds a benchmark that preserves the original materials, data, deliverable formats, and information boundaries from real paid engagements.

Key Contributions

1. A dataset grounded in paid industrial OR work. 31 instances across 10 industrial scenarios, derived from commissioned OR projects with $3.68 million in total project value, de-identified by experts with at least three years of professional OR experience while preserving the original requirements, input data, deliverable formats, and information boundaries.

2. ORSim, a controlled client simulator. An interaction environment that discloses client-held business facts throughout formulation, solving, and refinement, but never supplies OR expertise. Each query discloses at most three non-brief facts; a simulator-side LLM selects candidate facts from a closed catalog, and a deterministic controller enforces ownership, visibility, and per-query disclosure rules before a separate response-only LLM renders the answer in natural language.

3. An end-to-end evaluation framework with hidden checking. Agents work inside coding-agent harnesses (Claude Code as primary, with a Codex comparison), may call ask through the Bash interface, and submit executable decisions that are checked by hidden, instance-specific verifiers and scored against objective functions recomputed from the submitted output.

4. Deployment-oriented metrics and trajectory analysis. Deployment-Adjusted Quality (DAQ), which gives zero credit for infeasible deliveries, plus a Reliability–Quality Profile, and manual annotation of client-question timing, research-chain (HCAEU) completion, and eight categories of error origin.

Main Findings

• Feasibility, not objective quality, separates models. Across the 11 models under Claude Code, feasibility spans 43.23 percentage points (from 45.16% for Hy3 to 88.39% for Grok 4.6), while conditional quality spans only 4.09 points (from 96.92% to 101.01%).

• Several models beat the expert on conditional quality but none matches expert DAQ. Models including Gemini 3.8 Flash (100.43%), GPT-5.6 Sol (100.59%), GLM-5.3-Flash (101.01%), and Claude Opus 5 (100.03%) exceed the 100% expert quality reference on their own feasible subsets. But the expert reference has DAQ = 1, and the best model, Grok 4.6, reaches only 86.55%.

• The score gap is mostly failed delivery, not weak optimization. Using DAQ = Feas × Q_feas, Grok 4.6's 13.45-point gap decomposes into 11.61 points from infeasible deliveries and 1.84 points from net objective shortfall on feasible ones — roughly 86% of the deficit comes from zero credit on failed deliveries.

• Rankings are stable under reliability penalties. All eleven models retain the same ordering across the 21 values of α from 1.0 to 3.0, while scores decrease and remain below the expert reference.

• Research chains are rarely completed. Task-equalized complete HCAEU (hypothesis, explanation, experiment, evidence, update) chain proportions range from 24% for Muse Spark 1.3 to 45% for Grok 4.6, staying below 50% for every model. GPT-5.6 Sol forms no testable hypothesis in 41% of opportunities; Qwen3.8-Max stops at HC and at HCAE in 21% each, and completes the chain in only 27% of opportunities despite having the highest conditional stage quality.

• High stage quality does not imply sustained inquiry. GPT-5.6 Sol pairs a DAQ of 85.02% with relatively low conditional stage quality, while Qwen3.8-Max has the highest conditional stage quality but a lower DAQ of 77.97% — the two dimensions rank models differently.

• Client questions are heavily front-loaded. Stage S1 (before solver code) accounts for 86.7%–98.3% of questions across models. Only about 13% of questions occur after S1 for Hy4 Preview and GPT-5.6 Sol, compared with 9% for DeepSeek-V4.1-Flash.

• Later questions resolve more issues when they happen. Confirmed issue-resolution proportions range from 10.6% to 37.0% at S1, versus approximately 56% for DeepSeek-V4.1-Flash at S4 and 71% for Hy4 Preview at S5.

• Errors concentrate in formulation, not just coding. Implementation and delivery (IMP) is the largest error category for only two of eleven models. Static constraint errors (CON) are 42.7% of Claude Opus 5's error events; dynamic and state-transition errors (DYN) are 28.7% for Hy4 Preview and 25.5% for GLM-5.3-Flash; objective/preference (OBJ) plus uncertainty/information structure (UNC) together account for 69.0% of Grok 4.6's and 70.5% of Muse Spark 1.3's error events. IMP remains largest for GPT-5.6 Sol and Hy3 (33.3% and 33.1%).

Methodology in Plain English

The authors collected real OR consulting projects and had experienced practitioners strip out identifying details while keeping the original requirements, data files, formats, and — importantly — the original gaps in information. Each task ships with a short business brief and the input files. Behind the scenes sit hidden constraint checkers and hidden objective functions, plus the original expert solution that the client had already accepted.

Agents solve each task inside a coding-agent harness with a normal shell. When an agent needs a fact it does not have, it can ask a simulated client. The simulator is deliberately constrained: a closed catalog of expert-annotated client facts defines what may ever be said, a rule-based controller decides what is allowed to be revealed on that query, and a separate language model only phrases the authorized content. The simulator can give business facts but cannot propose a formulation, pick an algorithm, or write solver code.

A submission counts as feasible only if it passes the hidden verifier on every official test case of that instance — a partial pass scores zero. Quality is measured as a ratio against the expert's objective, oriented so that 1.0 equals the expert, and it is only counted when the instance is feasible. Averaging that gated score over all instances gives Deployment-Adjusted Quality, which equals feasibility times conditional quality. Because that plain average lets a few strong instances compensate for failures, the authors also sweep a power-mean family over 21 values of α from 1.0 to 3.0, which penalizes low and zero scores more heavily.

Beyond scores, the authors manually labeled trajectories along three axes: when each question was asked (six workflow stages from before solver code to before final submission), whether a full hypothesis–explanation–experiment–evidence–update chain was completed for each research opportunity, and where each confirmed error originated (eight categories: problem understanding, data mapping, decision representation, static constraints, dynamics, uncertainty, objective formulation, implementation).

Why This Matters

The paper reframes OR agent evaluation: the hard part is not writing a solver once the model is given, but extracting the right model from messy client material and information that only the client holds. It also shows that a benchmark scoring only final objective values would badly misrank these systems, since conditional quality is nearly flat (a 4.09-point spread) while feasibility varies by 43.23 points.

Real-world applications:

• Logistics and routing: translating a shipper's informal operating rules and data exports into a constraint-correct routing model.
• Manufacturing and supply planning: converting production planning documents into executable optimization with the correct state transitions and business rules.
• Retail and pricing/promotion decisions: recovering objectives and preference structure that clients state loosely, a category the error analysis flags as dominant for several models.
• Financial and asset-return tasks: the benchmark includes instances whose objective is cumulative return, normalized via a net-asset-value factor.

Industry relevance: OR deployments are specialist work — the paper cites a Gurobi 2024 report finding that 82% of survey respondents hold a master's or doctoral degree. A benchmark that measures whether agents can produce client-acceptable deliveries, with an explicit penalty for infeasible output, is directly relevant to consultancies, enterprise analytics teams, and vendors of optimization tooling who want to know what can safely be delegated.

Future Directions

• Sustained information-gap resolution. Agents overwhelmingly ask questions before writing solver code; the open question is how to make them revisit business assumptions when execution feedback or intermediate results suggest the formulation is wrong.

• Faithful formulation over runnable code. Since constraint, dynamics, objective, and uncertainty errors dominate for most models, work is needed on keeping business semantics consistent through formulation, implementation, and validation — not just producing a program that runs.

• Evidence-linked self-improvement. The HCAEU analysis shows agents often interpret evidence without converting it into a revision; closing that last step, and raising chain completion above the observed 24%–45% range, is an explicit target.

• Harness and evaluation generalization. The paper includes an additional Codex harness comparison and a full reliability–quality profile as sensitivity analyses; whether conclusions hold under other harnesses, larger instance sets, or different α ranges remains open.

Target Audience

Researchers building and evaluating LLM agents for structured professional work; OR and optimization practitioners curious about what current models can and cannot deliver; benchmark designers interested in information-boundary simulation and feasibility-gated metrics; and enterprise or consulting teams assessing whether agentic systems can take on optimization delivery work. Readers without OR background will find the framing accessible, but the metric definitions and error taxonomy assume some familiarity with optimization modeling.

Authors’ abstract

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.