ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

AI summary

Overview

Research area: Operations Research (OR) combined with Large Language Models and reinforcement learning — specifically solver-in-the-loop benchmarks for model debugging and for behavioral rationality in inventory decisions.

Technical level: Advanced. The paper assumes familiarity with linear/mixed-integer programming, Irreducible Infeasible Subsystems (IIS), reinforcement learning with verifiable rewards (RLVR), GRPO, and process reward models.

Scope in one sentence: The paper introduces ORLoopBench, a two-part benchmark suite (OR-Debug-Bench for iterative repair of infeasible LP/MILP code using Gurobi IIS feedback, and OR-Bias-Bench for closed-form inventory decision rationality), and shows that an 8B Qwen3 model trained with solver-verified rewards can outperform frontier APIs on LP repair.

What This Paper Is About

Existing LLM benchmarks treat Operations Research as one-shot translation: a model reads a problem description and writes solver code, with no execution feedback. In real OR practice, the hard work starts when a solver returns Infeasible — the analyst inspects the IIS, finds the conflicting constraints, and iteratively repairs the formulation. This paper builds benchmarks and training methods for that diagnostic loop, and separately tests whether LLMs make rational inventory decisions that match closed-form optima.

Key Contributions

1. A solver-in-the-loop Markov Decision Process for OR debugging. Infeasible-model repair is formalized as a sequential decision problem where each action modifies the formulation, triggers solver re-execution, and returns updated IIS feedback. This shifts evaluation from static natural-language-to-code translation to the diagnostic loop used in practice.

2. Solver-verified training for small models. RLVR is adapted to OR repair with rewards for feasibility recovery, objective preservation, diagnostic accuracy, and faithful use of IIS evidence. An 8B model trained with GRPO reaches 95.3% RR@5 on LP repair versus 92.4% for the strongest frontier API summary result, and improves DA by +14.6 pp in the core 26-model evaluation.

3. The ORLoopBench benchmark suite. OR-Debug-Bench releases 5,362 LP/MILP repair instances spanning LP error types A–I and MILP repair settings. OR-Bias-Bench evaluates operational decision rationality in newsvendor and EOQ inventory settings with in-distribution (ID) and out-of-distribution (OOD) splits.

4. Benchmark findings on transfer and failure modes. The MDP repair framework transfers to MILP, reaching 87.1% RR@5 versus 71.0% for the best API baseline, and the evaluation exposes semantic drift in whole-model code regeneration, where feasible regenerated MILPs can solve the wrong problem.

Main Findings

• 8B beats frontier APIs on LP repair: Qwen3-8B-GRPO reaches 95.3% RR@5 versus 92.4% for Claude Sonnet 4.6, the strongest LP API in the frontier summary (a +2.9 pp gap).
• Better diagnosis, fewer steps: Qwen3-8B-GRPO achieves 62.4% DA versus 47.8% for o4-mini, with 2.25 repair steps versus 3.15 (a +14.6 pp DA gain in the core evaluation).
• "Diagnose once, repair correctly": trained models use 1.3 diagnostic actions per episode versus 2.1 for API models, reflecting systematic elimination rather than trial-and-error.
• Gains concentrate on hard problems: domain-specific training yields +9.6% average on harder error types A, D–G (94.4% vs 84.8%), +14.0% on medium types H, I (95.0% vs 81.0%), and only +3.0% on easy types B, C, where baselines already exceed 95%.
• MILP transfer works: the LP-trained model transfers zero-shot at 78.8% RR@5, and MILP-specific training reaches 87.1% RR@5 versus 71.0% for Claude Sonnet 4.6 as the best API baseline.
• Semantic drift is a real failure mode: on the MILP semantic-drift evaluation, GPT-5.4 reaches Optimal status in 90% of cases but only 28.2% preserve the correct objective; Claude Sonnet 4.6 reaches 85% Optimal but 22.4% correct objective; Gemini 3.1 Pro reaches 3% Optimal and 0.8% correct objective.
• Curriculum learning improves OOD inventory decisions: curriculum training achieves 48% bias reduction (20.0% → 10.4%) on OOD scenarios, a −9.6% drift — the only trained model with substantial OOD improvement.
• Some API models degrade OOD: gpt-5-mini moves from 1.2% ID bias to 53.3% OOD bias (+52.1%), while claude-haiku-4.5 has the lowest ID bias (0.0%) but +3.6% drift.
• Formula-grounded feedback helps EOQ decisions: on the 300-instance EOQ setting with a five-round multi-turn protocol, DeepSeek-R1 reduces bias from 56.9% to 0.5% in 2.6 rounds on average, while GPT-5.2 corrects immediately.
• Token efficiency: Qwen3-8B-GRPO at k=3 (92.1%) already surpasses o4-mini at k=10 (90.7%), using 2,109 tokens per success versus 5,976 (2.8× better).
• Difficulty scaling differs by error type: hard problems improve +26.8% from k=1 to k=5 versus +9.8% for easy problems.
• Ablations: curriculum pre-training gives the largest single gain (+3.5% RR@5), PRM raises DA from 68.0% to 72.7% with similar RR@5 (91.5% to 92.0%), and Curriculum + GRPO (best) reaches 95.3% RR@5 and 62.4% DA (+3.8% over SFT).
• Standard prompting is insufficient: Zero-Shot CoT reaches 23.0% RR@5 and three-shot ICL reaches 38.7%, still 54 points below SFT.

Methodology in Plain English

The researchers built a "saboteur" pipeline that injects controlled errors into otherwise valid linear programs. Each injected error must fail three checks: it must produce a verifiable Infeasible status, produce a non-empty IIS containing the sabotaged constraint, and have a unique ground-truth fix that restores Optimal status. Of the initial candidate pool, 87% pass all validation checks on first generation, with the remainder requiring at most two iterations.

For evaluation, the model does not regenerate code from scratch. Instead it receives a small set of actions — diagnostic queries such as Get_IIS and Check_Slack, repair actions such as Relax, Drop, and Rewrite, and a Submit action to end the episode. After every action, Gurobi 11.0 recomputes the IIS, so the next state is a deterministic result of solver execution rather than a heuristic judgment. Episodes allow up to 50 steps.

Training proceeds in two stages. First, supervised fine-tuning on 696 successful debugging trajectories (from 1,247 total, a 55.8% acceptance rate) collected from three teacher models: GPT-5.2-chat (40%), o4-mini (35%), and DeepSeek-R1 (25%). Second, GRPO training with a composite reward of 0.5·R_outcome + 0.3·R_diagnosis + 0.2·R_efficiency, where R_outcome is +100 for Optimal and −50 otherwise, R_diagnosis is DA·100, and R_efficiency is −1 per step. A faithfulness penalty of −20 discourages repairs targeting non-IIS constraints. Training converges after 4 epochs with RR@5 = 95.0%.

For the inventory side, the team generated newsvendor scenarios with controlled critical ratios using Q* = μ + σ·Φ⁻¹(CR) and CR = (p−c)/(p−s), then trained with a three-stage curriculum. A process reward model (PRM) scored individual debugging steps, achieving AUC-ROC of 0.94 on held-out labels. The foundation model was Qwen3-8B-Instruct, selected via a 100-sample pilot study where Qwen3-8B went from 51.2% base RR@5 to 93.1% after SFT, a +41.9% headroom. Experiments ran on 2× A100 80GB with SGLang inference (TP=2, concurrency=16).

Why This Matters

Impact on research. The paper reframes OR evaluation from static code generation to an interactive diagnostic loop with a deterministic oracle. Because Gurobi's IIS is a minimal certificate of infeasibility, every repair step is automatically verifiable — unlike self-generated feedback or sampled unit tests. This gives the self-correction literature a domain where feedback is noise-free and step-level progress is measurable through IIS size reduction, and it separates genuine root-cause diagnosis from lucky feasibility restoration.

Real-world applications.

• Debugging infeasible optimization models in supply chain and production planning, where analysts currently inspect IIS output manually.
• Inventory management, including newsvendor-style ordering and economic order quantity (EOQ) decisions, where LLM advisors show measurable pull-to-center bias.
• Automated translation of business-rule descriptions into solver code that must remain semantically faithful after repair.
• Auditing regenerated models for semantic drift — code that solves to Optimal but encodes the wrong objective or constraint semantics.

Industry relevance. The trained model uses local deployment that avoids per-call API charges in the authors' setup, with training cost of roughly 8 GPU-hours on 2× A100 that amortizes across high-volume evaluation. The paper emphasizes human-in-the-loop deployment requiring audit logs, data governance, solver/version reproducibility, and practitioner validation. Its EOQ result — DeepSeek-R1 cutting bias from 56.9% to 0.5% in 2.6 rounds — suggests that formula-grounded feedback can correct some, but not all, operational decision biases.

Future Directions

• Extend beyond LP/MILP infeasibility. The authors note that nonlinear, stochastic, robust, and multi-objective formulations require different certificates than IIS provides.
• Address incomplete diagnosis. IIS is a minimal infeasible subset rather than a complete causal explanation, and multiple IIS sets may exist; Type H–I failures often involve large IIS sets where one repair exposes another conflict.
• Hybridize model and optimizer. Repair-magnitude errors suggest combining a model that identifies the faulty constraint with an optimizer that computes the smallest valid adjustment.
• Broaden scope. The conclusion lists natural extensions: MINLP and stochastic debugging, multi-period operations, RAG integration with OR knowledge bases, and practitioner-in-the-loop validation. New benchmarks could also cover multi-agent debugging, since the present framework is single-agent.

Target Audience

This paper suits researchers and practitioners at the intersection of LLM agents, reinforcement learning with verifiable rewards, and optimization. It is most useful to those working on tool-use agents with deterministic oracles, to OR practitioners who want automated infeasibility diagnosis, to benchmark designers interested in process-level rather than outcome-level evaluation, and to teams evaluating whether LLMs can be trusted for inventory or operations management decisions. Readers focused on closed-form behavioral economics, nonlinear optimization, or production-deployment simulation will find the current coverage explicitly limited by the authors.

Authors’ abstract

Operations Research practitioners debug infeasible models through an iterative process: inspecting Irreducible Infeasible Subsystems ( IIS), identifying constraint conflicts, and repairing formulations until feasibility is restored. Existing LLM benchmarks mostly treat OR as one-shot translation from problem descriptions to solver code, omitting this diagnostic loop. We formalize infeasible-model repair as a solver-in-the-loop Markov Decision Process in which each action triggers solver re-execution and IIS recomputation, yielding deterministic, verifiable feedback. We introduce ORLoopBench, a benchmark suite with two components: OR-Debug-Bench releases 5,362 LP/MILP repair instances, while OR-Bias-Bench evaluates closed-form operational decision rationality across inventory settings. Solver-verified RLVR training enables an 8B model to surpass frontier APIs on LP repair (95.3% vs 92.4% RR @5), improves diagnostic behavior, and transfers to MILP repair. The same evaluation exposes semantic drift in whole-model code regeneration: feasible regenerated MILPs can solve the wrong problem. Process-level evaluation with solver oracles enables targeted training for reliable OR self-correction.