OR-R1: Automating Modeling and Solving of Operations Research Optimization Problem via Test-Time Reinforcement Learning

AI summary

Overview

• Research area: Large language models for Operations Research (OR) — specifically automated translation of natural-language optimization problems into formal mathematical models and executable solver code, combined with reinforcement learning at test time.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning (SFT), Group Relative Policy Optimization (GRPO), PPO-style clipping, KL regularization, PEFT/LoRA, and solver-code execution.
• Scope (one sentence): The paper introduces OR-R1, a two-stage framework that combines supervised fine-tuning with Test-Time Group Relative Policy Optimization (TGRPO) on unlabeled data to achieve state-of-the-art optimization modeling and solving accuracy using roughly one tenth of the synthetic training data required by prior methods.

Paper details: arXiv:2511.09092v1 [cs.AI], 12 Nov 2025. Authors: Zezhen Ding, Zhen Tan, Jiheng Zhang (corresponding), Tianlong Chen (corresponding). Affiliations: The Hong Kong University of Science and Technology; Arizona State University; University of North Carolina at Chapel Hill. Code: https://github.com/SCUTE-ZZ/OR-R1. License: CC BY 4.0.

What This Paper Is About

Turning a written description of an operations research problem (for example, a logistics or scheduling scenario) into a precise mathematical model and working solver code has traditionally required specialized human expertise and is time-consuming and error-prone. Recent LLM-based methods can automate parts of this pipeline, but they depend on very large amounts of annotated or synthetic training data, and their single-attempt outputs are inconsistent — the model often needs several attempts to produce the right answer. OR-R1 targets both problems at once: it aims to reach state-of-the-art solving accuracy with far less labeled data, while making single-attempt (Pass@1) predictions more reliable.

Key Contributions

1. OR-R1 framework. The paper introduces OR-R1, described as the first framework to integrate Supervised Fine-Tuning (SFT) with Test-Time Group Relative Policy Optimization (TGRPO) for automated operations research modeling and solving. The paper states this is also the first systematic application of test-time reinforcement learning and adaptation techniques to automated OR problem solving.
2. A multi-faceted reward system designed for optimization problems. Three components are combined: a Format Reward for structural correctness (the proportion of six required output fields present: ## Mathematical Model:, ## Decision Variables:, ## Objective Function:, ## Constraints:, ## Python Code Solution Using 'coptpy':, and ''' python), a Valid-Code Reward (binary: whether the generated code can correctly call coptpy), and a Majority Voting Reward (whether the executed result matches the consensus output across candidate generations, derived from the Test-Time Reinforcement Learning framework of Zuo et al. 2025). The final reward is the sum of the three.
3. State-of-the-art results with dramatically less data. OR-R1 attains an average solving accuracy of 67.7% using only 1/10 of the synthetic data required by ORLM, exceeding ORLM's solving accuracy by up to 4.2%, and outperforming ORLM by over 2.4% with just 100 synthetic samples.
4. Improved output consistency. TGRPO contributes an additional 3.1%–6.4% accuracy improvement and narrows the gap between single-attempt (Pass@1) and multi-attempt (Pass@8) performance from 13% to 7%.

Main Findings

• Best overall average. OR-R1 SFT(3K)-TGRPO achieves an average accuracy of 67.7% (±2.7) across the eight benchmarks, compared with ORLM(Llama3-8B) at 63.5% and LLMOPT(Qwen2.5-14B) at 60.1%.
• Strong results from very little data. OR-R1 SFT(100)-TGRPO reaches 65.9% (±2.2) average accuracy with only 100 SFT samples, still surpassing ORLM and LLMOPT.
• Per-benchmark highlights for OR-R1 SFT(3K)-TGRPO: NL4OPT 88.3 (±0.9), MAMO EasyLP 86.1 (±1.0), MAMO ComplexLP 49.9 (±15.0), IndustryOR 35.3 (±2.9), NLP4LP 84.6 (±0.8), ComplexOR 46.3 (±3.2), OptiBench 62.9 (±1.0), ICML Competition 88.3 (±1.8). The paper reports optimal performance on NL4OPT, MAMO ComplexLP, IndustryOR, NLP4LP, OptiBench, and ICML Competition. OR-R1 does not lead on MAMO EasyLP (LLMOPT reports 89.5) or on ComplexOR (ORLM reports 50.0).
• Base model matters, but TGRPO adds on top. After identical SFT with 3K data, Qwen3-8B (64.6% average) outperforms Qwen2.5-7B (61.9%) and Llama3-8B (58.2%); TGRPO then improves performance by 3.1%–6.4% regardless.
• Reward ablation (Table 2, base = Qwen3-8B SFT(3K), 66.0% average). The voting reward produces the largest single-component gain (68.0%, +2.0), followed by the code reward (67.2%, +1.2) and the format reward (66.5%, +0.5). Combining all three rewards gives the best result: 70.8%, +4.8 over baseline. Pairwise combinations give 67.8% (format+code), 68.5% (format+voting), and 69.7% (code+voting).
• Reward training dynamics. The format reward stays consistently high, above 0.98 throughout training; the valid-code reward rises early and stabilizes around 0.9; the majority voting reward starts around 0.7 and stabilizes near 0.8.
• Consistency improvement. Pass@1 improves consistently during TGRPO training and the Pass@8 versus Pass@1 gap falls from 13% to 7%. The Pass@1 curve is still rising when training stops.
• Data scale effect. Increasing TGRPO training samples from 10 to 50 per test set raises accuracy from 66.0% to 69.1%; adding more data yields diminishing returns (all models trained for 160 steps). The authors conclude TGRPO needs only a small amount of in-domain data and no additional labeled data.
• Inference setting. The paper reports solution accuracy as the primary metric, where a problem counts as solved only if the predicted optimal objective value exactly equals the ground truth. Values with "±" in the main table are the mean and standard deviation over three independent training runs.
• Not reported in the provided content: the full hyperparameter settings, which the paper defers to an appendix.

Methodology in Plain English

OR-R1 has two stages.

Stage 1 — Supervised Fine-Tuning (SFT). The base model, Qwen3-8B, is fine-tuned on a small, randomly selected subset of the ORInstruct dataset from ORLM. The paper's main configuration uses 3,000 synthetic samples (1/10 of the full dataset ORLM uses), and it also runs a low-data configuration with just 100 samples. The SFT objective is the standard negative log-likelihood loss over input–output pairs — essentially teaching the model the reasoning patterns for problem formulation and code generation. Training uses the AdamW optimizer, a warmup-decay scheduler, and standard settings.

Stage 2 — Test-Time Group Relative Policy Optimization (TGRPO). This stage runs on unlabeled test data. For each problem, the model samples a group of candidate outputs containing both a mathematical model and Python solver code. Each code block is extracted and executed to obtain a candidate optimal value. A majority vote over the executed results produces a consensus "pseudo-label." Three rewards are computed from these outputs — format adherence, code validity, and agreement with the majority vote — and summed. The model is then updated with a GRPO-style objective: it maximizes a clipped importance-weighted advantage term and subtracts a KL divergence penalty against a fixed reference policy (typically the SFT model). Because the baseline is estimated from group scores rather than a learned critic, TGRPO avoids the cost of a separate value model. This stage uses AdamW with a cosine scheduler and PEFT (LoRA). Both stages are trained on 4×A100 (40G) GPUs with BF16 precision.

The intuition: instead of paying for more labels, the framework lets the model generate many answers, checks which ones agree when actually run through a solver, and rewards that consistency — turning unlabeled data into a training signal.

Why This Matters

Impact on research. The paper shows that test-time reinforcement learning, previously explored for math and code reasoning generally, transfers to the structured domain of operations research, and that it can substitute for large volumes of annotated or synthetic data. It also provides a concrete reward design (format + executability + majority-vote consensus) that other domains with verifiable, executable outputs could borrow. A secondary research point is the explicit focus on consistency: the paper argues that the Pass@8 versus Pass@1 gap reflects latent capability that single-attempt generation fails to surface, and demonstrates that reward-based training can close much of that gap.

Real-world applications (domains the paper names as central to OR):

• Logistics — translating delivery or routing problem descriptions into solvable models.
• Resource allocation — automatically formulating and solving allocation problems.
• Scheduling — generating and solving scheduling formulations.
• Industrial decision making more broadly — the paper frames the general goal as lowering the expertise and data barrier for industrial OR applications, including the IndustryOR benchmark of 100 real-world OR problems.

Industry relevance. Building a competitive optimization-modeling assistant normally requires either expensive expert annotation or very large synthetic datasets whose quality can be questionable. OR-R1's reported ability to be competitive with roughly one tenth of ORLM's synthetic data, and to beat ORLM with only 100 samples, points toward cheaper deployment paths — and the fact that the second training stage consumes unlabeled in-domain data means a company can adapt the model to its own problem distribution without commissioning new labels.

Future Directions

• Longer TGRPO training. The authors state that computational resource constraints forced them to limit training duration, and that the Pass@1 curve was still trending upward with no plateau reached. Whether extended training continues to pay off is left open.
• Generalization beyond the current setup. The reward for executable code is defined specifically around correctly calling coptpy. Extending the valid-code reward to other solver libraries and to other formulations (the paper discusses MILP and LP) is an obvious next step but is not demonstrated.
• Understanding the data-scale ceiling. TGRPO accuracy improves from 66.0% to 69.1% as samples per test set grow from 10 to 50, then stops improving. The paper does not explain why additional in-domain data brings diminishing returns.
• Reducing reported variance. Several per-benchmark results carry large standard deviations across three training runs — most notably MAMO ComplexLP at 49.9 (±15.0) — suggesting training stability is an unresolved question.

Target Audience

• LLM researchers working on post-training, reinforcement learning from verifiable rewards, test-time adaptation, and self-training or pseudo-labeling.
• Operations research and optimization practitioners interested in automating model formulation and solver-code generation.
• Applied machine learning engineers in industry who need domain-specific optimization assistants but cannot afford large-scale annotation.
• Graduate students and advanced readers who already understand supervised fine-tuning and reinforcement learning basics; the methods section uses GRPO-style math without extensive introductory scaffolding, so beginners may need background reading first.

Authors’ abstract

Optimization modeling and solving are fundamental to the application of Operations Research (OR) in real-world decision making, yet the process of translating natural language problem descriptions into formal models and solver code remains highly expertise intensive. While recent advances in large language models (LLMs) have opened new opportunities for automation, the generalization ability and data efficiency of existing LLM-based methods are still limited, asmost require vast amounts of annotated or synthetic data, resulting in high costs and scalability barriers. In this work, we present OR-R1, a data-efficient training framework for automated optimization modeling and solving. OR-R1 first employs supervised fine-tuning (SFT) to help the model acquire the essential reasoning patterns for problem formulation and code generation from limited labeled data. In addition, it improves the capability and consistency through Test-Time Group Relative Policy Optimization (TGRPO). This two-stage design enables OR-R1 to leverage both scarce labeled and abundant unlabeled data for effective learning. Experiments show that OR-R1 achieves state-of-the-art performance with an average solving accuracy of $67.7\%$, using only $1/10$ the synthetic data required by prior methods such as ORLM, exceeding ORLM's solving accuracy by up to $4.2\%$. Remarkably, OR-R1 outperforms ORLM by over $2.4\%$ with just $100$ synthetic samples. Furthermore, TGRPO contributes an additional $3.1\%-6.4\%$ improvement in accuracy, significantly narrowing the gap between single-attempt (Pass@1) and multi-attempt (Pass@8) performance from $13\%$ to $7\%$. Extensive evaluations across diverse real-world benchmarks demonstrate that OR-R1 provides a robust, scalable, and cost-effective solution for automated OR optimization problem modeling and solving, lowering the expertise and data barriers for industrial OR applications.