Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization

AI summary

Overview

Research area: Operations research (OR) modeling with large language models (LLMs), specifically the underexplored step that happens before a mathematical program is written — deciding whether a natural-language business request contains enough information to formulate the problem at all.

Technical level: Intermediate. The paper assumes familiarity with optimization modeling concepts (objectives, constraints, decision variables, feasible regions) and with LLM agent benchmarking, but its central idea is stated plainly enough for a motivated reader from either side.

Scope: The paper defines and benchmarks "pre-formulation clarification," then proposes a two-stage agent framework (InterOPT) for deciding what to ask and when to stop asking before formulating an optimization model.

What This Paper Is About

When non-experts describe business problems to an LLM and ask for an optimization model, their descriptions are usually incomplete: an objective may be missing, a routing rule may be ambiguous, or a hard-versus-soft policy may never be stated. Because these omissions can change the mathematical structure of the resulting program, an agent that silently fills them with defaults, or that declares itself ready too early, can produce a formulation for the wrong problem. This paper turns that failure mode into an explicit research task — pre-formulation clarification — and asks how an agent can recover the missing formulation-critical facts through bounded interaction while knowing when to stop.

Key Contributions

1. The pre-formulation clarification task and the InterOPT framework. The paper formalizes clarification as a joint problem: judging whether a public specification is "formulation-complete" (all plausible business completions induce the same formulation structure) and, when it is not, recovering the missing facts with minimal interaction. InterOPT is a two-stage framework in which Stage 1 (Dynamic Gap Search) identifies and tracks unresolved formulation gaps in a persistent ledger, and Stage 2 (Gap-Guided Action Search) uses that ledger state to decide what to ask or whether to emit READY_TO_MODEL. The two stages are deliberately separated so that gap diagnosis and interaction control are not conflated.

2. The OR-Clarify benchmark and evaluation framework. OR-Clarify converts fully specified, source-grounded optimization tasks into controlled clarification instances by decomposing each task into single-fact statements, screening out facts whose meaning is already implied by visible information, masking roughly half of the eligible facts with a deterministic pseudorandom procedure using a fixed case-specific seed, and generating public briefs, fact-bounded simulated-user responses, and slot-level evaluation rubrics. It supports both an open/free-form protocol and a Choice protocol with candidate answer options.

3. A severity taxonomy for hidden slots. Each masked fact becomes a hidden slot labeled P0 (blocking, e.g. whether a route is open or closed), P1 (substantive modeling condition, e.g. whether unmet demand is penalized), or P2 (secondary boundary or interpretive condition). Severity is assigned only after masking, so it does not influence which facts are hidden.

4. A systematic empirical study. The paper compares InterOPT against LLM baselines and task-aligned adapters of prior systems, and reports exact slot recovery, readiness/stoppping behavior, silent assumptions, and interaction cost, plus system-level ablations of its two stages.

Main Findings

• The benchmark scales to 100 cases and 178 hidden slots. OR-Clarify comprises 100 clarification cases and 178 hidden slots, with 1–5 slots per case (mean 1.78): 75 P0, 83 P1, and 20 P2 slots. Human auditing verifies slot boundaries, severity labels, answer support, and rubric consistency.

• Off-the-shelf models remain far from solving the task. Under Open/FreeQA, Opus-4.8 reaches the best reported All-Slot Exact of 0.548 and Core Exact of 0.583; under Choice/MC-D it reaches 0.542 and 0.583. The paper states that no model exceeds 60% Core Exact and that all retain substantial silent assumptions, concluding OR-Clarify remains challenging.

• InterOPT substantially outperforms baselines in the Choice setting. Under Choice, InterOPT reaches All-Slot Exact 0.638 and Core Exact 0.675 with Silent/run 0.366, versus MC-D at 0.474 / 0.506 / 0.692 and ReadyGate at 0.476 / 0.517 / 0.614. This is the paper's headline result.

• InterOPT is competitive but not best in the open-ended setting. Under Open, ORPilot attains the strongest recovery (All-Slot Exact 0.546, Core Exact 0.583), GATE remains competitive (0.528 / 0.560), and InterOPT lands at 0.492 / 0.538 while still leaving more silent assumptions than the strongest open baselines. The paper states plainly that formulation-gap-guided clarification does not uniformly dominate strong free-form baselines.

• Recovery comes at a cost, described as a coverage–cost trade-off. In the Choice setting InterOPT uses 10.674 Avg Turns and 10.036 Avg Q, far more than MC-D (3.258 and 2.350). The paper frames this honestly: structured gap-guided clarification improves recovery most under constrained answer spaces, but stopping and question efficiency remain limiting factors.

• Budget-matched curves qualify the endpoint numbers. Figure 4 compares methods at the same cumulative number of atomic questions within a protocol. The paper notes that under Choice, InterOPT continues gaining exact recovery over later questions after MC-D and ReadyGate largely plateau, and that its endpoint advantage should be read together with the overlapping portions of the curves.

• Both stages matter, and their contributions differ by protocol. Under Open, removing Stage 1 yields 0.460 Core Exact, while removing Stage 2 yields 0.506 Core Exact at 9.514 Avg Turns and 9.502 Avg Q — nearly twice the interaction — with a much lower Silent/run of 0.254. The paper's discussion of these ablations is truncated in the provided content after noting the turn increase under the open protocol.

• Choice protocols can actively suppress clarification. MC (which forces the user to choose the closest of options A–C) shows the lowest recovery (All-Slot Exact 0.414, Core Exact 0.447) and the highest Silent/run (0.774) among the Choice methods, while MC-D's added free-form option D improves matters (0.474 / 0.506 / 0.692).

• The information boundary is strictly enforced. Tested agents see only the public brief and public transcript; the simulated user never volunteers unasked hidden facts and answers only from private case facts; hidden slots, severity labels, and judge rubrics never enter the agent. Facts volunteered without being requested, assumptions introduced only in the final model, vague catch-all questions, and partial matches receive no exact-recovery credit.

Methodology in Plain English

The authors start from complete, source-grounded OR tasks and break each one into individual facts expressing a single requirement. Facts about the business setting and numerical inputs stay visible to the agent; facts about objectives, constraints, and assumptions become eligible for hiding. A fact is excluded from hiding if its meaning is already pinned down by visible information — for instance, a stated 15,000 available production hours already implies an upper bound, whereas a stated demand of 1,000 units does not reveal whether it must be met exactly or may be backlogged. Eligible facts are then masked by a deterministic pseudorandom procedure with a fixed per-case seed, hiding roughly half of them, and the selection is frozen so every method faces identical missing information.

Each masked fact becomes a hidden slot with supporting evidence, a simulated-user answer restricted to that fact, acceptable question examples, a semantic recovery rule, and a severity label. An agent then interacts under a strict boundary: it sees only the brief and the transcript, asks questions or declares READY_TO_MODEL, and the user answers only what is asked, drawing on private facts. Interaction ends at READY_TO_MODEL or at T_max turns. Afterwards, a judge compares the frozen transcript against the frozen hidden-slot rubrics and awards exact-recovery credit per slot, while a separate protocol detector checks format without supplying recovery information.

InterOPT sits on top of this harness. Each turn, Stage 1 (a structured LLM call followed by deterministic validation) searches the public brief and transcript for unresolved requirements across six categories — objectives and trade-offs, decision scope, operational constraints, time boundaries, relationships among entities or decisions, and hard-versus-soft policies — proposes at most three new gaps, deduplicates them by normalized text, and assigns them the status Open. Stage 2 then generates three candidate clarification states and three corresponding gap-bound questions, with a selector choosing one based on potential changes to objectives, business constraints, decision scope, and silent-assumption risk. Only the agent itself decides Ask versus READY_TO_MODEL; the ledger is not allowed to block a stop, and an empty ledger does not force readiness. Asked gaps are marked Asked (queried), not resolved. Invalid outputs are regenerated up to three times.

For evaluation, the paper runs 100 cases with five runs per case, T_max = 20, tested model DeepSeek V4 Pro, agent temperature 0.2, and simulator/judge/selector temperature 0.0. The primary metric is Core Exact, which requires all P0/P1 slots in a core-eligible case to be exactly recovered and covers 94 of the 100 cases; All-Slot Exact applies the same all-or-nothing criterion to every slot including P2. Diagnostics include Silent/run, Avg Turns, Avg Q, and stopping labels of premature, appropriate, over-questioning, or no-stop.

Why This Matters

Impact on research. Most LLM-for-optimization benchmarks assume the problem statement is already complete, so they measure formulation and solving while skipping the prior question of whether the description supports any meaningful formulation. OR-Clarify makes that prior question measurable and, to the authors' knowledge, is the first OR-specific framework to jointly evaluate formulation-gap recovery, readiness decisions, and interaction cost before LLM-driven autoformulation. It also positions "is the current information sufficient to build the correct model?" as a distinct question from generic abstention ("can I answer this?").

Real-world applications.

• Vehicle routing and dispatch: deciding whether routes are open or closed, and whether time windows are hard or soft — omissions that change the constraint structure.
• Multi-period production planning: clarifying whether stated demand must be met exactly or may be backlogged, and whether unmet demand is penalized.
• Fleet scheduling: resolving secondary boundary interpretations such as whether vehicles may be scheduled across day boundaries (a P2-type condition).
• LLM modeling copilots in general: any system that turns a stakeholder's narrative problem description into a solver-ready optimization model.

Industry relevance. The work targets the practical gap between a business stakeholder's narrative and a solver-ready model — the step where OR consultants and internal analytics teams typically spend interview time. Its severity taxonomy (a P0 omission can change the problem itself) offers a concrete way to triage which missing facts must be resolved before committing to a model, and its explicit accounting of silent assumptions surfaces a risk that is invisible in one-shot formulation pipelines. The author affiliations (Cardinal Operations and Shanghai Jiao Tong University) point to direct commercial interest in OR modeling tooling.

Future Directions

• Closing the coverage–cost gap. InterOPT's Choice-setting recovery advantage comes with 10.674 Avg Turns and 10.036 Avg Q, and the paper explicitly names stopping and question efficiency as limiting factors. Better stopping policies are the obvious next target.
• Reducing silent assumptions in the free-form setting. InterOPT's Silent/run of 0.560 under Open exceeds the strongest open baselines, and the w/o Stage 2 variant achieves a markedly lower Silent/run of 0.254 at much higher interaction cost — suggesting a tunable trade-off worth studying.
• Scaling and diversifying the benchmark. The paper states the same construction procedure (decomposition, screening, masking, annotation) can be applied to additional complete, source-grounded OR task records, implying the current 100 cases and 178 slots are a first instantiation rather than a ceiling.
• Connecting clarification to downstream modeling quality. OR-Clarify evaluates recovery, readiness, silent assumptions, and interaction cost, but stops before the formulation is actually built and solved; whether better slot recovery translates into better models and decisions is left open.

Target Audience

Researchers working on LLM agents, tool use, and interactive problem solving; operations research and optimization practitioners who build or consume autoformulation systems; benchmark designers interested in underspecification, clarification, and abstention evaluation; and product teams building LLM-based modeling assistants for enterprise decision support. Readers from OR will find the problem framing immediately recognizable, while readers from the LLM side will find the severity taxonomy and the strict information boundary useful design patterns.

Note on the provided content: the paper text supplied here is truncated mid-sentence during the discussion of the Table 3 ablations, and it does not report model sizes, training details, wall-clock or dollar costs, or the full open-protocol ablation analysis.

Authors’ abstract

Large language models (LLMs) are increasingly used to formulate optimization models from natural-language problem descriptions, yet realistic operations research (OR) requests are often incomplete: missing objectives, constraints, or business rules can change the resulting mathematical program. Existing evaluations largely assume a complete specification and therefore overlook whether an agent knows when clarification is needed before modeling. We introduce OR-Clarify, a benchmark for pre-formulation clarification. Each task presents a partial public problem description, withholds structured hidden slots, and evaluates agents through bounded interaction with a simulated user. The benchmark supports both openended and choice-based clarification, and measures slot recovery, stopping behavior, silent assumptions, and interaction cost. We further propose Interactive Optimization (InterOPT), a two-stage framework that identifies unresolved formulation-critical gaps and uses them to guide whether to ask the next question or to stop. In our choice-based experiments, InterOPT substantially outperforms all baselines in exact slot recovery; in the open-ended setting, it remains competitive with strong prior methods. Together, OR-Clarify and InterOPT reframe OR assistance as a selective completeness decision: clarify when needed, stop when ready, and quantify what remains missing.