Programming over Thinking: Efficient and Robust Multi-Constraint Planning

AI summary

Overview

Research area: Artificial Intelligence — LLM-based automated planning, multi-constraint reasoning, and code-generation-as-reasoning approaches.

Technical level: Intermediate. The paper is readable without deep specialization, but familiarity with LLM prompting strategies (chain-of-thought, tree search) and planning benchmarks helps.

Scope: This paper introduces SCOPE (Scalable COde Planning Engine), a multi-agent framework that separates query-specific reasoning from reusable code execution to solve multi-constraint planning tasks more accurately, cheaply, and reliably than text-based or per-query coding approaches.

What This Paper Is About

Multi-constraint planning asks a system to find plans that satisfy several conditions at once — conditions that may conflict — such as arranging a trip that meets budget, scheduling, and travel rules simultaneously. Current LLM approaches handle this poorly: pure natural-language reasoning accumulates errors and grows expensive as constraints multiply, while code- or solver-based methods either write problem-specific code from scratch for every query or depend on fixed solvers that do not generalize. The paper's goal is a framework that keeps reasoning flexible per query but pushes the heavy combinatorial work into reusable, deterministic code.

Key Contributions

1. A reusable abstraction that separates query-specific reasoning from execution logic. The framework identifies the key parameters that generate or filter candidate plans, so the same solver functions can handle different queries with only minimal changes to input parameters.

2. An autonomous multi-agent pipeline that induces this abstraction from a single example. The pipeline identifies relevant combinations and constraints and produces solver functions that operate on parameter values rather than query-specific content, allowing the abstraction to be applied broadly without manual prompt engineering.

3. A parameter-free refinement mechanism. Using one example query and answer for output reflection, the system automatically adjusts prompts and regenerates functions until expected outputs are produced, creating a fully automated pipeline requiring no additional training or manual intervention.

4. Empirical evidence across five models and two benchmark families. The authors report state-of-the-art results with lower cost and latency than six baselines, including improvements in robustness and scalability as problem complexity rises.

Main Findings

• Large gains over the best text-based baseline. With GPT-4o, SCOPE reaches 93.1% success on TravelPlanner, a 61.6% gain over the best baseline (CoT), while cutting inference cost by 1.4x and time by 4.67x.

• Improvements are largest on weaker models. For GPT-4o, Trip Planning success rises from 12.5% (ToS) to 87.1%, Meeting Planning from 59.8% (ToS) to 100%, and TravelPlanner from 31.5% (CoT) to 93.1%.

• Smaller models with SCOPE can match or beat larger models with baselines. In Trip Planning, GPT-4o with SCOPE (87.1%) outperforms the best baseline results for GPT-o3 (78.8%) and GPT-5 (84.6%), which the authors attribute to deterministic executable logic rather than model scale.

• Full results across five models. SCOPE achieves TravelPlanner micro/macro success of 97.8/97.6 (GPT-4o), 97.5/97.2 (GPT-o3), 98.3/97.8 (GPT-5), 85.6/89.3 (Gemini-1.5-Pro), and 98.2/98.0 (Gemini-2.5-Pro). Hard constraint success rates are 93.1, 92.2, 93.9, 71.6, and 94.2 respectively.

• Robustness under increasing complexity. SCOPE maintains a high success rate (often above 90%) across all complexity levels and datasets, with only minimal decline at the most extreme levels, while baselines — especially those on weaker foundational models (GPT-4o, Gemini-1.5-Pro) — degrade rapidly.

• Cost scales better than multi-step reasoning. Baselines such as ToT, EvoAgent, and HTP incur exponentially growing API costs from multiple text generations; SCOPE limits textual output and does plan enumeration at the code level.

• Every component matters. Ablation on GPT-4o: removing Problem Formalization drops TravelPlanner to 25.2 and Trip to 13.3; removing Problem Optimization drops to 53.6 and 0.0; removing Solver Refinement drops to 93.1 and 0.0. Full SCOPE scores 93.1, 87.1, and 100 on TravelPlanner, Trip, and Meeting respectively.

• Failures trace back to the Input Agent. Smaller models like Gemini-1.5-Pro may assign parameter values incorrectly; one-shot demonstration can cause overgeneralization of patterns from examples; and tasks requiring large structured outputs (Meeting Planning) can introduce subtle errors such as incorrect numerical values.

Methodology in Plain English

SCOPE runs in two stages before any test query is answered.

Stage one — query-specific problem reasoning. Given a single example query and its answer, a Planning Agent converts the query into a structured representation with two parts: combinations, which list the parameters needed to enumerate every candidate plan exhaustively, and constraints, which define the conditions a valid plan must satisfy. In parallel, a Solution Agent converts the example answer into a structured representation that fixes the expected output format. Optimization Agents then iteratively clean up these representations — removing redundancy, resolving ambiguity, and making sure combinations fully cover the plan space while constraints remain clear and non-conflicting.

Stage two — generic solver generation. Three agents each build one reusable function. The Combination Function Generator produces a function that takes parameter values and enumerates candidate plans in structured form. The Filter Function Generator produces a function that checks each candidate against the constraint values and returns only valid plans. The Deliver Function Generator produces a function that turns a structured plan into a natural-language answer. If a function fails — the candidate list misses the known solution, the filter returns the wrong plan, or the final answer mismatches the ground truth — the agent regenerates it using the same example as supervision, looping until the expected output appears.

Inference. For each test query, a single Input Agent is given the example query and its optimized combinations and constraints as a one-shot prompt, plus the new query, and outputs combinations and constraints for that query. No code is generated at inference. The pre-built Combination, Filter, and Deliver Functions then run in sequence to produce the final answer. The authors argue this does not overfit because the Function Generator Agent is instructed to design functions that operate on input parameters rather than hardcoding instance values, and the Planning Agent captures general domain structure rather than the example's specific keys or values.

Evaluation setup. The authors test on TravelPlanner (Xie et al., 2024) and Natural Plan (Zheng et al., 2024), focusing on the Trip Planning and Meeting Planning subsets of the latter. They compare against six baselines — Direct prompting, CoT, ToT, EvoAgent, HTP, ToS, and CPMPy — across five models: GPT-5, GPT-o3, GPT-4o, Gemini-2.5-Pro, and Gemini-1.5-Pro. All baselines receive few-shot examples for fairness, all code is written by the model itself with no human-written snippets, and all experiments run in a closed environment with deterministic constraints. Because of limited budget, the authors run half the queries on Trip Planning with each level of complexity equally sampled, except for Direct, CoT, and SCOPE, where they run the full dataset with separate reported results.

Why This Matters

The paper argues that pushing combinatorial search into reusable deterministic code, while keeping only parameter estimation in the LLM, is a more scalable pattern for planning than long natural-language reasoning chains. It offers a concrete comparison point for a field where many recent methods still require human-written hints or template code — approaches the authors deliberately exclude from their comparison to keep attribution to the model's own capabilities.

Real-world applications:

• Travel itinerary planning, where budget, dates, transport, and accommodation rules interact — the paper's primary benchmark domain.
• Meeting scheduling, where participants, times, and location constraints must all be satisfied simultaneously.
• Trip planning and routing, the Natural Plan task SCOPE is evaluated on, involving ordering decisions under constraints.
• General constraint satisfaction pipelines where the same task family is queried repeatedly, making one-time solver construction cost-effective.

Industry relevance: The reported 1.4x cost reduction and 4.67x time reduction for GPT-4o matters for deploying planning agents at scale, where per-query reasoning cost dominates. The finding that smaller models equipped with SCOPE can rival larger models on baselines suggests a route to cheaper production systems. Code and resources are released at https://github.com/DerrickGXD/SCOPE.

Future Directions

• Cross-domain solver generalization. The authors state that solver functions are still tied to a specific domain and cannot be applied to a new domain without redefining the problem structure.
• Automatic domain drift detection and adaptive solver regeneration. SCOPE currently does not explicitly handle queries whose constraints cannot be expressed within the existing solver's structural specification.
• Extension to open-source models. The authors note that their evaluation focuses on closed-source models because the framework relies on LLM coding and formatted generation capabilities; testing open models is left open.
• Multi-example solver construction. The authors observe that a single example suffices in their experiments because it contains full constraint coverage, and show in an appendix analysis that multiple examples with incomplete constraint coverage can improve robustness through union constraint coverage — a direction they flag for applications needing additional assurance.

Target Audience

Researchers and practitioners working on LLM agents, automated planning, and constraint satisfaction; engineers building cost-sensitive planning pipelines in travel, scheduling, or logistics; and readers interested in code-generation-as-reasoning methods who want a concrete alternative to long-chain text reasoning. Readers without background in prompt-based reasoning or planning benchmarks will find the core idea accessible, but the benchmark tables and ablation details assume some familiarity with LLM evaluation practice.

Authors’ abstract

Multi-constraint planning involves identifying, evaluating, and refining candidate plans while satisfying multiple, potentially conflicting constraints. Existing large language model (LLM) approaches face fundamental limitations in this domain. Pure reasoning paradigms, which rely on long natural language chains, are prone to inconsistency, error accumulation, and prohibitive cost as constraints compound. Conversely, LLMs combined with coding- or solver-based strategies lack flexibility: they often generate problem-specific code from scratch or depend on fixed solvers, failing to capture generalizable logic across diverse problems. To address these challenges, we introduce the Scalable COde Planning Engine (SCOPE), a framework that disentangles query-specific reasoning from generic code execution. By separating reasoning from execution, SCOPE produces solver functions that are consistent, deterministic, and reusable across queries while requiring only minimal changes to input parameters. SCOPE achieves state-of-the-art performance while lowering cost and latency. For example, with GPT-4o, it reaches 93.1% success on TravelPlanner, a 61.6% gain over the best baseline (CoT) while cutting inference cost by 1.4x and time by ~4.67x. Code is available at https://github.com/DerrickGXD/SCOPE.