GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI

AI summary

Overview

• Research area: Agentic AI / LLM-based automated planning and inference-time reasoning.
• Technical level: Advanced (assumes familiarity with LLM prompting paradigms, Chain-of-Thought, ReAct, Tree-of-Thoughts, and multi-agent orchestration).
• Scope: The paper introduces GRASP, a three-module framework that decomposes LLM plan generation into context-isolated generation, revision, and verification stages, and evaluates it on four benchmarks plus a multi-task scaling study.

What This Paper Is About

LLM-based agents become less reliable as tasks get more complex — a problem the authors call the "Curse of Instructions," where constraint collisions and attention fatigue cause hallucinations and multi-step execution failures. The paper's goal is to generate high-quality, natural-language executable plans for complex tasks by splitting the planning pipeline into specialized modules that never share a context window, so that errors in one stage cannot contaminate another.

Key Contributions

1. GRASP framework: An autonomous, multi-stage planning framework that enforces global macro-regularization and context-isolated strategic refinement to eliminate compounding error feedback loops in LLM planning.
2. Ablation showing modular interdependence: A comprehensive ablation study demonstrating that local search optimization collapses into unconstrained hallucination loops without global macro guardrails.
3. Empirical state-of-the-art frontier: Evaluations spanning programmatic execution and pure LLM execution showing GRASP sets a new frontier, including neutralizing multi-task degradation and beating frontier reasoning models (GPT-5-mini) by 14.5%.
4. Structural decoupling: Three modules — GenPlan (global constraints and guidelines), RevPlan (localized strategy exploration in isolated context windows), and VerPlan (independent multi-criteria scoring) — that together form the pipeline.

Main Findings

• Natural Plan Calendar Scheduling: GRASP outperforms LLM planners by ~12.4% for GPT-4o and by 20% for GPT-4o-mini. GRASP(GPT-4o-mini) outperforms the GPT-4o planner by 11.3%. Exact-match accuracies: GPT-4o-mini baseline 41.9, GPT-4o-mini Planner 54.3, GRASP(GPT-4o-mini) 74.3; GPT-4o baseline 58.3, GPT-4o Planner 63, GRASP(GPT-4o) 75.4.
• ZebraLogic: GRASP(GPT-4o) reaches 61.4% versus GPT-4o baseline 29.5% and GPT-4o Planner 30.6% — a 30.8% improvement over the planner.
• SciBench Math: GRASP(GPT-4o) outperforms the GPT-4o planner on all subsets (Stat 80.56 vs 70.83; Calc 78.05 vs 73.17; Diff 62 vs 54). GRASP(GPT-4o-mini) beats the GPT-4o-mini planner on Stat (81.94 vs 70.83) and Diff (50 vs 28) but ties on Calc (73.17).
• GPQA (negative result): GPQA results are averaged over 3 runs and GRASP does not outperform either the GPT-4o baseline (47.99) or planner in a statistically significant way. GRASP (GPT-4o, 2 strat.) scores 47.54; GRASP (GPT-4o, 4 strat.) scores 48.23.
• Against state-of-the-art planners: GRASP (74.3 Calendar Scheduling, 81.94 SciBench Stat, 73.17 SciBench Calc, 50.0 SciBench Diff, 58.0 ZebraLogic) outperforms all PlanGEN variants, Best-of-N, and Tree-of-Thoughts on Calendar Scheduling, SciBench Stat, and ZebraLogic. PlanGEN equals GRASP on SciBench Calc and Diff.
• Multi-task scaling — degradation flattened: All planners achieve parity on isolated single tasks (hovering between 44.5% and 45.8%). Under interleaved dual tasks, GPT-4o planner scores 30.5 and GPT-4o-mini planner 24.5; on triple tasks these decay to 29.1 and 21.2. GRASP(GPT-4o) achieves 45.7 on dual tasks and rises to 46.6 on triple tasks; GRASP(GPT-4o-mini) reaches 41.2 on dual tasks, a 16.7% absolute gain over its direct baseline. In dual-task environments, GRASP achieves an absolute accuracy gain of up to 16.7% over direct LLM planners.
• Beats frontier reasoning models: The direct GPT-5-mini planner scores only 26.7% on dual tasks and drops to 25.5% on triple tasks. GRASP(GPT-4o-mini) achieves 41.2% on dual tasks, a 14.5% absolute margin over the direct GPT-5-mini planner. GRASP also exceeds its direct planner counterpart by 17.5%.
• Ablation — modules need each other: Direct Planner 30.5; RevPlan Only 10.9; VerPlan Only 30.7; GenPlan Only 40.8; RevPlan + VerPlan 12.5; GenPlan + RevPlan 42.5; full GRASP 45.7 (dual-task mode). Removing global guardrails while running local revision causes a "garbage-in, garbage-out" collapse to 12.5%.
• Strategy count matters: Varying RevPlan strategies from 1 to 4 yields 73.0%, 74.1%, 75.4%, and 71.2% — a sweet spot at three strategies.
• Guidelines and constraints are load-bearing: Using 3 strategies while removing constraint and guideline generation in GenPlan drops accuracy from 75.4% to 58.5%.
• Cost profile: GRASP (GPT-4o) increases cost by ~13.5x versus the baseline GPT-4o model while improving accuracy by 12.4% over the GPT-4o planner. GRASP (GPT-4o-mini) with three strategies achieves an 11.3% accuracy gain at ~0.95x the cost. Token usage is reported as competitive with PlanGEN(Mix) and PlanGEN(BoN); PlanGEN(BoN) consumed 22% extra tokens on Calendar Scheduling while performing worse.
• Capacity boundary for smaller models: GRASP(GPT-4o-mini) performs robustly on dual tasks (41.2%) but collapses to 28.1% on triple tasks, suggesting a hardware-bound capability threshold.

Methodology in Plain English

GRASP splits planning into three sequential modules, each running in its own separate context window so the model never has to juggle global rules, runtime variables, and step generation at once.

GenPlan builds a structural blueprint from the task description alone, ignoring the specific instance. It maintains a shared knowledge base — a tuple of the task, hard constraints, soft guidelines, and a tentative plan — and iterates with three prompt-engineered agents: a Constraint Agent, a Guidelines Agent, and a Plan Generation Agent. Constraints act as hard regularizers that prune infeasible states; guidelines act as soft regularizers that minimize generative drift. The loop repeats until a judge block says the knowledge base has converged.

RevPlan specializes the blueprint for a specific task instance. Rather than one linear revision pass, it induces a set of strategies (typically 2 to 4) and runs each in its own isolated track. Each track uses ReAct to produce a solution trajectory, extracts the executable steps from it, then merges that instance-specific plan back with the GenPlan blueprint. Each track also gets its own localized constraints as guardrails.

VerPlan acts as an independent evaluator. It scores each candidate plan on a 0–100 scale against the task, instance, GenPlan constraints, strategy-specific constraints, and internal rubrics, then selects the highest-scoring plan as the final output.

For evaluation, all generated plans were executed through RunAgent, a multi-agent execution platform, using frontier LLMs. RunAgent (GPT-4o) was used for Natural Plan Calendar Scheduling, SciBench Math, and GPQA; RunAgent (GPT-4.1-mini) for ZebraLogic. The authors ran a bare-bones version of RunAgent using only LLM or Python code execution, with error correction invoked only for runtime errors, so poor plan quality would not be masked by verification protocols. Equivalence between outputs and gold answers was judged by an LLM judge running GPT-5; manual evaluation of 50 random instances per dataset classified by GPT-5 showed full agreement with human judgment.

Why This Matters

The paper's central claim is that explicit structural guardrails let ordinary LLMs beat the unguided internal reasoning chains of frontier reasoning models — a notable result because it suggests planning architecture, not raw model scale, is the binding constraint on complex agentic tasks. It also shows that the multi-task performance collapse suffered by standard planners can be completely flattened rather than merely mitigated.

Real-world applications:

• Administrative logistics and scheduling — the Natural Plan Calendar Scheduling benchmark directly models calendar coordination workflows.
• Enterprise workflow automation — multi-step business processes where constraint violations are costly and compound across steps.
• Scientific and technical reasoning assistance — the SciBench Math evaluation targets expert-level quantitative problem solving.
• Agent orchestration platforms — the RunAgent integration demonstrates GRASP plans being executed as tool-use workflows, relevant to anyone building multi-agent systems.

Industry relevance: The framework is fully autonomous and designed for zero-shot plan generation, meaning constraints can be auto-generated from the task description even with no user-defined priors. The cost analysis matters for deployment: the GPT-4o-mini configuration achieves an 11.3% accuracy gain at roughly 0.95x cost, while the GPT-4o configuration costs ~13.5x more for a 12.4% gain over the planner — a trade-off enterprises will need to weigh. The authors also note GRASP's token usage is competitive with other baselines such as PlanGEN(Mix) and PlanGEN(BoN), addressing the environmental and computational overhead concerns that plague verbose multi-agent self-reflection loops.

Future Directions

1. Human-in-the-loop interaction: Adding an interactive chat interface that automatically formulates clarification queries when task specifications are ambiguous or incomplete, and that can solicit high-level plan-construction guidance such as decomposition strategies or expert heuristics.
2. Runtime plan modification: Examining each step's output, verifying correctness and relevance of intermediate results, and feeding verification feedback back to the LLM to re-execute steps — or insert and revise steps when needed, potentially using task-specific rubrics obtained from a human.
3. Automated strategy tuning: The optimal number of strategy-based plans is currently a task-dependent hyperparameter requiring manual tuning (peaking at three in Calendar Scheduling); the authors propose automated, dynamically adjusting strategy counts.
4. Reducing latency: The iterative generation, refinement, and verification inherently incurs higher runtime latency than single-pass generation, limiting suitability for real-time interactive applications. The authors suggest exploring parallelized generation pipelines or early-exit optimization heuristics.

An open question the paper raises directly: GRASP cannot bridge a genuine cognitive deficit in the backbone model. It did not outperform baselines on GPQA, SciBench Diff, or SciBench Calc — datasets that rely not just on building a good plan but on improving the underlying LLM's reasoning capability itself.

Target Audience

Researchers and practitioners working on LLM agents, automated planning, and inference-time reasoning architectures will benefit most. It is particularly relevant to engineers building production multi-agent orchestration systems who need to understand the trade-offs between planning accuracy, token cost, and latency, and to those evaluating whether frontier reasoning models or structured planning frameworks are the better investment for complex multi-step tasks. Readers seeking a gentle introduction to agentic planning will find the formal notation (knowledge-base tuples, module equations, and algorithm listings) demanding, as the full algorithms and prompts are deferred to appendices.

Authors’ abstract

Large Language Models (LLMs) typically exhibit a performance profile where reliability degrades as task complexity increases. We address the challenge of generating high-quality natural language executable plans for complex tasks by introducing $\textbf{GRASP}$, a strategy-aware, multi-stage planning framework. GRASP decouples the planning pipeline across specialized, context-isolated modules: it pre-compiles global macro-guidelines (GenPlan), explores alternative localized strategies within isolated context windows (RevPlan), and independently evaluates trajectories using a multi-criteria discriminator (VerPlan). Empirical evaluations show that GRASP consistently establishes a new state-of-the-art frontier across diverse datasets, yielding substantial accuracy gains over direct LLM planners on Natural Plan Calendar Scheduling ($\sim$12.4$\%$$\uparrow$), ZebraLogic ($\sim$30.8$\%$$\uparrow$), and SciBench Math. Crucially, under multi-task scaling-where standard planners suffer immediate performance collapse-GRASP completely flattens the multi-task degradation penalty. In interleaved dual-task environments, GRASP achieves an absolute accuracy gain of up to 16.7$\%$ over direct LLM planners. Furthermore, by isolating context and enforcing strict macro-regularization, GRASP outperforms frontier reasoning models (such as GPT-5-mini) by a margin of 14.5$\%$.