MA-Bench: Measuring Operational Planning in Language Models

AI summary

Overview

Research area: Evaluation of large language models on open-ended, high-stakes operational planning — specifically mission analysis, the first step of the military Joint Planning Process.

Technical level: Intermediate. The benchmark methodology is described plainly, but the paper assumes familiarity with rubric-based LLM evaluation, judge calibration metrics (Cohen's and Fleiss' kappa, Spearman correlation, F1), and bootstrap intervals.

Scope: MA-Bench measures whether a language model can turn a large, incomplete and partly contradictory packet of planning documents into the determinations a competent planning staff would make, grading the brief against 1,257 expert-authored atomic requirements rather than a reference answer.

What This Paper Is About

Most agent benchmarks score a task with an externally checkable outcome — a resolved software issue, a navigated website, a satisfied set of constraints. Operational planning has no such outcome, and a mission analysis brief can be fluent, plausible and internally coherent while silently omitting a critical determination. The paper introduces MA-Bench, ten expert-built scenarios in which a model receives a synthetic staff packet of ten to twelve documents and must produce a mission analysis brief, graded item by item against an expert rubric covering extraction, synthesis and inference, and separating omissions from detrimental moves. Twenty frontier and open-weights models are evaluated under one fixed protocol.

Key Contributions

1. A formulation of operational planning evaluation for open-ended, high-stakes tasks with heterogeneous and partly conflicting evidence and no unique reference answer, grounded in the Joint Planning Process (JP 5-0).

2. MA-Bench itself: ten expert-validated scenarios with long multi-document planning packets (36k to 243k characters; packets of 10 to 12 documents) and 1,257 expert-authored atomic evaluation items, with a structured methodology separating extraction (Tier 1), synthesis (Tier 2) and inference (Tier 3) while explicitly scoring both omissions and detrimental actions.

3. An empirical evaluation of twenty frontier and open-weights models under a single protocol, decomposing performance by grounding tier, mission analysis step, severity, and failure mode.

4. A calibrated judge and full release: the rubric-grading judge is checked against four credentialed planners, reproduced by judges from two other providers, and every packet, rubric, prompt, brief and item-level verdict is released.

Main Findings

• The best model gets 62% of rubric items right. Muse Spark 1.3 leads the benchmark condition with a weighted score of 0.574, followed by Grok 4.6 at 0.540, Claude Opus 5 at 0.514, and GLM-5.3 — the best open-weights model — at 0.505. The sixteen models run in the benchmark condition span 0.30 to 0.57.

• No model approaches the ceiling, and adjacent models are not separated. Bootstrap intervals over the ten scenarios are about ±0.08 wide, so only the ends of the table separate cleanly.

• Performance falls sharply as reasoning requirements rise. Positive items pass at 58.1% when stated in a single document (Tier 1), 49.0% when assembled from several (Tier 2), and 33.4% when inferred (Tier 3). Tier 3 items make up 36% of the rubric, so the planning judgments the benchmark was built to measure account for the largest share of missing credit.

• Models make the headline determinations and miss the second-order ones. Critically important items pass at 56.9% and important items at 37.5%.

• The failure profile is omission, not error. Of 31,882 failed positive items, a keyword classification assigns 60.2% to outright omission, 8.2% to incomplete treatment, 2.5% to content filed under the wrong step, and 1.6% to incorrect or unsupported content, with 28% unassigned. The proportions barely move across models, tiers or severities, which is why a reference-similarity or accuracy metric would miss most of what the rubric catches.

• Weakness concentrates on forward-looking steps. Models are strongest on constraints and restraints (72%) and higher mission and intent (66%), and weakest on commander's critical information requirements (31%), decision points and named areas of interest (38%), and force and resource analysis (41%). The ordering holds on the 285 briefs that never reached the word cap.

• Coverage and commission are separable qualities. Every brief commits three to five flagged wrong moves, about two of them critical. GPT-6 Astra commits the fewest (3.1 per brief, 1.4 critical) while ranking fifth on coverage. Flagged mistakes are committed on 25.9% of detrimental and 28.3% of critically detrimental opportunities, more often on plainly stated traps (35% of Tier 1 opportunities against 21% of Tier 3).

• Length is not a proxy for score. Within the capped condition, the rank correlation between raw brief length and score is 0.26 (Pearson 0.15) across the twenty systems, and the longest writer, Gemini 3.8 Flash, ranks fourteenth. Removing the cap raises scores (GPT-6 Astra by 0.078, GPT-5.4 by 0.123), showing models hold more rubric-relevant content than a staff-length brief can carry.

• Neither more reasoning compute nor a bigger model closes the gap alone. Moving reasoning effort from low to high shifts a model's score by at most 0.08 across five models with no consistent sign, even as reasoning tokens rise by one to two orders of magnitude. Within families the stronger member wins in three of four cases: Claude Opus 5 over Sonnet 5 by 0.12, GPT-6 Astra over GPT-5.4 by 0.115, DeepSeek-V4-Pro over V4-Flash by 0.05, with the Gemini pair tied.

• The judge grades like a fifth credentialed planner. Four JPME-credentialed planners independently graded 190 item verdicts; they agree with one another at Fleiss κ = 0.42. The production judge agrees with their majority on 78.0% of the 164 items with a majority (κ = 0.54, F1 = 0.82 on the fail class), and treated as a fifth rater leaves the panel's Fleiss κ effectively unchanged (0.42 to 0.41). Agreement is flat across tiers (κ 0.53, 0.56 and 0.51), so the judge is no less reliable on inference items.

• Results do not depend on one grader. Three judges from three providers agree on 80.0% to 85.1% of verdicts (κ 0.61 to 0.70), all above expert–expert agreement, with model rankings stable (Spearman 0.89 to 0.94). Gemini 3.1 Pro regrades all briefs and agrees with the production judge on 84.0% of 131,985 item verdicts (κ = 0.678, Spearman 0.94 between the sixteen benchmark-condition model means). No judge favors its own model family by more than 0.03.

• A content filter is a result, not an exclusion. Claude Fable 5.1's wide interval reflects refusals: its content filter terminated every brief on Jagged Anvil and four of five on Pacific Harrier, mid-sentence and without a refusal message. Those outputs are graded as delivered. Without those nine briefs it scores 0.510 (0.432, 0.595) over the nine scenarios that keep a brief.

Methodology in Plain English

Each scenario is a synthetic, expert-validated operational crisis with real geography and real actors, written in the format of an actual staff packet: strategic context, intelligence summary and preparation brief, orders of battle, country study and geography, the initiating order, legal annexes, the commander's initial planning guidance and running staff estimates. Withheld documents — downstream products of a completed mission analysis — never reach the system under test, and exclusion is enforced by construction and checked against content hashes before every run.

Grading does not compare a brief to a reference text. Instead, each scenario has a frozen expert-authored rubric of 74 to 170 items (1,257 in total). Every item is atomic, binary, observable in the submitted brief, cited to doctrine or packet evidence, self-contained and non-duplicative. Items carry three labels: a grounding tier (Tier 1 stated outright in one document, Tier 2 requiring synthesis, Tier 3 requiring inference), a signed severity weight (critically important +9, important +5, slightly important +2, detrimental −5, critically detrimental −9), and one of twelve mission analysis steps. Negative items name a specific wrong move and are penalized when that move appears. Ordering, formatting and style earn nothing.

A single reasoning model (gpt-5.6 at high reasoning effort) grades each brief in one call against the full rubric, returning only the failed items with quoted evidence and rationale, in a schema-enforced format; unlisted items pass. The judge never sees the original packet, so it measures the brief against the rubric rather than its own reading of the scenario. Candidate judges were admitted only if their chance-corrected agreement with the expert majority was at least the experts' mean agreement with one another and their F1 on the fail class reached 0.80; the judge was frozen before any benchmark brief was graded.

Scoring is the unweighted fraction of items passed, plus a severity-weighted score in which positive items contribute their weight when satisfied and negative items subtract their absolute weight when present, divided by the total positive weight. A perfect brief scores 1; a brief meeting no requirement and making no listed wrong move scores 0. The score is deliberately not clamped at zero. Every model received the same prompt naming the twelve mission analysis steps followed by the full packet, a 3,000-word cap enforced before grading, medium reasoning effort where set and the provider default otherwise, and at least three independent briefs per scenario (five for the three models run first). The benchmark comprises 1,020 graded briefs, with 540 uncapped briefs extending the length ablation to every model. The scenario is the unit of analysis.

Why This Matters

Impact on research: The paper makes the case that a whole class of high-value tasks is invisible to accuracy-style and reference-similarity metrics, because there is no wrong sentence to detect — what is wrong is what is missing. It shows a way to build expert ground truth for tasks with no unique correct answer, and it separates extraction, synthesis and inference into measurable tiers. It also demonstrates a judge-design discipline that reporting practice rarely includes: selecting the grader by measured agreement with credentialed experts, then verifying that adding it as a fifth rater does not change the panel's agreement.

Real-world applications:

• Military staff planning and decision support. The paper notes the U.S. Department of Defense has moved from studying generative AI to piloting it across operational and administrative applications; MA-Bench gives a measured instrument for the task models are already being asked to perform.
• Any profession where an expert product must survive review against doctrine or regulation. Legal, medical and other credentialed domains share the structure of a defensible written analysis graded against requirements rather than a reference answer.
• Assurance and model procurement. Coverage and commission are separable qualities, so an organization can distinguish a system that misses second-order determinations from one that confidently states materially wrong ones before fielding it.
• Evaluation infrastructure for open-ended generative work. The tier and severity taxonomy is a reusable template for scoring long analytical artifacts where omissions and detrimental moves must be priced differently.

Industry relevance: The results bear directly on claims made about reasoning scaling. Going from low to high reasoning effort moved scores by at most 0.08 with no consistent sign while reasoning tokens rose by one to two orders of magnitude, and larger models within a family did not automatically close the gap. A word cap about the length of a real staff brief changed the ranking only modestly (Spearman 0.95), and uncapping raised scores, which suggests current systems hold more rubric-relevant content than a usable brief can carry. OpenAI-family, Google, Anthropic, xAI, Mistral, Qwen, and open-weights systems (GLM, DeepSeek, Kimi, MiniMax, Nemotron, Gemma) are all scored on the same standard, and the entire release — packets, rubrics, prompts, briefs and item-level verdicts — ships at huggingface.co/datasets/ScaleAI/MA-Bench.

Future Directions

• Build a human baseline. Planners graded model briefs in the expert study but wrote none of their own, so the paper has no human baseline and the scores are not calibrated against human-written briefs. A small set of human-written briefs would anchor the scale.
• Validate that a higher score means a better mission analysis. This requires a holistic expert rating of briefs against the packets, rather than item verdicts alone.
• Improve agreement on negative items. Agreement is lower for the judge (κ 0.21 to 0.37) and for the experts alike on detrimental and critically detrimental items, which make up 12% of the rubric and enter the score subtractively; thirteen items were reworded for polarity in a documented errata before benchmark grading.
• Extend beyond the current scope. The benchmark covers one step of one planning process in one doctrinal tradition, with ten scenarios whose bootstrap intervals are about ±0.08 wide, and the length cap truncates many briefs — the uncapped ablation is offered as the price of that constraint.

Target Audience

Researchers working on LLM evaluation and benchmark design for open-ended, non-verifiable tasks; practitioners building planning or decision-support systems where an artifact must be defensible against a doctrine or regulation; military and defense analysts evaluating generative AI for operational planning; and anyone interested in rubric-based grading, LLM-as-judge calibration, or expert-agreement methodology.

Authors’ abstract

How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss $κ$ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts.