DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems

AI summary

Overview

Research area: Evaluation of LLM systems — specifically, benchmarks for "decision models," the small or specialized models that sit inside LLM pipelines and make discrete choices (route, allow, verify, accept).

Technical level: Intermediate. Readers should be comfortable with accuracy, calibration, and selective prediction, but the paper explains its metrics from first principles.

Scope: The paper introduces DecisionPivot, a two-track diagnostic benchmark of 48 tasks and 40,033 items across seven decision roles, and reports an evaluation of 40 decision models plus 14 general models.

What This Paper Is About

LLM systems constantly make binary or small-option decisions: which model should handle a request, whether a tool call is allowed, whether an output is correct, whether an agent has finished its task. The models that make these calls are typically scored by item accuracy, which hides whether they can tell near-identical cases apart and whether their confidence is trustworthy. DecisionPivot builds matched sets of cases that differ only in a decisive detail, with answers computed and independently re-verified, so failures can be traced to specific kinds of reasoning rather than averaged away.

Key Contributions

1. A controlled benchmark with independently verified answers. 13 generated task families spanning seven decision roles, organized into contrast groups where changing a "pivotal detail" flips the correct answer while surrounding context is preserved. Every answer is computed by a generator and re-derived by a solver written independently by an author who had not read the generator.

2. A complementary public reference track. 32,233 items drawn from 35 public test sets, reconstructed from official source files and scored with their original metrics, so the new diagnostic measures can be read alongside established comparisons.

3. A unified reliability framework. Metrics are combined across four levels: item accuracy, pair accuracy (every member of a contrast group must be correct), probability ordering (does the positive member of a binary pair get a higher probability of "yes"), and confidence quality (calibration, accuracy at 0.90 confidence, and retrospective coverage at a 5% empirical error rate).

4. Empirical diagnosis of 40 decision models and 14 general models, including analyses by difficulty level, reasoning setting, error asymmetry, and input presentation, plus stability checks on 4,200 fresh items from seven generators.

Main Findings

• Aggregate scores hide large gaps in controlled performance. Two 27B models separated by only 2.6 points on the reference track — Perplexity Decider (reference score 57.0) and JEV-27B (54.4) — reach contrast-group accuracies of 70.1% and 44.2%, a gap of nearly 26 percentage points.

• Correct probability ordering is not the same as correct decisions. Over 2,400 binary contrast pairs, Perplexity Decider ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%. Clef 27B shows the pattern more starkly: 78.6% pair direction versus 25.0% pair accuracy. Applying a single global threshold recovers only part of the gap (74.2% and 30.6%).

• Complete contrast groups remain hard. Among one-pass decision models, the top score is SL-27B Instant at 75.4% pair accuracy, followed by Decision 3.0 d3 at 71.3% and Perplexity Decider at 70.1%. Of the 40 decision models, 19 fall below the 16.1% uniform-guessing pair-accuracy baseline overall, and 30 fall below it at difficulty level 3.

• Similar accuracy, different confidence reliability. Kev 27B and JEV-27B have nearly identical item accuracy (70.6% and 70.3%), but among predictions at confidence 0.90 or above, Kev reaches 93.8% accuracy on 34.1% of items while JEV-27B reaches 88.9% on 47.4% — the latter is confident more often, and wrong more often when it is.

• Selective coverage separates models that accuracy does not. At a 5% empirical error target, Kev 27B and JEV-27B cover 35.2% and 34.8% of items, while Perplexity Decider reaches 72.3% and Instant 79.6%, with non-overlapping intervals.

• Reasoning effort helps on the sampled subset but not uniformly on aggregate. On a 1,296-item subset, GLM-5.3 reaches 93.5% pair accuracy at its higher reasoning setting versus 81.5% at its lowest, with average reasoning length rising from 121 to 265 tokens per item. Yet across full tracks, AutoThink scores 86.9% core pair accuracy against 75.4% for Instant, while their reference scores are essentially level (58.9 versus 59.4).

• Errors are asymmetric across models. On the 12 annotated families, Kev 27B scores 56.5% on cases marked "no planted problem" versus 77.5% on cases with a planted problem; Clef 27B scores 32.7% versus 76.7%. Perplexity Decider is balanced at 80.9% in both categories.

• Specific arithmetic and temporal contrasts collapse. The four displayed one-pass models reach at most 5% pair accuracy on total-price and minimum-discount contrasts (AutoThink: 57% and 76%), and all four score 0% on current-time contrasts in triage_order (AutoThink: 85%).

• Performance degrades with difficulty. From level 1 to level 3, Perplexity Decider drops from 87.7% to 57.4% pair accuracy, Instant from 90.7% to 65.3%, and AutoThink from 89.2% to 82.2%.

• Stable aggregate scores can mask unstable predictions. A second AutoThink run scores 87.2% versus the original 86.9% while changing 5.6% of individual answers. Under an option-order variant, Kev 27B changes 4.9% of predictions, with 2.0% correct-to-incorrect and 1.5% incorrect-to-correct, netting only a 0.5-point accuracy decline.

• Shortcut auditing matters. A content-blind rule reaches 73.0% on llmbar against a 50% chance baseline, and a schema-only heuristic reaches 91.7% item accuracy on toolcall_valid, showing that surface cues can inflate scores even in carefully built tasks.

Methodology in Plain English

The benchmark is split into two halves with different jobs.

The core track is generated. For each task family, a Python program samples a structured case, computes the answer, then renders the case as text using invented names and varied wording. Crucially, cases are produced in matched sets called contrast groups: the same situation is rendered twice, with one decisive fact changed — a shipping fee moved by two cents so a total crosses a price limit, or two steps reordered so a file is deleted before it is copied. Because the answer is computed rather than annotated, correctness is known by construction, and a second solver written independently by an author who never saw the generator confirms all 600 answers per family. Each family has 600 items balanced across three difficulty levels.

The reference track takes the opposite approach: 35 existing public test sets are converted into the same decision interface but scored with their original metrics, and 3,016 items (8.5%) detected in the training files of the authors' own 27B models were removed.

Scoring goes beyond counting right answers. Pair accuracy requires every member of a contrast group to be correct — predicting "yes" for both members of a pair earns 50% item accuracy but zero pair accuracy. Pair direction checks only whether the positive case gets a higher probability of "yes" than the negative case, separating ranking from thresholding. Confidence metrics sort items by the model's top probability and ask how much of the ranking can be trusted at a 5% error rate, plus accuracy among items at 0.90 confidence and calibration error over 15 bins. Models are run through their own documented interfaces, and failed or missing predictions count as wrong rather than being dropped.

Why This Matters

Decision models are the load-bearing component of LLM systems, and the paper shows that the metric most teams use — item accuracy on a public test set — is a poor proxy for whether a model can distinguish cases that differ by one decisive fact, or whether its confidence can be trusted to gate automation.

Real-world applications:

• Model routing: choosing which model handles a request, where a router that scores well on average may still fail both members of a near-identical pair of prompts.
• Guardrails and policy enforcement: allowing or denying tool calls and actions, where false rejections and false acceptances carry asymmetric costs — the paper's error-asymmetry analysis shows models differ sharply here.
• Output verification: judging whether a generated answer or claimed event count is correct before it reaches a user.
• Agent loop control: deciding whether a tool call is valid and whether a task's goal state has actually been reached, where current-time and ordering contrasts proved especially brittle.

Industry relevance: the benchmark is designed for procurement and regression testing. Pair accuracy, coverage at a fixed error rate, and confidence-conditional accuracy give teams numbers they can map onto an operating point, and the finding that reference scores and controlled-contrast scores can diverge by tens of points is a direct argument against single-number leaderboards.

Future Directions

• Extending beyond written, English-language cases to end-to-end system operation, multilingual inputs, and inference efficiency, all of which the authors list as outside the current scope.
• Deployment-grade calibration. Empirical selective coverage in the paper is retrospective and computed per family; turning it into a single threshold for future requests would require held-out calibration, which the paper explicitly does not establish.
• Independent, third-party evaluation. The authors note that DecisionPivot and the top-scoring StartLux-Decision models come from the same team, making independent replication a stated need.
• Stronger contamination and shortcut controls. Item-level training-overlap detection is only possible for the authors' own models, and the paper concedes that partial-information heuristics (91.7% on toolcall_valid) can still succeed; fresh-sample checks so far test stability within existing generator designs rather than generalization to independently designed tasks.

Target Audience

Researchers and engineers working on LLM evaluation, model routing, guardrails, and agent infrastructure benefit most, particularly those choosing among small decision models or building regression suites for production pipelines. The paper is also useful for benchmark designers interested in contrast-group methodology and independent answer verification, and for anyone who needs to argue that calibration and selective prediction should accompany accuracy in model selection.

Authors’ abstract

Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles. Its core track contains 7,800 items from 13 generated task families, organized into contrast groups with computed answers verified by independently implemented solvers. The complementary reference track contains 32,233 items from 35 public test sets, preserving their original evaluation metrics. We evaluate 40 decision models and 14 general models using documented, model-specific protocols, assessing item and contrast-group accuracy, probability ordering, calibration, and empirical selective coverage. Our results reveal substantial differences obscured by aggregate scores: two 27B models separated by only 2.6 points on the reference track achieve contrast-group accuracies of 70.1% and 44.2%. Across 2,400 binary contrast pairs, Perplexity Decider correctly ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%, exposing a gap between probability ordering and decision correctness. Models with similar item accuracy also differ in the reliability of their high-confidence predictions. On a separate 1,296-item subset, GLM-5.3 achieves 93.5\% contrast-group accuracy at a higher reasoning setting, compared with 81.5% at its lowest effort. By linking errors to pivotal details and difficulty levels, DecisionPivot provides fine-grained diagnostics of decision failures that aggregate evaluation alone cannot reveal.