System1-Arena: Which Decisions Can Be Left to System 1?

AI summary

Overview

Research area: LLM evaluation and benchmarking, specifically comparing a new class of "System-1" decision models (which return probability distributions over options in a single forward pass, without generating text) against generative LLMs and reasoning LLMs on a shared set of decision requests.

Technical level: Intermediate. Readers should be comfortable with LLM evaluation concepts such as accuracy, calibration, Brier scores, reasoning tokens, and agentic/closed-loop benchmarks.

Scope (one sentence): The paper introduces a 92,125-question, 105-task benchmark with 138 closed-loop environments and evaluates 30 models to map which discrete decisions can be reliably delegated to fast, non-generative System-1 models and which still require generative or reasoning LLMs.

What This Paper Is About

Many real applications ask LLMs to make discrete decisions — flagging phishing, deciding whether to escalate to a human, checking whether a task is complete, or picking the next action in an environment. A new class of "System-1" models (Jev, the Decision-2.0 family, Microsoft-Decision-1, and the OpenAI Decisions API) answers such requests by returning a probability distribution over the options in one forward pass, without emitting any text.

The paper's goal is to determine empirically which decisions these fast probability-returning models can handle well, and where generative or reasoning LLMs are still needed — measuring not just accuracy but also probability quality (calibration), latency, cost, and behavior inside closed-loop agentic environments where the world may keep evolving during inference.

Key Contributions

1. A shared decision interface and benchmark (System1-Arena): 92,125 typed questions across 105 tasks drawn from 268 public sources in 16 domains, organized into four groups (text classification and knowledge, applied decisions, evaluation and control, agent and action), where labels and instructions are inherited from the original sources rather than newly created.

2. A real-time evaluation protocol: In Atari, Doom, driving, and quadrotor environments, the world advances by the model's measured median latency while it is deciding, so slower models act on older observations and response time becomes part of task performance.

3. A method for extracting decision probabilities from open LLMs: Probabilities are read from answer-label tokens (A, B, C … for choices; 0–9 for score levels; yes/no for booleans) and normalized with a softmax, including a read-only denoising step for the diffusion model DiffusionGemma, and reading labels at the start of the final answer for GPT-OSS, which requires reasoning.

4. A 30-model evaluation: 7 closed LLMs (using verbalized probabilities in JSON), 13 open LLMs (read at the answer token), and 10 System-1 models, scored on accuracy and Brier score, plus latency, cost, output tokens, and agentic step and closed-loop success — including five paired comparisons between a System-1 model and the LLM it was derived from or shares a backbone with.

Main Findings

• The largest System-1 models match fast closed LLMs on non-agentic tasks. Decision-2.0-Vega-27B reaches 73.0% and Jev reaches 72.6% overall accuracy on 78,362 non-agentic questions across 87 tasks, versus 71.2% for GPT-6 Luna and 70.0% for GPT-5.6 Luna. Both System-1 models stay within 1.4 percentage points of the best open LLM, Gemma-4-31B (73.9%).

• System-1 models produce better-calibrated probabilities than fast LLMs. Decision-2.0-Vega-27B and Jev achieve mean Brier scores of 0.343 and 0.349, better than Gemma-4-31B (0.474), GPT-6 Luna (0.461), and GPT-5.6 Luna (0.458). Only four models score lower, all closed LLMs costing 20–105 times as much as GPT-6 Luna.

• A reasoning LLM is the most accurate, at much higher cost. Claude Opus 5.5 reaches 81.1% accuracy with the lowest Brier score (0.232) and leads in 9 of 13 domains, but costs $10.38 per 1,000 requests — described as about 70 times the cost of a fast LLM (and roughly 74 times GPT-6 Luna's cost) for a 9.9-percentage-point accuracy gain. GPT-6 Astra is the most expensive at $15.05 yet less accurate (80.1%).

• Speed favors local and System-1 models. Without reasoning, open LLMs take 21–105 ms per request and local System-1 models 41–228 ms, while Jev takes 321 ms and Microsoft-Decision-1 304 ms through their gateways, and closed LLMs take 1.3–6.7 s. Jev achieves 72.6% at $0.052 per 1,000 requests, and Microsoft-Decision-1 is the cheapest hosted model at $0.038 with 70.9%.

• Thinking raises accuracy but can worsen calibration. With no reasoning, all four open LLMs tested with reasoning budgets are less accurate than Jev (59.3–70.2% versus 72.6%), and a 128-token budget hurts — DiffusionGemma drops from 69.7% to 56.3%. At 2,048 tokens, Qwen3.8-27B rises from 70.2% to 75.6% but its Brier score worsens from 0.391 to 0.430 as mean confidence in the chosen option climbs from 81.4% to 97.4%.

• Forecasting is the weakest domain for System-1 models. The best System-1 model on forecasting, the OpenAI Decisions API, reaches 48.3% versus 61.6% for Claude Opus 5.5 — a 13.3-percentage-point gap — and forecasting is the hardest domain for 25 of the 29 models with non-agentic results, with no model exceeding 61.6%.

• Small gaps on selection-style tasks; large gaps on knowledge and code. The System-1-versus-LLM gap is smallest for filtering, grouping and ranking (0.5 points), retrieval and ranking (2.3), and dialogue decisions (3.9), and largest for code understanding (13.1), event forecasting (13.3), and structured data (12.2).

• Similar single-step accuracy does not predict closed-loop success. Nine models land at 67.4–71.3% step accuracy, but closed-loop success ranges from 5.7% to 32.9%. Decision-2.0-Vega-27B, Jev, and Qwen3.5-9B all hit 67.8% step accuracy yet succeed in 26.5%, 17.4%, and 5.7% of episodes respectively.

• Large open LLMs win closed loops; small System-1 models fail. Gemma-4-31B has the highest closed-loop success rate (32.9%), followed by Decision-2.0-Vega-27B (26.5%) and Qwen3.8-27B (26.2%). The four small System-1 models reach at most 3.1%, and the four open LLMs under 10B parameters succeed in fewer than 6% of episodes.

• Deriving a System-1 model from an LLM helps static accuracy but hurts agentic steps. Across five paired comparisons, System-1 models improve non-agentic accuracy by 2.9–11.0 points in three pairs (Vega, Microsoft-Decision-1, Decision-2.0-Kai-0.6B) but lose 2.6 and 17.8 points in two (the Decisions API and CLM-v0.1-8B), and lower step accuracy by 1.2–19.3 points in all five pairs.

• Answer-token probabilities can be badly overconfident. Gemma-4-31B assigns a probability of at least 0.999 to its selected option on 87% of questions, which helps explain its weaker Brier score despite strong accuracy.

Methodology in Plain English

The researchers defined a common "decision request" format: a state made of named text fields (a message, a code diff, a rendered web page) plus a set of typed questions sharing that state. Questions are boolean (yes/no), choice (2–255 options with exactly one correct answer), or score (2–10 ordered levels). Each question carries instructions and each option carries a description, so no model needs a dataset-specific prompt.

Every model receives the same requests, but probabilities are obtained differently by family. System-1 models return a distribution directly through their API or official code. Closed LLMs are asked to write probabilities as numbers in a JSON object, which are normalized (missing options get 0) — token probabilities are not used. Open LLMs get one question per prompt ending at the start of the answer, and the log-probabilities of the answer label tokens are softmax-normalized into a distribution. Each model family uses one fixed prompt with no per-model tuning.

Scoring uses accuracy (top-probability option versus the hard label) and the Brier score, computed as the sum over options of squared differences between predicted probability and the target (one-hot label or annotator distribution), ranging 0 to 2 with lower being better. Scores are averaged within a task, then across tasks so every task has equal weight; additional metrics include expected calibration error with 10 equal-mass bins, NDCG, and pair accuracy.

For agentic tasks, a fixed task-specific harness converts the current state and interaction history into text and presents available actions as typed questions, executes the highest-probability option, and observes the next state. Free-form outputs such as search terms, tool arguments, and patches are turned into candidate choices. To prevent the harness from trivializing tasks, environments where a uniform-random policy succeeds in at least 20% of episodes or models generally reach at least 90% success were flagged, and assistance was removed or the decision budget reduced (for example, limited to twice the shortest solution length in BabyAI, MiniHack mazes, and Baba Is AI), with a scripted policy confirming solvability.

Why This Matters

Impact on research. Existing System-1 evidence was thin: JevBench and ReflexBench each contain only a few hundred synthetic decisions, mainly compare System-1 models to each other, and omit agentic tasks. System1-Arena replaces that with 92,125 source-labeled questions, a difficulty-checked harness, a real-time protocol, and a paired design that isolates the effect of the decision interface versus the underlying LLM. It also separates accuracy from calibration, showing that rankings change when probability quality is measured.

Real-world applications:

• Security triage and moderation: flagging phishing, judging safety and toxicity content, and routing alerts, where the benchmark shows small System-1-versus-LLM gaps (4.2 points on safety and moderation, 4.7 on text classification).
• Human-in-the-loop escalation: probability outputs directly support decision thresholds and escalation rules, and Jev's $0.052 per 1,000 requests makes high-volume use viable.
• Model and tool routing: the paper frames its results as informing routing between System-1 models and generative LLMs, and the routing domain gap is 5.8 points.
• Latency-critical embodied systems: robotics, driving, games, and quadrotors, where the world keeps changing during inference; but the results show small System-1 models succeed in at most 3.1% of closed-loop episodes, so this use case remains out of reach for them today.

Industry relevance. The cost and latency table gives practitioners concrete trade-offs: paying roughly 70 times more for a reasoning LLM buys about 10 accuracy points, while a System-1 model can match a fast closed LLM at a fraction of the price and latency — but not in long-horizon loops, where larger models dominate.

Future Directions

• Closing the forecasting gap. Forecasting is the hardest domain for 25 of 29 models and the largest System-1 shortfall (13.3 points), yet low latency is attractive for trading-style uses; the authors state that using System-1 models as forecasting components requires careful design and validation.
• Better harnesses and planning for small System-1 models in closed loops. The four small System-1 models top out at 3.1% success, which the paper attributes to limited long-horizon planning ability.
• Understanding why step accuracy fails to predict loop success. Errors accumulating across interaction is offered as one possible explanation, but the mechanism behind the wide spread (5.7% to 32.9% at comparable step accuracy) is left open.
• Reconciling accuracy gains with calibration losses under reasoning. Reasoning improves accuracy but concentrates probability mass and worsens Brier scores, motivating methods that preserve calibration when deliberation is added.

Target Audience

Researchers and engineers working on LLM evaluation, inference cost and latency optimization, model routing and cascading, calibration, and agentic systems. It is most useful to practitioners deciding whether a given decision pipeline needs a reasoning LLM or can be served by a fast probability-returning model, and to benchmark designers interested in a shared interface across model classes and in real-time evaluation protocols. Readers without background in calibration metrics will need to pick up Brier scores and expected calibration error along the way, though the paper defines both.

Authors’ abstract

Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.