The Pulse
ArgGYM Tests Language Models on Revisable Arguments
ArgGYM is a benchmark of 1,440 verified instances across 12 tasks for testing how language models handle structured arguments and revisable conclusions. Its paper reports performance differences across models and tasks, with results declini
AI.info Team · Written with AI ·
Researchers submitted ArgGYM to arXiv on September 29, introducing a benchmark built to test whether language models can revise conclusions when arguments meet counterevidence. Its fixed evaluation set contains 1,440 verified instances, and its procedural generators can produce new ones for evaluation or reinforcement learning. The authors position the release as a test of reasoning that can change when information changes, rather than a contest with one stable answer key.
From support to defeat, across 12 tasks
ArgGYM focuses on defeasible reasoning: conclusions that hold provisionally, but can be defeated by stronger or conflicting reasons, reinstated by other arguments, or revised after an update. The benchmark breaks that work into 12 tasks, including identifying why an argument fails, determining which claims are accepted, predicting how a theory changes after new information, and constructing attacks or defenses.
Tasks share an ASPIC+ formal framework, where arguments are built from premises and rules, attacks can undermine premises or conclusions, and preferences help determine which attacks count as defeats. A symbolic engine computes or validates the formal states used for scoring. The engine handles generation and verification; it is not available to the model while the model answers.
The frozen set spans 15 curriculum configurations, two argument preference orderings—weakest-link and last-link—and two ways of ordering sets of arguments, called elitist and democratic. The paper’s count also includes 12 tasks and two accepted instances for each combination: 12 × 15 × 2 × 2 × 2 = 1,440. Separate generators can create fresh, engine-verified cases, while the fixed set stays unchanged for reproducible comparisons.
Partial answers do not always add up
The paper evaluates 18 model variants from five families: Gemma, GPT-OSS, Qwen, DeepSeek and Gemini. The reported primary metric is complete-task success: whether a model meets the full task-specific output requirements, rather than recovering only part of a structured answer.
Gemini 3.8 Flash at high reasoning effort posts the highest complete-task success rate, 82.4%. But the aggregate hides weak spots: the same configuration completes 57.5% of defeat-diagnosis tasks and 60.0% of counter-argument tasks that allow strict-rule additions. Results also vary by task and model; the authors say that similar overall scores can mask different strengths and failures.
“Models can therefore recover large portions of the required argumentative state while still failing to assemble a globally correct solution.”
İbrahim Ethem Deveci, Funda Tan Çalık, Barış Deniz Sağlam and Duygu Ataman, authors of the ArgGYM paper
Longer and more interconnected cases also expose a performance gap. For GPT-OSS-120B at high reasoning effort, complete-task success falls from 45.2% in curriculum configurations L1–L5 to 16.0% in L11–L15. The authors caution that these configurations vary several structural features at once, so the bands are diagnostic rather than a calibrated, one-dimensional scale of difficulty.
The test is formal by design—and limited by that choice
The engine makes it possible to check outputs against computed argument states, but most tasks use symbolic theories rather than varied natural-language scenarios. The paper says its one controlled natural-language task uses template-generated descriptions, leaving open how well performance transfers to less constrained language and world knowledge.
The evaluation also omits Anthropic’s Claude models and OpenAI’s proprietary GPT systems, a limitation the authors identify directly. ArgGYM therefore gives researchers a way to inspect specific operations—such as defeat analysis and theory revision—within a formal system, but its results do not establish how models handle the same reasoning in open-ended everyday settings.