LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning

AI summary

Overview

• Research area: Natural Language Processing; specifically commonsense reasoning evaluation for large language models, with a focus on compositional and multi-answer plausibility judgments.
• Technical level: Intermediate. The paper is readable without deep formal logic training, but it assumes familiarity with multiple-choice benchmarking, zero-shot/few-shot prompting, chain-of-thought prompting, and supervised fine-tuning of language models.
• Scope: The paper introduces, validates, and empirically evaluates a benchmark that recasts commonsense question answering as logical composition over pairs of atomic statements using the plausibility-level operators AND, OR, and NEITHER/NOR.

What This Paper Is About

Most commonsense reasoning benchmarks ask a model to pick a single correct answer out of several options, even though real-world commonsense questions often admit several plausible interpretations at once. This paper builds Logical-CommonsenseQA, which reformulates existing commonsense questions so that each answer option is a composition of two atomic statements joined by AND, OR, or NEITHER/NOR, forcing models to judge whether statements are jointly plausible, partially plausible, or jointly implausible. The goal is to expose reasoning failures that single-answer benchmarks hide.

Key Contributions

1. A new benchmark for compositional plausibility. Logical-CommonsenseQA extends CommonsenseQA (described in the paper as 12,247 multiple-choice questions with one correct answer and four distractors) into 19,996 instances, with 4,999 instances each for AND, OR, NEITHER/NOR, and a Mixed condition in which operators vary across options within the same question.

2. A three-stage, hybrid construction pipeline. Candidate answers are over-generated with GPT-4o-mini, refined and pruned with GPT-4o-mini, and then combined into logical compositions by a deterministic symbolic program rather than by further model inference.

3. Human validation grounded in an awareness–consensus framework. Two independent annotators judged 250 questions from the Stage 2 test split, corresponding to 50% of the 500-question test set, rating both personal belief (awareness) and perceived social agreement (consensus).

4. A broad empirical evaluation plus a decomposition analysis. The paper evaluates encoder-only classifiers, encoder–decoder models, open-weight decoder-only LLMs, and proprietary closed-source models under zero-shot, few-shot, chain-of-thought, and fine-tuned regimes, then separates atomic plausibility estimation, operator grounding, and distractor competition to locate the source of failures.

Main Findings

• Conjunction is easiest, negation collapses. On the human-validated test set, 0-shot LLaMA-3.3-70B reaches 80.9 macro-F1 on AND, 70.9 on OR, 53.0 on Mixed, but only 13.4 on NEITHER/NOR. LLaMA-3.1-8B follows the same pattern (71.9 AND, 62.2 OR, 41.8 Mixed, 13.1 NEITHER/NOR), as does Qwen2.5-7B (79.6 AND, 68.9 OR, 53.2 Mixed, 12.9 NEITHER/NOR).

• Closed-source models show the same qualitative trend. Gemini-2.5-Flash scores 84.1 on AND, 76.7 on OR, 60.9 on Mixed, and 23.5 on NEITHER/NOR; Gemini-3-Flash-Preview scores 82.5, 70.1, 61.8, and 39.6 respectively.

• Mixed operator conditions are hard. In the Mixed condition, macro-F1 drops to 43–56% for all few-shot models, which the authors interpret as evidence that models do not reliably maintain operator-level representations and instead fall back on surface-level heuristics.

• Fine-tuning closes most of the gap. Fine-tuned models achieve 83–95% F1 across all operators: Flan-T5-base (92.8 AND, 92.4 OR, 89.2 NEITHER/NOR, 89.6 Mixed), DeBERTa-v3-base (87.6, 87.2, 84.8, 82.4), and fine-tuned LLaMA-3.1-8B (93.6, 94.4, 89.5, 91.5).

• Un-finetuned encoder and encoder–decoder models perform poorly. The paper reports that encoder-only and encoder–decoder models without fine-tuning exhibit uniformly low performance; for example, AufEntailer-11B scores 5.5 macro-F1 on the non-validated AND split and 3.2 on the human-validated AND split.

• The headline comparison with CommonsenseQA. LLaMA-3.1-8B achieves 72.2% accuracy on CommonsenseQA, but on Logical-CommonsenseQA it scores 72.0 accuracy on AND, 62.2 on OR, 42.7 on Mixed, and only 13.9 on NEITHER/NOR. The authors argue this shows single-answer benchmarks substantially overestimate commonsense ability.

• Humans do not show the same negation collapse. A small-scale human evaluation using 10 examples per condition reports human accuracy of 0.90 on AND, 0.70 on OR, 0.70 on NEITHER/NOR, and 0.90 on Mixed.

• The difficulty is not just atomic plausibility knowledge. Atomic plausibility classification on Stage 2 statements reaches 79.4 accuracy and 77.8 macro-F1 on the human-validated subset (78.5 accuracy, 78.1 macro-F1 on the non-validated subset). Yet operator verification with gold atomic labels stays imperfect at 52%–69%, and adding competing distractor options pushes NEITHER/NOR from 71% to 47%.

• Human label validation was strong, but negation labels were weaker than the rest. Inter-annotator agreement using Gwet's AC2 with ordinal weighting was 0.84 for awareness and 0.91 for consensus. Label accuracy after composition was 89.2% for AND, 96.4% for OR, 88.4% for Mixed, and 73.6% for NEITHER/NOR.

• The authors argue the dataset is not degenerate. Distinct-2 is 0.70 for atomic options and 0.23 for composite options; global semantic diversity is 0.89 for both; intra-question diversity is 0.73 for atomic and 0.47 for composite options. The lower composite numbers are attributed to deterministic reuse of atomic statements and fixed operator tokens.

Methodology in Plain English

The researchers start from CommonsenseQA and sample 5,000 of its questions. In Stage 1, GPT-4o-mini is prompted to over-generate plausible and implausible atomic answer candidates—4–6 plausible and 4–6 implausible per question—that require multi-step causal, social, or situational reasoning rather than keyword matching.

In Stage 2, GPT-4o-mini refines and prunes those candidates: removing inconsistent or trivial options and turning incorrect options into near-miss distractors that satisfy most constraints but fail on one subtle point. This yields three correct and four incorrect atomic options per question. One sampled question is removed, leaving 4,999 curated question sets.

In Stage 3, a symbolic program—not a language model—deterministically combines pairs of refined atomic options into compositions labeled AND, OR, or NEITHER/NOR, producing 14,997 operator-specific instances (4,999 per operator). A Mixed setting adds 4,999 more instances in which operators vary across the answer choices of a single question, for a final total of 19,996 instances.

The dataset is split into 11,996 training, 6,000 development, and 2,000 test examples, stratified so each of AND, OR, NEITHER/NOR, and Mixed makes up 25% of the data. The test set is further divided into a human-validated (HV) and a non-validated (NV) subset, each 50% of the test data, and results are reported as accuracy and macro-F1.

Models are evaluated under several regimes. Open-weight decoder-only models (Llama-3.3-70B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct) are run in 0-, 1-, 2-, and 3-shot settings with temperature 0.0, top-p 0.9, and max_new_tokens of 3; invalid outputs count as incorrect. Chain-of-thought prompting is run only for Llama-3.1-8B-Instruct, with a larger generation budget of 200–300 new tokens. Proprietary models (Gemini-2.5-Flash and Gemini-3-Flash-Preview) are evaluated zero-shot only. DeBERTa-v3-base, Flan-T5-base, and Entailer-11B are evaluated, with DeBERTa-v3-base, Flan-T5-base, and Llama-3.1-8B-Instruct also fine-tuned. A decomposition analysis then tests atomic plausibility classification, operator verification with gold atomic labels, and composition with and without competing distractors.

Why This Matters

The paper's central claim is that single-answer commonsense benchmarks inflate measured model competence. A model that scores 72.2% on CommonsenseQA falls to 13.9% on the negation-based subset of Logical-CommonsenseQA, which suggests that reported commonsense performance partly reflects answer-selection shortcuts rather than the ability to reason about how plausible statements relate to one another.

The decomposition analysis adds a second insight for research practice: the failure is not one thing. Atomic plausibility estimation is comparatively strong, operator grounding is imperfect even with gold labels, and distractor competition compounds both. That points researchers toward operator-level and competition-aware evaluation rather than treating commonsense as a single skill.

Potential real-world applications, framed as implications of the benchmark rather than results the paper reports:

• Dialogue systems that must decide whether multiple suggested interpretations of a user's request can hold at the same time, are alternatives to each other, or are all ruled out.
• Question answering and search where several answers are simultaneously valid and a system should not force a single choice.
• Decision-support and planning assistants that weigh candidate plans which may be jointly viable, mutually exclusive, or jointly infeasible.
• Model auditing and safety review, where a controlled template for probing negation handling helps identify brittle inference before deployment.

On industry relevance: the paper's finding that fine-tuned models reach 83–95% F1 across all operators while prompting alone collapses on NEITHER/NOR implies that applications requiring negation-sensitive commonsense judgments may need task-specific supervision rather than prompt engineering alone. The paper notes, however, that it does not examine transfer to downstream tasks such as open-ended question answering, dialogue, or planning, so downstream benefits are not demonstrated here.

Future Directions

• Extend to generative and structured reasoning settings. The authors list extending the benchmark beyond multiple-choice to generative settings and structured reasoning frameworks as a next step.

• Expand the operator set. The current operators capture plausibility-level relations only and do not cover implication, exclusivity, or temporal and causal reasoning; richer operators would allow a more complete evaluation of symbolic and relational inference.

• Scale the awareness–consensus validation. The human validation currently covers 250 of the 500 Stage 2 test questions; the authors propose scaling it further.

• Test transfer to downstream tasks. The Limitations section states that transfer to open-ended question answering, dialogue, and planning is not examined, and the Future Work section names these as targets.

• Broaden model coverage. The authors note that despite evaluating a diverse set of architectures, the study remains limited in scope and that broader model comparisons could clarify which components most improve logical composition.

Target Audience

This paper is most useful to benchmark designers and evaluation researchers working on commonsense reasoning and compositional inference in LLMs, who will care about the construction pipeline, the awareness–consensus validation protocol, and the HV/NV test split. It also serves practitioners who rely on prompting or fine-tuning and want evidence on where negation-based reasoning breaks down, as well as researchers interested in the decomposition of a task into atomic plausibility estimation, operator grounding, and distractor competition. Readers looking for a formal logic benchmark should note the authors' own caveat: the operators encode plausibility-level judgments, not full propositional semantics.

Authors’ abstract

Commonsense reasoning often involves evaluating multiple plausible interpretations rather than selecting a single atomic answer, yet most benchmarks rely on single-label evaluation, obscuring whether statements are jointly plausible, mutually exclusive, or jointly implausible. We introduce LOGICAL-COMMONSENSEQA, a benchmark that reframes commonsense reasoning as logical composition over pairs of atomic statements using plausibility-level operators (AND, OR and NEITHER/NOR). Evaluating instruction-tuned, reasoning-specialized, and fine-tuned models under zero-shot, few-shot, and chain-of-thought prompting, we find that while models perform reasonably on conjunctive and moderately on disjunctive reasoning, performance degrades sharply on negation-based questions. LOGICAL-COMMONSENSEQA exposes fundamental reasoning limitations and provides a controlled framework for advancing compositional commonsense reasoning.