Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives

AI summary

Overview

• Research area: Natural Language Processing / LLM reasoning — specifically formal (syllogistic) reasoning versus natural language understanding.
• Technical level: Intermediate. The logical formalism (A/E/I/O moods, figures, validity) is explained from scratch, but the results section leans on paired t-tests, McNemar tests, Friedman tests, Wilcoxon signed-rank tests, and Spearman/Pearson correlations.
• Scope: An empirical evaluation of 14 large language models on 160 syllogisms across 4 prompting strategies and 3 temperature settings, testing whether models reason like formal logic engines or like humans subject to belief bias.

What This Paper Is About

The authors ask whether making LLMs better at logic actually makes them more human-like reasoners — and argue the answer may be no. They build a benchmark of syllogisms where each item carries two independent ground-truth labels: whether the conclusion follows logically (syntactic validity) and whether the conclusion is intuitively plausible (natural language believability). By testing 14 models on cases where these two labels agree and cases where they conflict, they can measure pure formal reasoning and belief bias separately.

Key Contributions

1. A dual ground truth framework. Every syllogism is annotated on two orthogonal dimensions — syntactic validity (valid/invalid) and natural language believability (believable/unbelievable) — allowing formal reasoning ability to be scored independently of natural language understanding.
2. A controlled benchmark of 160 syllogisms. 40 handcrafted base syllogisms were expanded into four variants each: normal (N), nonsense (X, with predicates replaced by abstract terms like "blargs" and "zimons"), order-switched (O, premises reversed), and combined (OX).
3. A large systematic evaluation. 14 state-of-the-art models across 8 organizations, 4 prompting strategies, and 3 temperature settings, producing 168 unique configurations and 26,880 model-instance evaluations at roughly $500 in total API cost.
4. A belief-bias quantification for LLMs. The authors adapt a direct accuracy-differential metric (congruent minus incongruent accuracy) from cognitive psychology rather than traditional endorsement-rate indices.

Main Findings

• Performance is bimodal, not uniform. Six models exceed 95% syntax accuracy; five score below 70%. Gemini 2.5 Flash is highest at 99.6%, and Llama 3.2 1B Instruct is lowest at 51.9%. The overall mean is 81.7% (SD = 17.1%), with a 47.7 percentage-point gap between top and bottom performers.
• Architecture matters more than scale. The authors report that the wide spread across a 1B-to-671B parameter range indicates syllogistic reasoning depends on architectural and training choices rather than raw parameter count.
• Models excel at syntax but fail at believability. Syntax accuracy (81.7%) exceeds NLU accuracy (56.2%) by 25.50 percentage points. Top models show the largest gaps: Gemini 2.5 Flash (47.9 pp), GPT-OSS-20B (47.9 pp), and Gemini 2.5 Pro (47.4 pp) perform near chance on NLU.
• This is the opposite of human behavior. Humans are typically dominated by belief bias; these top models correctly judge logical validity independent of whether the conclusion is plausible.
• Few-shot prompting hurts. Few-shot yields the lowest mean accuracy (79.1%) versus zero-shot (82.7%), a difference of −3.57 pp (t₄₁ = 2.50, p = 0.0165, Cohen's d = −0.39), which survives Holm-Bonferroni correction (p_adj = 0.0495). At the instance level, the McNemar test (N = 6720) shows error redistribution: zero-shot solves 786 instances few-shot fails, while few-shot solves only 546 that zero-shot fails (χ² = 42.88, p < 0.0001).
• Strategy effects are not universal. A Friedman test finds no significant overall strategy effect (χ² = 3.24, df = 3, p = 0.356), and a Wilcoxon signed-rank test makes the zero-shot vs. few-shot effect marginally non-significant after correction (p = 0.0195, p_adj = 0.0584).
• Temperature has negligible impact. With adaptive majority voting, mean accuracy is virtually identical across temperatures (Friedman χ² = 3.77, df = 2, p = 0.152).
• Belief bias is widespread. Twelve of 14 models (86%) show positive bias, with a mean effect of +10.81 pp (SD = 16.32, t₁₃ = 2.47, p = 0.0280, d = 0.66). Llama 3.2 3B Instruct shows the largest bias at +46.9 pp; Gemma 3 27B IT shows the most negative at −13.7 pp. Top-tier models stay under 2 pp.
• Better reasoning means less bias. The correlation between syntax accuracy and bias magnitude is moderately negative (Spearman ρ = −0.565, p = 0.0353), meaning higher-performing models rely less on semantic heuristics.
• Accuracy tracks consistency. Syntax accuracy correlates very strongly with overall consistency across all four variants (Pearson r = 0.877, p < 0.0001; Spearman ρ = 0.890, p < 0.0001), with similar results for the normal↔nonsense (ρ = 0.846) and order-switched (ρ = 0.837) pairwise metrics.
• Reasoning ability tracks human preference rankings. Syllogistic accuracy correlates strongly with LMArena rank (Spearman ρ = −0.825, p = 0.0010, N = 12), where lower rank is better — suggesting instruction-following quality and formal reasoning draw on the same rule-adherence capacity.
• Precision and recall reveal distinct failure modes. Qwen3-Next 80B A3B Thinking shows 99.2% precision but only 42.8% recall, labeling most syllogisms "incorrect"; Gemma 3 27B IT shows 93.1% recall but 61.0% precision, over-accepting conclusions. Top-tier models keep both above 97%.
• Syntax and NLU accuracy move in opposite directions. The moderate negative correlation (Spearman ρ = −0.543, p = 0.0449) suggests models optimized for logical structure diverge from intuitive plausibility judgments.

Methodology in Plain English

The researchers started with 40 handcrafted syllogisms covering different logical figures and validity conditions, drawn from cognitive science and psychology literature on how humans do syllogistic reasoning. Each base syllogism was rewritten into three additional versions: one where meaningful terms are swapped for invented words (to strip out real-world meaning), one where the premises are reordered (to test structural sensitivity), and one doing both. That yields 160 items.

Each item carries two labels. The syntactic label says whether the conclusion actually follows from the premises under formal syllogistic rules. The NLU label says whether a person would find the conclusion intuitively plausible. For some items these align (congruent), and for others they conflict (incongruent) — the latter being the diagnostic cases for belief bias.

Fourteen models from eight organizations were queried under four prompting conditions (zero-shot, one-shot, four-example few-shot including a belief-bias trap, and zero-shot chain-of-thought) and three temperatures (0.0, 0.5, 1.0). Regardless of condition, every prompt demanded the same one-word answer, "correct" or "incorrect," so results stay comparable. At temperature 0 the model is queried once and given full confidence. Above 0, the system samples up to 10 times, stops early if the first 5 samples are unanimous, and takes a majority vote, with ties defaulting to "invalid." Responses are then scored against both ground truths independently, so a single answer is graded twice — once for logic and once for believability.

Why This Matters

The paper reframes a common research goal. If the aim is to build systems that reason the way people do, then maximizing formal logic accuracy may be moving in the wrong direction — the best models behave like logic engines rather than fallible human reasoners.

• Research impact: It supplies a reusable dual-annotation benchmark and a within-subjects belief-bias metric, and it challenges the assumption that few-shot examples always help reasoning tasks.
• Logic and mathematics tutoring: A model with 99.6% syntax accuracy but near-chance plausibility judgment may correctly grade a proof step while being unable to explain why a conclusion feels wrong to a student.
• Legal and compliance reasoning: A system that prioritizes formal rule-following over plausibility may accept or reject arguments in ways that look counterintuitive to human reviewers, which matters when conclusions must be justified to a person.
• Fact-checking and misinformation detection: The finding that top models ignore believability cuts both ways — it could reduce susceptibility to plausible-sounding but invalid arguments, but it also means the model is not using real-world knowledge as a check.
• Benchmark design and model evaluation: The strong correlation with LMArena rank (ρ = −0.825) suggests syllogistic accuracy may serve as a cheap proxy for instruction-following quality, while the disconnect from NLU accuracy warns against treating any single score as a measure of general reasoning.

Industry relevance: The paper's practical claim is that architectural and training choices, not parameter count, determine formal reasoning ability — relevant to anyone choosing a model for logic-heavy workloads who might otherwise default to the largest available option. The finding that few-shot prompting degraded performance also argues against a common default practice in production prompt design.

Future Directions

1. Extend to richer logical systems. The authors propose testing modal logics and transitive closure logics, especially systems with simple formal syntax but complex natural language semantics, to see whether the formal-vs-natural-language divide persists.
2. Mechanistic interpretability. It remains unknown whether models learn explicit logical rules, statistical approximations, or hybrid representations — a question that would explain the near-perfect syntactic performance of top-tier models.
3. Untangle causality between reasoning and bias. Does logical training reduce bias, or does reduced bias enable better reasoning? The authors suggest controlled fine-tuning experiments.
4. Investigate when few-shot prompting helps versus hurts. The counterintuitive degradation under few-shot prompting warrants systematic study to inform better prompting strategies.
5. Add finer-grained measures. The paper's limitations section calls for response time analysis, confidence calibration, and fine-grained error taxonomies, plus robustness tests against adversarial perturbations and manipulated distractors.

Target Audience

Researchers in NLP and LLM evaluation who study reasoning benchmarks; cognitive scientists interested in whether machine reasoning mirrors or diverges from human belief-bias effects; and practitioners selecting or prompting models for logic-heavy tasks such as legal analysis, mathematical verification, or structured decision support. The paper assumes no prior background in syllogistic logic — it defines A/E/I/O moods and the four figures — but the results section uses standard statistical tests without explanation, so some familiarity with hypothesis testing helps.

Limitations and Funding (as reported)

The paper restricts itself to categorical syllogisms plus a few others, which may not generalize to nested quantifiers or modal operators. It covers 14 models and four prompting strategies, and its consistency metrics test only content and order variations, not adversarial robustness. The belief-bias metric captures one dimension of the belief–logic relationship. The work was supported by the Indo-French Centre for the Promotion of Advanced Research (IFCPAR/CEFIPRA), project number CSRP-6702-2, with code at https://github.com/XAheli/Logic-in-LLMs.

Authors’ abstract

We study syllogistic reasoning in LLMs from the logical and natural language perspectives. In process, we explore fundamental reasoning capabilities of the LLMs and the direction this research is moving forward. To aid in our studies, we use 14 large language models and investigate their syllogistic reasoning capabilities in terms of symbolic inferences as well as natural language understanding. Even though this reasoning mechanism is not a uniform emergent property across LLMs, the perfect symbolic performances in certain models make us wonder whether LLMs are becoming more and more formal reasoning mechanisms, rather than making explicit the nuances of human reasoning.