Abductive Reasoning with Syllogistic Forms in Large Language Models

AI summary

Overview

• Research area: Natural Language Processing / LLM reasoning evaluation, at the intersection of formal logic and cognitive psychology.
• Technical level: Intermediate. The paper is readable without deep logic training, but familiarity with syllogisms, deduction, and belief bias helps.
• Scope in one sentence: The paper converts a syllogistic reasoning dataset into an abduction dataset in order to measure how accurately four state-of-the-art LLMs perform abductive inference and whether they show human-like belief bias.

What This Paper Is About

Most studies of LLM reasoning test deduction, but everyday human reasoning also relies heavily on abduction (drawing a tentative explanation from limited information). The authors build an abduction task by swapping the minor premise and conclusion of deductively valid syllogisms, following Peirce's original syllogistic account of abduction, and compare LLM performance on abduction versus deduction. They also test whether LLMs show the belief bias seen in humans, where logically valid inferences that clash with common beliefs are wrongly rejected.

Key Contributions

1. A new dataset for abductive reasoning in syllogistic form. The authors built 216 abductive inference problems (108 correct patterns and 108 incorrect patterns) derived from 27 manually created term triples, plus a matched set of 216 deductive problems (108 valid, 108 invalid) for direct comparison.
2. Belief-bias annotation. Each problem is labeled Consistent (66), Inconsistent (66), or Neutral (84), enabling content-versus-form analysis — the problem is Consistent if the Rule is true by common sense, Inconsistent if it contradicts common sense, and Neutral otherwise.
3. A systematic zero-shot and few-shot evaluation of four models — GPT-3.5, GPT-4, Llama-3-8B (8 billion parameters), and Llama-3-70B (70 billion parameters) — on both tasks.
4. An error analysis of why models fail, including a deduction-label analysis, a "Negative" selection rate analysis, and a test of whether the word "Hypothesis" misleads models toward entailment.

Main Findings

• LLMs are worse at abduction than deduction. For example, GPT-4 reached 41.67% overall accuracy on the zero-shot abduction task versus 72.22% on the zero-shot deduction task; and in the few-shot setting Llama-3-70B reached 75.46% on abduction versus 84.72% on deduction.
• The "Neither" answer type is the main failure mode. In zero-shot abduction, GPT-4 scored 0.00% on problems whose correct answer was "Neither"; GPT-3.5 scored 0.93% and Llama-3-70B scored 0.93%.
• Few-shot prompting helps deduction consistently but not abduction. Deduction gains from few-shot prompting ranged from 6.02 to 31.48 points in overall accuracy across all four models; GPT-4's abduction accuracy actually dropped from 41.67% zero-shot to 28.70% few-shot.
• Llama-3-70B is the exception. In the few-shot setting it improved to 75.46% overall on abduction and 62.96% on "Neither" problems, though its abduction score remained below its own deduction score (84.72%).
• Human-like belief bias appears in abduction. In zero-shot abduction, accuracy on Inconsistent problems was around 10% lower than the other two types (GPT-4: 34.85% Inconsistent versus 46.97% Consistent and 42.86% Neutral).
• Belief bias also appears in deduction, except for Llama-3-8B: GPT-4 in few-shot scored 100.00% on Consistent, 92.42% on Inconsistent, and 95.24% on Neutral.
• Models over-select negated answers. GPT-4 chose "Negative" in 67.90% of abduction cases where "No" or "not" appeared in the Rule or Observation, while the actual rate of the correct answer being Negative was 16.67%. In deduction the selection rate was 70.99% against an actual rate of 50%, so the tendency is stronger in abduction — consistent with an effect like the atmosphere effect.
• Models are partly, but not entirely, treating abduction as deduction. Scoring abduction problems with deductive labels, GPT-4's agreement rate was 51.85% overall — about 10 points higher than its original abduction accuracy — but still about 20 points below its deduction-task accuracy overall, and about 35 points below on "Neither".
• Rewording "Hypothesis" as "Reason" did little. The substitution had little effect on improving the score.
• Models rarely pick self-contradictory answers. Few cases were observed where the model chose an answer contradicting the given Rule or Observation.

Methodology in Plain English

The authors took syllogisms — two premises leading to a conclusion — and turned them into abduction problems by swapping the minor premise and the conclusion. A deductive syllogism of the form "All A are B; C is A; therefore C is B" becomes an abduction of the form "All A are B; C is B; therefore C is A." In the paper's terminology, the first sentence is the Rule, the second is the Observation, and the derived sentence is the Hypothesis.

They created 27 triples of terms where A is a subject term, B is an observational predicate (verifiable by direct observation, e.g. "are white"), and C is a non-observational predicate (e.g. "were in the bag"). Instantiating eight inference patterns with these triples produced 216 abduction problems, four patterns counted as correct and four as incorrect. Deduction problems were made from the same material by switching the Observation and Hypothesis roles, with eight corresponding patterns (four valid, four invalid).

Each problem was presented with three answer options: the positive Hypothesis, the negation of the Hypothesis, and "Neither is a good explanation." The authors ran zero-shot prompts and few-shot prompts containing eight exemplars covering the eight patterns. Maximum output token length was set to 10, other hyperparameters were left at defaults, and no fine-tuning was performed. Alternate prompt phrasings were tested but scored lower and were not adopted.

Why This Matters

Research impact: The paper extends LLM reasoning evaluation beyond deduction into abduction, which prior syllogism datasets (Dong et al., Guebelmann et al., Aghahadi et al., Ando et al.) do not cover. It connects the LLM bias literature (Dasgupta et al., Ando et al., Ozeki et al.) to the philosophical account of abduction from Peirce, and it provides a controlled dataset where correct and incorrect inference patterns are formally defined.

Real-world applications:

• Explainable AI (XAI): systems that must answer "why" questions by producing candidate explanations rather than only checking consistency.
• Diagnostic reasoning: inferring a cause from an observed effect, such as diagnosing faults or conditions from symptoms.
• Knowledge acquisition and scientific inquiry: generating hypotheses that go beyond what is stated in the available data.
• Everyday question answering: tasks where a plausible reason must be inferred from limited, incomplete information.

Industry relevance: The finding that few-shot prompting reliably improves deduction but not abduction is directly useful for practitioners who assume that more examples in a prompt will fix reasoning failures. It also warns that LLM outputs in hypothesis-generating roles are influenced by whether content matches common beliefs, which matters for any deployment where the model proposes explanations to users.

Future Directions

• Direct human-versus-LLM comparison on the abductive tasks, since the paper predicts humans should do better at abduction than deduction — a pattern opposite to what the LLMs showed.
• Richer abduction tasks, such as Inference to the Best Explanation, where the model must select the best hypothesis from multiple candidates that are each already logical explanations.
• Probabilistic and Bayesian treatments of abduction, to contrast with the authors' syllogistic formalization.
• More complex reasoning structures, including extended syllogisms and conditionals, as well as further investigation of the error tendencies and biases observed.

Target Audience

Researchers in NLP and LLM evaluation, cognitive scientists and psychologists who study syllogistic reasoning and belief bias, and logicians or philosophers of science interested in formalizing abduction. It is also useful for practitioners who deploy LLMs in explanation, diagnosis, or hypothesis-generation roles, since it shows where prompt engineering helps and where it does not.

Authors’ abstract

Research in AI using Large-Language Models (LLMs) is rapidly evolving, and the comparison of their performance with human reasoning has become a key concern. Prior studies have indicated that LLMs and humans share similar biases, such as dismissing logically valid inferences that contradict common beliefs. However, criticizing LLMs for these biases might be unfair, considering our reasoning not only involves formal deduction but also abduction, which draws tentative conclusions from limited information. Abduction can be regarded as the inverse form of syllogism in its basic structure, that is, a process of drawing a minor premise from a major premise and conclusion. This paper explores the accuracy of LLMs in abductive reasoning by converting a syllogistic dataset into one suitable for abduction. It aims to investigate whether the state-of-the-art LLMs exhibit biases in abduction and to identify potential areas for improvement, emphasizing the importance of contextualized reasoning beyond formal deduction. This investigation is vital for advancing the understanding and application of LLMs in complex reasoning tasks, offering insights into bridging the gap between machine and human cognition.