ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction

AI summary

Overview

• Research area: Evaluation of large language models (LLMs) on scientific reasoning, specifically whether models can recognize and structure fundamental inference paradigms (deduction, induction, abduction).
• Technical level: Intermediate. The task and metrics are conceptually accessible, but the paper assumes familiarity with chain-of-thought prompting, graph representations, and LLM-as-judge evaluation.
• Scope: The paper defines a new task (Latent Reasoning Chain Extraction), releases a 70-article benchmark (ARCHE Bench) with two new evaluation metrics, and benchmarks 10 leading LLMs on it.

What This Paper Is About

LLMs can produce text that looks like reasoning, but it is usually unstructured and informal, so it is unclear whether they actually grasp the basic reasoning paradigms used in science. ARCHE asks models to read the Introduction of a scientific paper and decompose its argument into a Reasoning Logic Tree (RLT), where every inference step is explicitly labeled as deduction, induction, or abduction. The goal is to test whether models can extract the hidden reasoning chain behind a scientific claim and ground each step in verifiable text.

Key Contributions

1. A new task definition. The authors define Latent Reasoning Chain Extraction (ARCHE), in which a model must decompose complex reasoning arguments into combinations of standard reasoning paradigms, represented as a Reasoning Logic Tree (RLT). The name draws on the Greek term arch, meaning principle, source, or cause, as formalized by Aristotle.
2. A new benchmark, ARCHE Bench. Built from 70 peer-reviewed, open-access Nature Communications articles published in 2025, balanced across Biological Sciences (35) and Physical Sciences (35). It contains 2,164 sentences from introductions, 1,891 cited references, 5,418 viewpoints from the articles themselves, 33,321 viewpoints from cited references, and 38,739 viewpoints combined.
3. Two logic-aware evaluation metrics. Entity Coverage (EC) measures content completeness by checking the proportion of a paper's core scientific entities that appear in the predicted RLT. Reasoning Edge Accuracy (REA) measures step-by-step logical validity using a multi-model voting judge.
4. A benchmark of 10 state-of-the-art LLMs. The authors run zero-shot evaluations across six model families and show that no model produces a complete and standard reasoning chain.

Main Findings

• No model is close to solving the task. The best models struggle to exceed an accuracy of 50% on the task, and the paper states that none of the 10 models can extract a complete and standard reasoning chain.
• Overall EC and REA averages. Across evaluations, average EC is 51.4% (median 66.7%, with a spread from 0% to 100%), while average REA is 28.3% (median 25%).
• A trade-off between completeness and correctness. In Table 1 (Overall), Claude-Opus-4 (Thinking) reaches the highest overall EC at 69.7% but only 24.2% REA, while Gemini-2.5-Pro (Thinking) reaches the highest overall REA at 41.4% with 54.1% EC. The authors describe a smooth boundary curve in the joint space of accuracy and coverage.
• Reasoning-optimized models do better. In Table 1 (Overall), o3 achieves 35.6% REA and 60.5% EC, and Grok-3 achieves 33.1% REA and 53.8% EC, both clearly above GPT-4o at 15.8% REA and 24.3% EC. Gemini-2.5-Pro and its thinking variant differ only slightly, with the thinking variant showing a small REA improvement (41.4% vs. 39.5% overall) but no substantial overall gain.
• Reasoning accuracy varies by inference type. In Table 3, which includes only reasoning steps with valid output formatting, Grok-3 leads on all three types: abductive 87.1%, deductive 74.0%, inductive 77.9%. GPT-4o reaches 63.4% on deductive reasoning, and Claude-Sonnet-4 reaches 62.0% abductive and 63.7% inductive. DeepSeek-R1 and o3 are weaker on deduction, each around 40%.
• Formatting failures are common. The paper notes that Table 3 accuracies are substantially higher than Table 1 because Table 3 only includes steps with valid output formatting. Despite prompt refinement and multiple rephrasings, formatting issues remain prevalent, which the authors attribute to models not grasping the structural constraints of the reasoning paradigms.
• Longer chains are not better. In Table 4, Grok-4 has the highest Average Total Steps (ATS 20.1) but an Average Effective Steps (AES) of only 4.9, the same AES as o3 with a much shorter ATS of 11.7. GPT-4o produces the lowest AES at 1.2. Gemini-2.5-Pro-thinking generates slightly more steps (ATS 13.2) than Gemini-2.5-Pro (ATS 12.4) but has a marginally lower AES (5.3 vs. 5.8). Even the best model extracts fewer than six valid inferences on average from introductions containing more than 30 sentences.
• Most initial generations need repair. In Table 5, the Stage 2 trigger rate (proportion of the 70 papers needing automated correction) ranges from 52.9% for DeepSeek-R1 to 82.9% for Claude-Opus-4. Lower rates indicate better initial generation quality.
• Judge reliability. The three-model voting system was evaluated against human annotations across the three reasoning types and achieved accuracy exceeding 88%, consistently outperforming individual models in alignment with human judgments.

Methodology in Plain English

The authors turn scientific argument structure into a graph-building exercise.

• Input. For each paper, a model receives the full Introduction plus viewpoints (ideas, arguments, or facts) extracted by GPT-4o from the Introduction itself and from the abstracts of papers cited in that Introduction. Cited abstracts are retrieved through the Semantic Scholar API. Viewpoints are extracted before any evaluated model sees the text.
• Output. The model must output a Reasoning Logic Tree (RLT) in DOT, a graph description language. Nodes hold a viewpoint plus its source, and directed labeled edges connect premises to conclusions.
• Edge types. Each inference uses one of six edge types that instantiate Peirce's three paradigms: Deduction-Rule (DR) and Deduction-Case (DC); Induction-Common (ICo) and Induction-Case (ICa); Abduction-Knowledge (AK) and Abduction-Phenomenon (AP). Each logical step should pair the appropriate premises with a shared conclusion node.
• Structural rules. An RLT must be a single-rooted directed acyclic graph in which all nodes converge on the central node, with no disconnected or irrelevant premises. Multi-hop reasoning requires intermediate nodes, and edges may cross document boundaries (for example, from a cited reference to the main paper).
• Two-stage generation. Stage 1 has the LLM generate an initial graph using a manually crafted prompt. Stage 2 runs a verifier script that checks for defects such as multiple roots, cycles, disconnected nodes, or invalid edge labels; if defects are found, the same LLM is re-prompted to fix them while preserving the reasoning content.
• Traceability. The pipeline uses a three-digit coordinate system (x, y, z) so every node maps to a whole sentence (x,0,0), a viewpoint within a sentence (x,y,0), a viewpoint from a cited reference (x,y,z), or a model-added sentence (0,0,0).
• Metrics. For EC, the o3 model extracts the paper's core idea and then typically 8-10 core scientific entities; coverage is the proportion of those entities appearing in nodes involved in correct reasoning steps, matched by case-insensitive string comparison on the original source text. For REA, each reasoning step is format-checked (invalid combinations, such as pairing a deduction case with an abduction-knowledge edge or using non-standard labels, are marked incorrect) and then judged by a three-model voting system using majority vote. Section 3.3 names the judges as o3, Claude-Sonnet-4-thinking, and Gemini 2.5 Pro; Appendix A names the panel as o3, GPT-4o, and Gemini.
• Experimental setup. Ten models from six families are evaluated under the same conditions with the same prompts, at a fixed temperature of 0.1 when configurable.

Why This Matters

The paper argues that fluent, reasoning-like natural language does not imply that a model has internalized reasoning paradigms. Because scientific discovery requires a transparent and grounded thought process that can be rigorously verified and supervised, this gap is a risk for LLM-driven scientific workflows.

Real-world applications:

• Literature review and evidence synthesis: Checking whether a model can correctly trace how cited work supports a new claim.
• Hypothesis generation and experimental design: Requiring models to state which inference type justifies a proposed hypothesis.
• Automated peer review or claim verification: Using EC and REA-style metrics to check whether a written argument covers the key concepts and whether each inference step is logically valid.
• AI for science pipelines: Providing structural constraints and reward signals so that model-generated scientific reasoning can be audited rather than taken on faith.

Industry relevance: the paper targets developers of reasoning models and AI-for-science tools. The authors advocate explicitly incorporating reasoning-paradigm-aligned data during pre-training and instruction, or augmenting reasoning supervision objectives with reward signals grounded in formal paradigms. The work was supported by Shanghai Artificial Intelligence Laboratory, and the code is released at github.com/Linsonng/ARCHEBenchmark.

Future Directions

• Scaling the corpus. The benchmark currently includes only 70 articles, and each ARCHE evaluation requires processing a full Introduction plus all cited abstracts, costing approximately $4 per paper across all 10 models. The authors plan to expand into multidisciplinary content such as chemistry and artificial intelligence to test cross-domain generalization.
• Covering more of the research context. The benchmark only uses the Introduction section. The authors note that excluding Methods and Results may underestimate an LLM's ability to reason across the full scientific workflow, and they plan to extend ARCHE to enable end-to-end evaluation of hypothesis generation, evidence collection, and conclusion formation.
• Fixing structural formatting failures. Formatting issues remained prevalent despite prompt refinement and rephrasing, which the authors say suggests the current setup approaches the expressive limits of these models. Whether better training data or reward design can close this gap is left open.
• Closing the EC/REA trade-off frontier. No model performs ideally on both metrics, and the best models align along a boundary curve. Whether any architecture can move into the preferable region of high coverage and high accuracy is an open question.

Target Audience

Researchers and engineers working on LLM reasoning evaluation, AI for science, and automated scientific claim verification. It is also useful for benchmark designers interested in logic-aware metrics that go beyond final-answer accuracy, and for anyone building systems that need auditable, paradigm-grounded reasoning rather than fluent-sounding text.

Authors’ abstract

Large language models (LLMs) are increasingly used in scientific domains. While they can produce reasoning-like content via methods such as chain-of-thought prompting, these outputs are typically unstructured and informal, obscuring whether models truly understand the fundamental reasoning paradigms that underpin scientific inference. To address this, we introduce a novel task named Latent Reasoning Chain Extraction (ARCHE), in which models must decompose complex reasoning arguments into combinations of standard reasoning paradigms in the form of a Reasoning Logic Tree (RLT). In RLT, all reasoning steps are explicitly categorized as one of three variants of Peirce's fundamental inference modes: deduction, induction, or abduction. To facilitate this task, we release ARCHE Bench, a new benchmark derived from 70 Nature Communications articles, including more than 1,900 references and 38,000 viewpoints. We propose two logic-aware evaluation metrics: Entity Coverage (EC) for content completeness and Reasoning Edge Accuracy (REA) for step-by-step logical validity. Evaluations on 10 leading LLMs on ARCHE Bench reveal that models exhibit a trade-off between REA and EC, and none are yet able to extract a complete and standard reasoning chain. These findings highlight a substantial gap between the abilities of current reasoning models and the rigor required for scientific argumentation.