ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems

AI summary

Overview

Research area: AI evaluation and benchmarking, specifically logical and symbolic reasoning in large language models applied to the scientific domain of chaotic dynamical systems.

Technical level: Intermediate. Readers need comfort with basic concepts from dynamical systems (chaos, Lyapunov exponents, attractors) and familiarity with first-order logic notation, but no advanced mathematics is required to follow the argument.

Scope: The paper introduces a 621-question benchmark grounded in a formal logical ontology to measure whether LLMs can reason consistently about chaotic and non-chaotic dynamical systems, rather than merely recall facts about them.

What This Paper Is About

Large language models handle fluent text well but break down on tasks requiring strict logical inference and consistency. Chaos theory is an unusually good stress test for this weakness because "chaos" is deterministic yet routinely confused with randomness, nonlinearity, or complexity, even by domain experts. The paper builds a benchmark that converts well-established implications in dynamical systems theory into explicit logical constraints, then measures whether LLMs can respect those constraints across single questions, multi-step chains, and multi-turn conversations.

Key Contributions

1. A curated benchmark of 621 questions over 30 dynamical systems. Systems span chaotic ordinary differential equations (Lorenz-63/84/96, Rössler, Chen, Hindmarsh–Rose neuron), non-chaotic ODEs (harmonic oscillator, Van der Pol, Brusselator, FitzHugh–Nagumo), discrete maps (logistic, Hénon, Ikeda, Baker, Arnold cat, standard map, circle map), partial differential equations (Kuramoto–Sivashinsky, Sine–Gordon), and stochastic processes (Ornstein–Uhlenbeck).

2. A unified first-order logic ontology. Eleven unary predicates (Chaotic, Deterministic, PosLyap, Sensitive, StrangeAttr, PointUnpredictable, StatPredictable, QuasiPeriodic, Random, FixedPointAttr, Periodic) plus a compact axiom system encoding one-way implications. Deliberately, no converse rules are included, so a model cannot infer Chaotic from Sensitive or PosLyap alone.

3. Seven reasoning task families. Atomic QA, multi-hop implication reasoning, cross-system analogies and non-analogies, counterfactual parameter reasoning, adversarial bias probes, multi-turn dialogues, and hard compositional questions. Six specific misconception biases are targeted, including chaos–randomness confusion, chaos–nonlinearity conflation, and PDE–chaos overgeneralisation.

4. Evaluation metrics and an open-source pipeline that separate local accuracy from global coherence. Beyond per-item accuracy, the benchmark scores dialogue-level accuracy (all turns correct), contradiction detection, implication consistency, and axiom-violation counts. Code and dataset are released publicly.

Main Findings

• Strong local performance, weak global coherence. All four evaluated models (GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, LLaMA-3 70B) reach 91–94% per-item accuracy, but dialogue-level accuracy drops to 53.1–75.5%. LLaMA-3 70B zero-shot achieves the highest dialogue coherence at 75.5%, while GPT-4 with chain-of-thought is lowest at 53.1%.

• Every model scores 0% on compositional items. Questions combining multiple constraints, such as chaining implications while rejecting a misleading analogy and staying consistent with earlier dialogue turns, defeat all tested systems completely. Models handle each component in isolation but fail to integrate them into one coherent decision.

• Chain-of-thought prompting actively hurts. GPT-4 loses 5.8 percentage points in accuracy and 16.3 points in dialogue accuracy under CoT; LLaMA-3 70B loses 2.1 and 10.2 points. Longer, more confident-looking reasoning traces introduce extra unsupported assumptions and amplify converse-reasoning errors.

• The dominant error is treating implications as equivalences. Because the axiom system only encodes one-way rules, models frequently infer Chaotic(s) from Sensitive(s) or PosLyap(s). The paper frames this not as a trick but as a genuine scientific reasoning failure, since many misconceptions arise exactly this way.

• Dialogue failures largely follow error compounding plus belief drift. The observed dialogue accuracy range roughly matches what independent per-turn error would predict (0.92³ ≈ 0.78 down to 0.92⁶ ≈ 0.61), but in practice errors cluster: models drift in how they interpret a system across turns rather than making isolated mistakes.

• Compounding math and clustered failures line up. The gap between per-item and dialogue scores is consistent with the strict all-turns-correct criterion, which collapses an entire dialogue to zero on a single incoherent turn.

Methodology in Plain English

The researchers encoded each of 30 dynamical systems in a structured JSON file containing equations, canonical parameter regimes, short natural-language summaries, and a truth assignment for each of the 11 predicates. They then wrote a set of global logical rules capturing standard relationships in dynamical systems theory, such as "if a system is chaotic, then it is deterministic" and "if a system is random, then it is not deterministic." Crucially, they wrote these rules in only one direction, mirroring how the science actually works.

Questions were generated from over 70 templates across seven task families. Each template carries explicit applicability constraints, so a question about randomness is only generated for systems where that is meaningful. Every generated item was manually checked for logical correctness against the rule set, clarity, and whether it genuinely tests the intended misconception. Ambiguous items and systems with disputed behaviour were removed. A final automated and manual filtering pass ensured every question is answerable from the system labels and rules alone, without any numerical simulation.

For evaluation, models were prompted zero-shot or with chain-of-thought at temperature 0. Responses were normalised with a pattern matcher that prefers the final explicit answer marker. Ground-truth answers for implication questions were derived by forward chaining to the least fixed point of the rule set, which also allows the pipeline to detect when a model's answers contradict each other or violate the axioms.

Why This Matters

Impact on research. The paper reframes LLM evaluation in scientific domains away from final-answer plausibility and toward logical consistency. By making an explicit, compact ontology the ground truth, it enables measurable, auditable diagnosis of how models fail, not just that they fail. It also provides a concrete testbed for neuro-symbolic methods, since the ontology is small enough to support lightweight constraint checking rather than full theorem proving.

Real-world applications:

• Scientific literature and modelling assistants. Tools that help researchers characterise dynamical systems (climate, fluid, neural, chemical) must not conflate chaos with randomness or assume chaos from any positive Lyapunov exponent.
• Simulation and forecasting pipelines. Systems that combine LLM front-ends with numerical solvers need the language layer to hold consistent beliefs about regime type across an extended session.
• Automated review and hypothesis triage. A model that drifts in its commitments mid-conversation produces mutually incompatible justifications, which is dangerous when used to filter or prioritise scientific claims.
• Educational tutoring in physics and applied mathematics. Chaos theory is a common source of student misconceptions; the benchmark's bias probes map directly onto errors a tutor should detect and correct.

Industry relevance. Any organisation deploying LLMs in scientific, engineering, or quantitative workflows faces the same failure profile: high per-item accuracy masking unstable global reasoning. The paper's metrics, particularly axiom-violation counts and dialogue-level accuracy, offer a template for auditing reliability in settings where a confidently wrong and internally inconsistent answer is worse than an admitted uncertainty. The released pipeline lowers the barrier to building domain-specific consistency benchmarks of the same shape.

Future Directions

1. Multiple parameter regimes per system. The current release assigns each named system a single canonical regime label. Extending to several regimes would test whether models can track state changes rather than memorising one label per system name.

2. Extracting regime information from equations or parameters. Present items in which the model must infer the regime from the equations or numerical parameter values, rather than receiving it as a labelled fact.

3. Hybrid symbolic–numeric items. Combine the logical constraints with tool-verified numerical diagnostics such as Lyapunov exponent estimation, which would test integration of symbolic reasoning with computation.

4. Repairing dialogue-level incoherence. Open questions include whether symbolic verifiers, logic-guided decoding, or fine-tuning with consistency losses across turns can close the gap between per-item accuracy and dialogue coherence, and whether these methods can move the compositional score above zero.

Target Audience

Researchers working on LLM reasoning evaluation, neuro-symbolic AI, and scientific machine learning will find the benchmark design and failure taxonomy directly useful. Practitioners building LLM-assisted tools for physics, applied mathematics, or quantitative modelling benefit from the concrete evidence that high benchmark accuracy can coexist with unstable beliefs. The paper is also accessible to graduate students in dynamical systems or AI evaluation who want a worked example of grounding a benchmark in formal logic rather than surface-level question answering.

Authors’ abstract

Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet often misinterpreted as randomness or complexity. We introduce ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic (FOL) ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. We define metrics for logical accuracy, implication consistency, dialogue coherence, and contradiction, and we release an open-source evaluation pipeline. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91-94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence. Dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zero-shot). ChaosBench-Logic provides a rigorous testbed for diagnosing such failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.