Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoning

AI summary

Overview

Research area: AI for science — specifically, benchmarking large language model (LLM) reasoning and multi-agent system design on graduate-level heliophysics (solar and space physics) problems.

Technical level: Intermediate. The paper assumes familiarity with LLM prompting and multi-agent architectures, but the core ideas are accessible; the physics content itself (Parker transport equations, pickup ions) is advanced but not required to follow the benchmark design.

Scope: The paper introduces Reasoning With a Star (RWS), a 158-item heliophysics question–answer benchmark built from NASA/UCAR Living With a Star summer school problem sets, plus an automatic grading system and a comparative study of four multi-agent reasoning patterns against a single-shot LLM baseline.

What This Paper Is About

LLMs are increasingly used as scientific assistants, but heliophysics — the study of how the Sun drives space weather and affects Earth and planetary environments — is almost absent from existing reasoning benchmarks. The authors argue that real scientific reasoning is not just fact recall: it requires carrying physical assumptions through a derivation, tracking units, and delivering answers in a specific scientific format. RWS provides a benchmark to measure exactly those abilities, along with a grader that can check them automatically, and it tests whether decomposing a problem across multiple specialist agents beats simply asking one model for the answer.

Key Contributions

1. A domain-grounded benchmark dataset. 158 heliophysics question–answer pairs, machine-converted from OCR'd Living With a Star summer school problem sets, cleaned manually, and normalized into a structured JSONL schema containing question, preamble, reasoning steps, final answer, answer type, format hints, and metadata.

2. An automatic, format-aware grading system. The grader combines unit-aware numerical tolerance checking, symbolic equivalence via a computer algebra system (SymPy), and schema validation, with a two-agent LLM verifier (a Parser and a Judge) as a fallback for paraphrased or algebraically equivalent-but-string-mismatched answers.

3. A comparative study of agentic coordination patterns. Five reasoning setups are evaluated under matched decoding conditions with no heliophysics-specific retrieval: a single-shot baseline plus four multi-agent designs (HMAW, PACE, PHASE, SCHEMA).

4. The STAR design template. A systems-engineering-inspired framework (Systems-engineering-of-Thoughts Agentic Reasoning) that treats an LLM's reasoning trace as an engineered pipeline with defined modules, interface contracts, and verification checkpoints — offered as a reusable architecture rather than a single fixed workflow.

Main Findings

• Single-shot prompting is a weak baseline on RWS. Gemini 2.5 Pro leads at 35.44% accuracy, followed by OpenAI OSS 20B and OSS 120B (both 32.91%), Meta Llama 3.3 (31.01%), and Mistral 24.11 (27.22%). Even the best direct-prompting result leaves most problems unsolved.

• Every multi-agent pattern beats single-shot on RWS, without any retrieval. RWS accuracy rises from 35.44% single-shot to 39.52% (HMAW), 41.92% (PACE), 42.51% (PHASE), and 44.31% (SCHEMA) — a gain of roughly 4 to 9 points from coordination alone.

• No single coordination pattern wins everywhere. PACE leads on GSM8K (93.41%) and MATH (81.51%); HMAW leads on GPQA (79.01%); SCHEMA leads on RWS (44.31%), HumanEval (43.29%), and SWE-bench Verified (63.23%); PHASE is consistently competitive but rarely best.

• Task structure, not agent count, determines the winning design. Compact self-critique loops (PACE) suffice for arithmetic-style problems, minimal hierarchies (HMAW) work for fact-heavy science QA, and requirement-tracking designs (SCHEMA) excel where success depends on satisfying explicit constraints, units, and output formats.

• SCHEMA's advantage tracks constraint-heavy tasks. The three benchmarks SCHEMA wins — HumanEval, SWE-bench Verified, and RWS — all require outputs that conform to precise specifications rather than merely looking plausible, which maps directly onto its assumption-tracking and interface-checking design.

• Complexity must be earned, not assumed. Adding more agents and stages does not automatically raise accuracy; PHASE's extra hypothesis stage sometimes increases the chance of deviating from the correct solution on narrowly defined problems.

Methodology in Plain English

The authors started with a set of graduate-level heliophysics homework problems written by domain experts and converted the scanned PDFs into machine-readable text using OCR. They then manually fixed scanning errors, misread symbols and units, and typos, and reorganized everything into a structured data format. Each problem was tagged by the kind of answer it expects: a number (often with units), a symbolic LaTeX expression or equation, or a short scientific statement. The dataset ends up with 68 textual items, 52 symbolic items, and 38 numeric items.

For grading, they built an automatic checker rather than relying on human reviewers. Numeric answers are accepted within a defined error tolerance and must carry correct units. Symbolic answers are compared for algebraic equivalence using a computer algebra system, so that an expression written differently but mathematically identical still passes. Textual answers are checked against required scientific claims. When the automatic checker flags a mismatch — often because a correct answer was simply phrased differently — a two-stage LLM verifier re-examines it: one agent normalizes both the prediction and the ground truth, and a second agent judges equivalence according to the answer type.

For the reasoning comparison, they tested a plain single-shot prompt, then four multi-agent pipelines that split the work across roles. HMAW passes a task down a fixed chain of CEO to Manager to Worker with no retries. PACE has an agent plan, an agent answer, a critic that can send the answer back once for a fix, and a final agent that formats the output. PHASE adds a hypothesis stage that lists assumptions and required units before solving. SCHEMA, the most elaborate, first drafts an architecture describing the question type and acceptance checks, assigns a small team of specialist experts, synthesizes their outputs, runs a guard check against requirements, and only then finalizes the answer. All systems ran under matched decoding settings with no retrieval of external heliophysics knowledge, so the benchmark isolates reasoning ability rather than memorized facts.

Why This Matters

Impact on research. RWS fills a genuine gap: heliophysics is critical to space weather forecasting and satellite operations, yet it has been essentially absent from LLM reasoning benchmarks. More broadly, the paper reframes agentic architecture design as an engineering discipline with explicit interfaces and verification checkpoints, rather than an exercise in stacking more agents onto a prompt. The finding that gains come from constraint enforcement — not from raw model scale — gives the field a concrete design principle to test against.

Real-world applications.

• Space weather forecasting: Models that reliably propagate units and state physical assumptions could assist in translating solar observations into operational forecasts for geomagnetic storms.
• Satellite and infrastructure protection: Better automated reasoning over heliophysics problems supports risk assessment for communications, navigation, and power grid systems vulnerable to solar activity.
• Scientific education and training: The dataset is drawn from a summer school curriculum, so it doubles as a tool for evaluating AI tutoring systems on graduate-level physics reasoning.
• Auditable AI for mission-critical science: Failure annotations for unit mismatches, unstated assumptions, and formatting violations point toward AI outputs that domain experts can verify rather than merely trust.

Industry relevance. The benchmark's constraint-satisfaction framing overlaps directly with software engineering evaluation (HumanEval, SWE-bench Verified), where SCHEMA also performed best. Organizations deploying agents for regulated or safety-critical workflows — aerospace, defense, energy, scientific computing — have a stake in the paper's central claim that structured verification beats unstructured scaling.

Future Directions

• Expand the dataset. The authors plan to add more heliophysics problem sets beyond the Living With a Star summer school material, widening coverage of the field.
• Richer failure annotation. Future versions aim to label not just wrong answers but why they are wrong — unit mismatches, unstated assumptions, formatting violations — to make model reasoning more auditable.
• Better output-format guidance. Improving the benchmark's instructions around expected answer formats could reduce grading ambiguity and sharpen the measurement of genuine reasoning versus format compliance.
• Adaptive or hybrid architectures. Since no single pattern dominates, an open question is whether a system could route a problem to the appropriate coordination strategy based on its detected structure, rather than committing to one pipeline in advance.
• Retrieval-augmented comparisons. This study deliberately excluded domain-specific RAG to isolate reasoning; testing RWS with retrieval enabled is a natural follow-up for understanding where knowledge access versus reasoning ability is the true bottleneck.

Target Audience

This paper is most useful to AI researchers working on agentic systems, reasoning benchmarks, and LLM evaluation methodology, particularly those interested in scientific domains. It also speaks to heliophysicists and space weather researchers curious about where current AI systems fail on their field's problems, and to engineers designing multi-agent pipelines for tasks where output correctness must be verifiable rather than merely plausible. Practitioners in safety-critical industries evaluating agent architectures for structured, constraint-heavy work will find the comparative results directly applicable.

Authors’ abstract

Scientific reasoning through Large Language Models in heliophysics involves more than just recalling facts: it requires incorporating physical assumptions, maintaining consistent units, and providing clear scientific formats through coordinated approaches. To address these challenges, we present Reasoning With a Star, a newly contributed heliophysics dataset applicable to reasoning; we also provide an initial benchmarking approach. Our data are constructed from National Aeronautics and Space Administration & University Corporation for Atmospheric Research Living With a Star summer school problem sets and compiled into a readily consumable question-and-answer structure with question contexts, reasoning steps, expected answer type, ground-truth targets, format hints, and metadata. A programmatic grader checks the predictions using unit-aware numerical tolerance, symbolic equivalence, and schema validation. We benchmark a single-shot baseline and four multi-agent patterns, finding that decomposing workflows through systems engineering principles outperforms direct prompting on problems requiring deductive reasoning rather than pure inductive recall.