Evaluating Inquiry Learning in LLM-Based Agents: Hypothesizing, Reasoning, and Exploration

AI summary

Overview

Research area: LLM-based agents, scientific discovery, and learning sciences (inquiry learning).

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with LLM prompting, agent loops (Action/Observation/Thought), and benchmark design.

Scope: An interactive benchmark and evaluation framework that tests whether seven LLMs can perform a full inquiry cycle — systematically exploring variables, forming and refining hypotheses, and reasoning quantitatively — to recover hidden multivariable mathematical expressions, with and without learning-sciences-inspired scaffolding.

What This Paper Is About

Prior work shows LLMs are good at isolated pieces of scientific inquiry — inductive reasoning, hypothesis generation, data analysis — but almost always tests these abilities separately on fixed datasets. This paper asks whether a model can actually chain them together: actively probing an unknown system, deciding what data to collect next, and revising hypotheses as evidence arrives. The authors build a controlled interactive environment where an agent sets values for input variables, observes a target output, and must eventually state the exact underlying equation.

Key Contributions

1. An interactive benchmark for inquiry-based mathematical modeling. The authors extend the Countdown task into an environment where variables (not fixed numbers) are chosen by the agent, which then infers the underlying expression composed of +, -, ·, ÷. It covers equations with 3 to 5 independent variables (174 distinct equations), plus a set extended with coefficients and intercepts and a set of 26 real-world scientific equations.
2. A Guided Inquiry Learning (GIL) paradigm. Two external components are added on top of the base inquiry loop: an IL guide that conditions generation on the agent's current phase of the inquiry cycle and injects scaffolding for exploration and reasoning, and a verifier that checks hypotheses against ground truth.
3. A systematic evaluation across models and prompting strategies. Seven instruction-following LLMs (Llama-3.1-8B-Instruct, vicuna-13b-v1.3, Qwen2.5-14B-Instruct, gemma-2-27b-it, Qwen2.5-32B-Instruct, Llama-3.3-70B-Instruct, GPT-4o) under zero-shot, one-shot, and three-shot prompting, with ablations on hypothesizing, reasoning, controlled exploration, and verification (AO, AOT, AOT-control, IL-ver).
4. A behavioral rubric for trajectories. An annotation scheme covering the three inquiry components — exploring (strategic vs. non-strategic), hypothesizing (repeated hypotheses, numerical modification, term modification), and reasoning (qualitative + quantitative, qualitative only, pure statement) — applied to 10,926 annotated steps, with inter-annotator Cohen's Kappa of 0.94 on 11 sampled trajectories.

Main Findings

• Inherent inquiry ability is weak. In the IL setting, all open-source models scored at most 12.6% correct in zero-shot; in-context examples helped every model except Llama-8B, with Llama-70B reaching 24.1%. The authors describe the overall success rates in discovering underlying expressions as low.
• Structured guidance is the biggest lever. GIL improved accuracy over IL for most models and prompting strategies (exceptions: Qwen-14B in 1-shot and Vicuna-13B in 0-shot). In the zero-shot setting the paper reports increases from 3.4% (GPT-4o) to 12.1% (Llama-70B). Llama-70B was best in the 0- and 1-shot GIL settings (24.7% and 25.3%) and matched Qwen-32B in the 3-shot case.
• Complexity hurts sharply. In IL 3-shot, going from 3 to 5 independent variables dropped accuracy from 34.5% to 12.1% (Qwen-32B), 34.5% to 10.3% (Llama-70B), and 31% to 13.8% (GPT-4o). GIL softened this: 44.8% (Qwen-32B) and 43.1% (Llama-70B) in the 3-shot, three-variable setting.
• Reasoning steps matter; explicit hypothesizing matters less. Removing the Thought step decreased most models' performance, consistently so for Qwen-14B, Qwen-32B, Llama-70B, and GPT-4o. Only Qwen-32B consistently benefited from explicit hypothesizing across all shot settings. Controlling the exploration strategy (AOT-control) consistently improved both Qwen models, Gemma-27B, Llama-70B, and GPT-4o.
• Verification alone is not enough. Adding the external verifier to the IL pipeline (IL-ver) consistently helped only Qwen-14B, Qwen-32B, and GPT-4o.
• Exploration is rarely strategic without scaffolding. In 0-shot IL, only 21.63% of exploration steps were strategic (one variable changed at a time); under GIL this rose to 91.64% (0-shot) and 85.21% (3-shot).
• Hypotheses repeat and drift toward prior beliefs. Repeated hypotheses occurred in up to 44% of trajectories in 0-shot IL with Qwen-32B. Qwen-14B and Qwen-32B tended to refine by adding numerical factors, while Llama-70B added new variable terms — the authors read this as a confirmation-bias-like pattern.
• Reasoning guidance produces causal statements. The proportion of Thought steps containing explicit cause–effect relations was at most 0.44 in IL but 1.00 in all GIL conditions, and reasoning guidance increased the share capturing both direction and magnitude of effects.
• Domain priors help. In an appendix study on 26 real-world scientific equations (zero-shot AOT), all models did best when variables were given as acronyms with full names in context, with Llama-70B and Qwen-14B reaching 100%.

Methodology in Plain English

The agent faces a black box with three to five input variables and one output. It can "set" variable values (an Action), receive the resulting output (an Observation), write down internal reasoning (a Thought), or state a candidate equation (a Hypothesis), and it ends by issuing a finish action. Each episode has a step budget: in IL up to 1 + 6n set actions (effectively permitting up to six changes per variable); in GIL up to 1 + 3n data points in the first exploration phase, then n set actions per phase, over at most three inquiry cycles. Correctness is judged by exact match of normalized equations.

Two pipelines are compared. The IL pipeline simply lets the model generate steps freely, parsing out whatever thoughts and hypotheses it emits before its next action. The GIL pipeline removes that freedom: the environment tracks which phase of the inquiry cycle the agent is in and prompts accordingly — iteratively act during exploration, reason after observations, hypothesize once enough data is collected — and additionally instructs the agent to change one variable at a time (the "controlling variables" strategy) and to relate changes in the output to changes in the inputs. Hypotheses are then checked by a verifier, and a correct one ends the episode.

To keep the task honest about exploration rather than recall, variable names are abstract (A, B, C). A separate condition replaces them with real scientific variable names to test whether domain knowledge changes behavior. Two researchers annotated trajectories against a rubric covering exploration, hypothesizing, and reasoning, first agreeing at Kappa 0.94 on 11 trajectories, then annotating the full sample of 10,926 steps drawn from the best-performing models per size tier.

Why This Matters

The paper reframes LLM evaluation from "can a model produce a correct answer" to "can a model run an experiment" — which is the capability that matters when models are handed instruments rather than datasets. It also imports scaffolding techniques designed for human students, showing they transfer to agents, and it identifies failure modes (simultaneous variable changes, early commitment to a first hypothesis, qualitative-but-not-quantitative reasoning) that mirror documented difficulties in human learners.

Real-world applications:

• Automated scientific discovery and symbolic regression, where models propose and test candidate laws against experimental data.
• AI-assisted data analysis and causal discovery, e.g., deciding which interventions or A/B tests to run next in a business analytics pipeline.
• Education technology, where the same inquiry-cycle scaffolding is used to tutor students or to build pedagogical agents that model good experimental practice.
• Agent design for lab automation and simulation control, where the agent must design informative experiments rather than exhaustively sweep parameters.

Industry relevance: teams deploying "agentic" systems that plan and act in unknown environments get a concrete measure of whether a model explores efficiently, plus evidence that prompt-level scaffolding can substitute for model scale in some settings — with the caveat that absolute success rates remain low and that verification modules alone do not fix the underlying weakness.

Future Directions

• Broaden model coverage. The authors note they did not extend the analysis to more LLMs of different model families, and call for IL analysis across more models and prompting strategies beyond the annotated subsample.
• Reintroduce tools. Reasoning was deliberately isolated from external tool use; studying the interplay between tool use and reasoning in more complex or computationally demanding scenarios is left open.
• Expand the guidance taxonomy. The work focuses on exploration, cause–effect reasoning, and structured scaffolding; other forms of guidance remain untested.
• Close the remaining GIL gap. Persistent failures in iterative hypothesizing and identifying cause–effect relationships are framed as a significant and challenging direction, stemming from inherent LLM limitations rather than scaffold design.

Target Audience

Researchers in LLM agent evaluation and AI for scientific discovery; machine learning practitioners building agents that must plan experiments or analyze unfamiliar systems; and learning-sciences or educational-technology researchers interested in whether inquiry-learning scaffolds transfer from human students to models. It is also useful for benchmark designers looking for a template that scores process as well as outcome.

Authors’ abstract

Scientific inquiry involves iterative cycles of exploration, hypothesis generation, and reasoning over observations to uncover relationships in complex systems. Large language models (LLMs) have demonstrated strong capabilities in several components of this process, including inductive reasoning, hypothesis generation, and data analysis. However, prior work typically evaluates these abilities in isolation, leaving open how models integrate them across a full inquiry cycle. In this work, we introduce a framework for evaluating LLMs' inquiry capabilities in a unified, interactive setting. We extend the Countdown task to a mathematical environment where an agent probes the system by manipulating variables, collecting observations, and inferring relationships between them. This setup enables analysis of exploration strategies, causal discovery, and hypothesis refinement during inquiry. Across seven LLMs and multiple prompting strategies, our experiments reveal limited inherent inquiry capabilities and low success rates in discovering the underlying expressions. However, structured guidance inspired by educational scaffolding significantly improves performance. Our findings highlight current limitations of LLMs in strategic exploration and iterative hypothesis refinement, and show how simple guidance strategies developed for human learners in inquiry learning can support more effective inquiry behavior.