Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models

AI summary

Overview

Research area: Agentic large language models (LLMs), benchmark design, and autonomous data science.

Technical level: Advanced. The paper assumes familiarity with agentic LLM scaffolds (ReAct-style tool use), benchmark/evaluation design, test-time scaling, and reinforcement-learning-era training pipelines for models such as the Qwen, GPT, Gemini, Claude, and Kimi families.

Scope: The paper formalises an open-ended "Deep Data Research" setting in which LLMs explore a raw structured database without a question, proposes the checklist-based DDR-Bench benchmark over three real-world databases, and empirically analyses the agency behaviour of current frontier and open-source models.

What This Paper Is About

Most evaluations of agentic LLMs assume the task or research question is already given, so they measure how well a model executes a predefined goal rather than whether it can decide what is worth investigating. The authors call these two capacities executional intelligence and investigatory intelligence, and argue that data science is the natural testbed for the latter because human analysts start from raw data, not from questions. The goal is to build an open-ended, verifiable benchmark — DDR-Bench — that measures whether models can autonomously explore large databases, form and test hypotheses, decide when to stop, and report insights that hold up factually.

Key Contributions

1. Formalisation of Deep Data Research (DDR). An open-ended agentic setting where a model receives only a database, a generic toolset (SQL and Python), a minimal start prompt naming the task entity, and no predefined question, objective, or limit on interaction rounds. The model must autonomously explore, generate and validate hypotheses, self-terminate, and produce insights.

2. DDR-Bench, described as the first large-scale benchmark for DDR. Across three scenarios it contains 291 task entities, 40 tables, 6,372 fields, over 203M records, and 2,058 verified checklist items. Evaluation is drawn from the unstructured components of the databases via a checklist, with more than fifty domain experts screening that every checklist fact is reasonably explorable and inferable from the corresponding data.

3. Systematic benchmarking of mainstream LLMs. Proprietary and open-source models are scored on four accuracy variants (sample-averaged and item-averaged, for message-wise insights and trajectory-wise insights), plus a novelty analysis of insights not covered by the checklist.

4. An analysis of "investigatory dynamics" and module design. The paper studies interaction, token, and cost scaling; exploration breadth and depth via database coverage and normalised exploration entropy; self-termination probability; and how training choices and agent modules (for example, reasoning budget) shape behaviour.

Main Findings

• Frontier models are far from saturated. Only Claude 4.5 Sonnet achieves an average accuracy above 40% (47.73 overall average in the reported table). Top open-source models including GLM, Kimi, and MiniMax approach proprietary LLM performance, while models such as Llama3.3-70B fall to 12.30 overall average.

• Message-wise and trajectory-wise insights diverge. Accuracy on message-wise insights (per-round interpretation) and trajectory-wise insights (a final report synthesising the whole history) does not necessarily align within a model, since the latter requires integrating global context and higher-level reasoning.

• Novelty tracks accuracy. Using GPT-5-mini for anonymised, order-swapped pairwise comparisons of useful novel insights, aggregated with the Bradley-Terry model, the novelty ranking correlates highly with the checklist-accuracy ranking in all three scenarios. Differences are small, especially among top-performing models, and the number or length of novel insights shows no systematic relation to those differences. The authors read this as evidence that checklist evaluation captures the dominant insight signal rather than systematically undervaluing models.

• Interaction scaling is sigmoid-shaped, but with different timing and ceilings. Higher-ceiling models such as Claude, GLM, and DeepSeek often delay the rapid-improvement phase. The authors attribute this to implicit planning — no explicit plan is stated, and the benchmark has no planning prompts, yet the sequence of reasoning steps and function calls unfolds as if plan-guided.

• Token scaling is not sigmoid. Scaling by costed tokens shifts curves to initially flat and then sharply increasing without clear saturation, indicating disproportionately high value in late-stage tokens. Token consumption is dominated by environmental feedback (tool execution results), so the most valuable late-stage tokens are few, reflecting a shift from breadth-oriented to depth-first exploration.

• Cost scaling shows exponential trade-offs. Claude is reported as the most expensive but consistently highest-performing, while DeepSeek shows particularly strong cost-effectiveness. In the 10-K scenario, at comparable performance, Gemini is an order of magnitude cheaper than GPT, and DeepSeek an order of magnitude cheaper than GLM.

• Exploration patterns are consistent, and balance wins. Advanced models such as Claude and GPT operate in a balanced regime combining adequate database coverage with focused access, with low variance; GLM shows high variance across trajectories. Weaker models (for example Qwen and Gemini) access a limited subset of fields while showing high entropy, meaning insufficient breadth without focused, selective access. Balanced instances are consistently associated with higher checklist accuracy. GLOBEM is omitted from this analysis because all models almost always access all fields.

• Self-termination behaviour differs by model generation. Collecting Qwen-family trajectories and measuring termination-token probability, Qwen3 and Qwen3-Next show a consistently increasing probability of terminating as information accumulates, whereas the Qwen2.5 series shows pronounced fluctuation and remains uncertain. Qwen3-Next maintains higher confidence with lower variance.

• Scaling alone does not buy agency. Within the Qwen family, even a tenfold increase in parameters yields less than a 3% gain in final accuracy, and smaller models sometimes outperform larger counterparts. Extending the context window (Qwen2.5 7B and 14B long-context variants) does not consistently improve performance or systematically increase interaction rounds. Newer generations, Qwen3 and Qwen3-Next with activated parameter sizes of 4B and 3B, increase exploration rounds on MIMIC and 10-K while GLOBEM rounds stay largely unchanged but reach a higher performance ceiling — pointing to agentic-first training rather than scale.

• Agent modules mostly reshape interaction patterns. Raising the reasoning budget of Qwen3-Next-80B-A3B increased reasoning tokens per turn from 1.20 to 357.78 on 10-K, from 15.26 to 397.11 on GLOBEM, and from 249.75 to 417.65 on MIMIC, while interactive turns fell from 27.93 to 11.89 on 10-K, from 12.93 to 9.41 on GLOBEM, and from 14.56 to 11.49 on MIMIC. Final performance fluctuated significantly, indicating a trade-off between reasoning depth and interaction frequency.

• Hallucination is not confounding the results. The authors state that cases where models interact poorly with the database yet produce correct insights from memorised knowledge are extremely rare, and that hallucination ratio and evaluation accuracy are not statistically correlated.

Methodology in Plain English

The researchers built a benchmark rather than a dataset of question-answer pairs. For each task entity — a patient, a wearable-sensor user, or a company — a model is told only which entity to investigate, given a simple ReAct-style system prompt with no workflow, memory, or planning modules, and handed two tools (SQL and Python) through the Model Context Protocol. There is no cap on how many rounds it may take, and the model itself decides when it has learned enough.

To make such unconstrained output scoreable, the authors extract verifiable factual statements from the unstructured text side of each database using GPT-5 mini and assemble them into checklists. Domain experts screen the checklists so that every item can be supported by analysing some subset of the data. At evaluation time, GPT-5-mini judges whether the model's collected insights support each open-ended fact (MIMIC, 10-K), or answers closed-form checklist questions using the insights as context (GLOBEM). Accuracy is the proportion of supported items, reported both per entity and per checklist item.

The three scenarios are deliberately different: MIMIC-IV (deidentified EHR data from Beth Israel Deaconess Medical Centre, with 100 patients sampled with stratification by note count) requires reasoning across tables to reconstruct care trajectories; GLOBEM (a Sport and Exercise Psychology database with wearable signals and surveys, 91 users retained after sampling and filtering) requires temporal and cross-domain reasoning about behaviour and mental state; 10-K (built from US public company annual reports via the SEC API, 100 companies each with more than 5,800 distinct financial facts) requires building financial models through exploration. Data are anonymised by rewriting column names and metadata without changing meaning, and because no questions are posed during exploration, the benchmark separates trajectory generation from evaluation to resist contamination and overfitting.

Why This Matters

Impact on research. The paper reframes agent evaluation around autonomy — deciding what to investigate — rather than task completion, and supplies an objective, checklist-based alternative to subjective LLM-as-a-Judge rubrics for open-ended analysis. It also provides evidence that scale, context length, and scaffold complexity are not the decisive factors for agentic data research, which redirects attention toward training strategies and the internal coordination between reasoning and exploration.

Real-world applications:

• Clinical data analysis: reconstructing longitudinal patient trajectories from EHR databases such as MIMIC-IV without a clinician framing the question first.
• Personal health and wellbeing: linking wearable signals (Bluetooth, walking activity, Wi-Fi, sleep) to psychological and social states in the GLOBEM setting.
• Financial due diligence: autonomously interrogating XBRL financial statements and filing text to characterise a firm's economic condition.
• General exploratory analytics: letting an agent surface irregularities in a corporate database before anyone knows what to ask.

Industry relevance. The benchmark gives a comparable signal for selecting models for long-horizon data agents, and the cost-scaling analysis speaks directly to deployment economics, where intelligence gains frequently entail exponential cost increases and where efficient models can shift the cost curve leftward by multiple orders of magnitude. The finding that minimal scaffolds suffice for strong models also argues against relying on elaborate framework engineering for gains that could come from the model itself.

Future Directions

• Agentic-first training. The authors argue meaningful agency requires a systematic emphasis on reasoning and agentic ability across pre-training and post-training, not parameter or context scaling. Designing and validating such pipelines is an open problem.

• Adaptive reasoning and interaction. The reasoning-budget experiment shows a trade-off where neither extreme is optimal and information loss occurs when shifting between more reasoning and more interaction. Treating these as dynamically adaptive model capabilities rather than fixed scaffold settings is proposed but not solved.

• Robust open-ended evaluation. Checklists cannot exhaustively enumerate valid insights, and the paper analyses rather than eliminates false-positive novel insights. Extending objective verification coverage without prohibitive manual annotation remains open.

• Termination and global exploration policy. The self-termination and exploration-pattern analyses show models fail by stopping prematurely, exploring too narrowly, or spreading attention too diffusely. How to instil a globally stable exploration policy that reliably identifies information gaps and adapts to uncertainty is left as the central challenge.

• Beyond the three domains. DDR-Bench covers MIMIC, GLOBEM, and 10-K; whether the observed exploration regimes and scaling behaviour generalise to other structured domains is not established here.

Target Audience

Researchers working on agentic LLMs, long-horizon tool use, and test-time scaling; benchmark and evaluation designers interested in replacing subjective judging with verifiable checklists; model trainers interested in the evidence that agency comes from training strategy rather than scale; and applied teams in healthcare, wearable health, and finance who are considering LLM agents for exploratory analysis of their own structured databases.

Authors’ abstract

The agency expected of Agentic Large Language Models goes beyond answering correctly, requiring autonomy to set goals and decide what to explore. We term this investigatory intelligence, distinguishing it from executional intelligence, which merely completes assigned tasks. Data Science provides a natural testbed, as real-world analysis starts from raw data rather than explicit queries, yet few benchmarks focus on it. To address this, we introduce Deep Data Research (DDR), an open-ended task where LLMs autonomously extract key insights from databases, and DDR-Bench, a large-scale, checklist-based benchmark that enables verifiable evaluation. Results show that while frontier models display emerging agency, long-horizon exploration remains challenging. Our analysis highlights that effective investigatory intelligence depends not only on agent scaffolding or merely scaling, but also on intrinsic strategies of agentic models.