DeepAnalyze: Agentic Large Language Models for Autonomous Data Science

AI summary

Overview

Research area: Autonomous data science and agentic large language models (LLMs) — training a model that can plan, interact with, and analyze structured data end to end.

Technical level: Advanced. The paper assumes familiarity with LLM fine-tuning, reinforcement learning (specifically group relative policy optimization), agent trajectories, and data science benchmarks.

Scope: The paper introduces DeepAnalyze-8B, an 8B-parameter agentic LLM trained to autonomously carry out the full data science pipeline — from raw data sources to analyst-grade research reports — and evaluates it on 12 data science benchmarks.

What This Paper Is About

Most existing "data agents" rely on hand-designed workflows that repeatedly prompt large closed-source LLMs to solve individual data tasks. These systems cannot fully automate data science, because they depend on predefined procedures and cannot adaptively explore real data environments. The paper's goal is to replace prompt-based workflows with a single trainable agentic model that learns, through curriculum-based agentic training, to autonomously orchestrate and iteratively refine its own actions.

Key Contributions

1. Agentic model: DeepAnalyze, described as the first agentic LLM tailored for autonomous data science, with two capabilities the authors identify as indispensable — autonomous orchestration and adaptive optimization. It is positioned as a foundation model that can be used directly or further customized via prompting or supervised fine-tuning.
2. Agentic training paradigm: A curriculum-based agentic training scheme paired with a data-grounded trajectory synthesis framework, designed to overcome two named obstacles — reward sparsity and trajectory scarcity.
3. Open training data: DataScience-Instruct-500K, a roughly 500K-sample dataset (about 470K single-ability samples, 20K cold-start samples, and 15K RL samples), released alongside the model and code.
4. Strong reported performance at small scale: Results on 12 benchmarks where the 8B model is reported to surpass most advanced proprietary LLMs, and to be the first agentic model capable of open-ended data research and analyst-grade report generation.

Main Findings

• End-to-end pipeline (DataSciBench): DeepAnalyze-8B ranks second only to GPT-4o among 17 compared open-source and proprietary LLMs, outperforming GPT-4-Turbo, GPT-4o-mini, and Claude-3.5-Sonnet. It is noted as the only entry here that achieves high-quality end-to-end pipelines without an external orchestration framework such as ReAct.
• Data analysis (DSBench): DeepAnalyze-8B outperforms previous LLM prompting and workflow-based agents, including Code Interpreter, Master-Slave, and Blackboard.
• Data modeling (DSBench): DeepAnalyze-8B reaches a 90.63% success rate with a performance score of 39.41 and a cost of $0.00, described as comparable to AutoGen-based workflows built on proprietary LLMs (AutoGen with GPT-4 is listed at 87.84% success, 45.52 performance, $19.34 cost).
• Multi-step analysis on harder tasks (DABStep): DeepAnalyze-8B scores 70.83 on easy-level (72 cases) and 32.80 on hard-level (378 cases), for 38.88 overall on 450 cases — above I2I-Agent with Claude-3.5-Sonnet (36.44) and Open Data Scientist with Deepseek-v3 (27.33).
• Table question answering: DeepAnalyze-8B reaches an average of 64.47 across WikiTQ (83.24), HybridQA (42.95), MultiHiertt (48.29), OTT-QA (64.73), FinQA (63.30), TAT-QA (70.64), and HiTab (78.16) — above DeepSeek-R1-0528 (60.22), GPT-4o (58.96), and Claude (58.79).
• Code generation (DS-1000): DeepAnalyze-8B scores 61.7 overall, versus 54.8 for the single-ability stage, 30.4 for the DeepSeek-R1-0528-Qwen3-8B base model, 53.9 for GPT-4-turbo, 40.2 for Kimi-K2-Instruct, 53.2 for GLM-4.5, and 54.8 for LIMI.
• Deep research (DABStep-Research): DeepAnalyze-8B is reported to consistently outperform all compared systems across every task category. Agent systems built on proprietary LLMs with tool calls reportedly drop sharply on open-ended data research relative to more directed tasks such as data preparation, analysis, and insight.
• Ablation on the Understand action: Removing the Understand action reduces WikiTQ from 83.24 to 80.78, MultiHiertt from 48.29 to 45.43, DS-1000 from 61.70 to 61.20, and DABStep from 38.88 to 31.78.

Methodology in Plain English

DeepAnalyze-8B is built on the DeepSeek-R1-0528-Qwen3-8B foundation model. Instead of being handed a workflow, the model is given five special actions it can generate itself: Analyze (planning, reasoning, reflection, self-verification), Understand (reading a data source such as a database, table, or document), Code (writing Python), Execute (running the code and capturing environment feedback), and Answer (producing the final output). At inference, whenever the model emits a Code block, the code is run in the environment and the feedback is inserted back as an Execute block, and the loop repeats until an Answer appears. The vocabulary of the base model is extended to support these tokens.

Training happens in two curriculum stages that mirror how human data scientists learn. Stage one is single-ability fine-tuning, strengthening reasoning, structured data understanding, and code generation separately using reasoning trajectories plus 100K general reasoning samples from AM-DeepSeek-R1-0528-Distilled. Stage two is multi-ability agentic training: a cold start on synthesized interaction trajectories, then reinforcement learning with GRPO in real-world environments.

Because high-quality trajectories for data science are scarce, the authors synthesize them in two ways. Reasoning trajectories are distilled from advanced teacher LLMs, verified against ground-truth responses, reformulated into Analyze and Understand components, and refined with a keyword-guided step that inserts reasoning keywords such as "but" and "wait". Interaction trajectories are generated by a multi-agent system with three roles — questioner (invents a data science problem and a checklist of constraints), solver (interacts with the environment using the five actions), and inspector (validates the trajectory against the checklist and environmental changes). These are grounded in structured data sources from NL2SQL datasets including Spider and BIRD.

Rewards are hybrid. Wrong output format yields R = -1. For question answering and data tasks with reference answers, R = 1/2 × (correctness + interaction-trajectory quality). For open-ended research, R = 1/3 × (report score + min(|T|/10, 1) + fraction of successful interaction turns), where the report score covers usefulness, richness, soundness, interpretability, and readability. Training uses ms-swift and SkyRL, inference uses vLLM, and everything runs on NVIDIA A800 GPUs. Training sequence length is 8K in stage one and 32K in stage two.

Why This Matters

The paper argues that prompt-engineered agent workflows, however strong the underlying proprietary model, cannot reach fully autonomous data science because they are not trained in interactive environments. DeepAnalyze shows a path where a small open model internalizes orchestration and self-correction, which changes the economics and reproducibility of data agents: the reported cost on DSBench data modeling is $0.00 versus $19.34 for AutoGen with GPT-4 and $38.81 for Code Interpreter with GPT-4.

Real-world applications suggested by the work:

• Automated analytics pipelines: Turning raw files or databases into prepared, analyzed, modeled, and visualized results without a human specifying each step.
• Analyst-grade report generation: Producing deep research reports from structured data sources, including open-ended research with no prescribed procedure.
• Natural-language data question answering: Querying tables and databases across formats such as CSV, Markdown, JSON, and XLSX-like sources.
• Data science code assistance: Generating and executing Python across libraries such as Pandas, NumPy, Matplotlib, Scikit-learn, SciPy, TensorFlow, and PyTorch, with iterative debugging from execution feedback.

Industry relevance: The model, code, and training data are open-sourced, which lowers the barrier for organizations that cannot route every data task through expensive proprietary APIs. The 8B size also makes deployment on modest hardware plausible, and the $0.00 cost figures reported for DSBench reinforce that claim.

Future Directions

• Extending beyond the current data scope: The architecture is motivated by structured data in external files; how far the same action-based design transfers to broader modalities is an open question not addressed here.
• Scaling the training recipe: All reported results use one 8B base model (DeepSeek-R1-0528-Qwen3-8B); the paper does not report how the curriculum and reward design behave at other model scales.
• Improving the harder regime: Easy-level DABStep performance (70.83) is far above hard-level (32.80), and on DataSciBench only GPT-4o is reported above DeepAnalyze-8B — narrowing the remaining gap to the strongest proprietary model is a concrete next step.
• Strengthening research-report evaluation: The open-ended research reward depends on an LLM-as-a-judge report score across five aspects, and DABStep-Research was constructed by the authors; independent and more standardized evaluation of data-oriented deep research remains an open problem.

Target Audience

Researchers and engineers working on LLM agents, autonomous data science, and reinforcement learning for tool-using models will get the most from this paper. It also suits data platform and analytics teams evaluating whether an open 8B agent can replace workflow-based orchestration, and practitioners interested in the released DataScience-Instruct-500K dataset for training their own data science models. Readers without background in RL-based agent training or benchmark methodology will find parts of Section 3 and the results tables demanding.

Authors’ abstract

Autonomous data science, from raw data sources to analyst-grade deep research reports, has been a long-standing challenge, and is now becoming feasible with the emergence of powerful large language models (LLMs). Recent workflow-based data agents have shown promising results on specific data tasks but remain fundamentally limited in achieving fully autonomous data science due to their reliance on predefined workflows. In this paper, we introduce DeepAnalyze-8B, the first agentic LLM designed for autonomous data science, capable of automatically completing the end-toend pipeline from data sources to analyst-grade deep research reports. To tackle high-complexity data science tasks, we propose a curriculum-based agentic training paradigm that emulates the learning trajectory of human data scientists, enabling LLMs to progressively acquire and integrate multiple capabilities in real-world environments. We also introduce a data-grounded trajectory synthesis framework that constructs high-quality training data. Through agentic training, DeepAnalyze learns to perform a broad spectrum of data tasks, ranging from data question answering and specialized analytical tasks to open-ended data research. Experiments demonstrate that, with only 8B parameters, DeepAnalyze outperforms previous workflow-based agents built on most advanced proprietary LLMs. The model, code, and training data of DeepAnalyze are open-sourced, paving the way toward autonomous data science.