BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence

AI summary

Overview

Research area: Large language model (LLM) agents for data management, specifically the automation of business intelligence (BI) workflows; the paper spans benchmark construction, agent/tool system design, and domain-specific post-training of LLMs.

Technical level: Advanced. The paper assumes familiarity with LLM agents, tool-calling, supervised fine-tuning, reinforcement learning, and data management concepts such as star/snowflake/constellation schemas, joins, and table-reshaping operators.

Scope: A 1-sentence scope: the paper builds the first end-to-end BI benchmark from real Power BI projects, shows that frontier LLMs fail on over 50% of its queries, and proposes a tool-augmented agent plus a trajectory-synthesis post-training framework that substantially improves accuracy.

What This Paper Is About

Business intelligence tools such as Power BI and Tableau require users to first find relevant tables, transform messy raw data (including reshaping non-relational tables), and define join relationships before they can answer a business question. This paper asks whether LLMs can do all of those steps automatically, end-to-end, from raw tables and a natural-language question to the correct result table. To study this, the authors build a benchmark of real business questions and their ground-truth answers, then build an agent that combines data-management algorithms as tools with LLM post-training.

Key Contributions

1. BI-Bench, the first end-to-end BI benchmark. The authors crawled over 3K real Power BI project files (.pbix) and manually curated 100 verified (business question, ground-truth answer table) pairs from real user dashboards, spanning domain types including sales, finance, and education. Construction took over 400 person-hours, or over 4 person-hours per query.

2. Extensive evaluation of LLMs on BI-Bench. Using 24 models and systems, the authors show that frontier models fail on over 50% of queries using SQL, and that state-of-the-art NL2SQL systems that score above 70% on their own benchmarks achieve only 6.0–17.3% and 23.8–26.3% SQL accuracy on BI-Bench.

3. BI-Agent, a tool-augmented agent. The system exposes a generic coding tool plus three specialized data-management tools — transform, join, and search — inside an LLM reasoning and tool-call loop. The authors frame the novelty as system-level co-design of LLM orchestration and domain-informed tools, rather than the individual algorithms themselves.

4. A post-training framework with BI-specific trajectory synthesis. The framework automatically synthesizes training trajectories from real BI projects, enabling the backbone model to be post-trained with supervised fine-tuning (SFT) and reinforcement learning (RL). The authors report that combining tool use with domain-specific post-training produces synergistic gains on end-to-end BI, and that gains also transfer to out-of-domain data analysis tasks such as Spider 2.0.

Main Findings

• Frontier LLMs perform poorly on end-to-end BI. Even frontier LLMs score less than 50% accuracy; the best-performing model reported, o4-mini, achieves only 48.2% accuracy using SQL, and frequently requires many rounds of interaction.

• The benchmark's hardest case requires reasoning over 38 tables. All 38 tables are required to answer that query, making table selection, transformation, and join individually challenging.

• NL2SQL leaders do not transfer. The top four open-source models on the BIRD leaderboard and the top two open-source agents on the Spider 2.0-lite leaderboard all score above 70% on their respective benchmarks, yet achieve only 6.0–17.3% and 23.8–26.3% SQL accuracy on BI-Bench respectively.

• BI-Bench projects are large and complex. On average each project contains 10.6 tables (maximum 52), 80,464 rows (maximum over 7M), and 10.7 columns per table (maximum 325), with on average 12.75 join relationships (maximum 94).

• Tools improve vanilla LLMs substantially. BI-Agent's tool design improves accuracy by up to 40 percentage points across frontier LLMs, and improves frontier models by over 10 percentage points on average.

• Post-training adds further gains. Post-trained BI-Agent yields gains of up to 30 points. Both the tool gains and the post-training gains are described as highly statistically significant.

• A small post-trained model becomes cost-competitive. BI-Agent with post-training enables Qwen3-8B to achieve quality comparable to much larger frontier models at up to 50× lower cost.

• LLM struggles are concentrated in specific steps. LLMs struggle with predicting joins and transformations over complex schemas, and with recognizing when table-reshaping is needed (e.g., pivot, unpivot, transpose, wide-to-long), partly because SQL lacks native reshaping operators.

• A large fraction of curated cases were hard even under simplifying hints. In the LLM-assisted verification step, models failed to reproduce the expected result table across 10 attempts in a simplified setting in 23.1% of cases; this triggered further review, leading to repair in 14.4% of all cases, discarding in 5.2% of all cases, and unchanged retention in 3.5% of all cases.

Methodology in Plain English

The researchers started from real artifacts rather than synthetic questions. They crawled publicly accessible Power BI project files and programmatically extracted each project's raw data files and metadata, including user-programmed transformation steps, manually defined join relationships, and final analytical dashboards, using DAX-studio and RPA (Power Automate).

From those projects they sampled user-created visualizations and kept only ones whose intent was semantically clear from the title and surrounding context, whose paraphrased natural-language query matched the visualization's underlying computation logic, and which were in English. They paraphrased each visualization's intent into a business question and exported the visualization's underlying result table as a CSV file to serve as the ground truth. Where the visualization title omitted implicit filters, they revised the query to include them (for example, adding a "since 2007" filter).

To avoid brittle evaluation, they augmented each test case with multiple semantically valid ground-truth tables when the query semantics justified extra columns, with augmentations reviewed and agreed upon by two domain experts without consulting model outputs. They also ran an LLM-assisted verification step using two independent frontier LLMs given simplifying hints such as necessary transformations and transformed tables, ground-truth joins, and the subset of tables relevant to the query. Failure of both models after 10 attempts triggered another round of human review (averaging 1.5 hours per case) rather than automatic exclusion, so model solvability was not an inclusion criterion.

For the agent, they wrapped specialized data-management algorithms as callable tools. A transform tool built on transformation-prediction algorithms inspects all input raw tables, decides whether reshaping such as transpose, pivot, or unpivot is needed, applies the predicted transformations, and returns original and transformed tables. A join tool built on a join-prediction algorithm optimized for snowflake-like schemas returns all predicted joins for a list of input tables. A search tool prompts an LLM to conservatively select potentially relevant tables from a query and table representations consisting of column headers and sampled rows. A standard coding tool executes LLM-generated Python or SQL, returns results or errors, and caches program variables across interaction rounds.

Because models still struggled with multi-step workflows even with tools, the authors added a post-training framework that synthesizes training trajectories from real BI projects, then post-trains the backbone model with SFT and RL. They evaluated BI-Agent and its post-trained variants against 24 models and systems on BI-Bench.

Why This Matters

Impact on research. The paper argues that end-to-end BI is a distinct and under-benchmarked problem: NL2SQL benchmarks assume tables are already cleaned and analysis-ready, and other data-analysis benchmarks come from Jupyter notebooks or scientific papers rather than BI. The authors position BI-Bench as complementary to those efforts and report the first demonstration that tool use and domain-specific post-training produce synergistic gains on end-to-end BI. The work also connects decades of data management research on join prediction and transformation prediction to modern LLM agent orchestration.

Real-world applications:

• Self-service analytics by non-technical users. BI tools are used by over 100K organizations worldwide for decision-making; the paper targets the pain point that selecting, transforming, and joining tables remains hard without database or programming expertise.

• One-off, low-touch analysis over fresh or external raw data. The authors explicitly target cases where no curated data model exists and users are unwilling to pay the up-front ETL cost just to answer a small number of questions.

• Cost-sensitive deployments. A post-trained Qwen3-8B matching much larger frontier models at up to 50× lower cost matters for organizations running many BI queries.

• Data preparation as the bottleneck. Errors traced to reshaped tables and join prediction — for example, pivot tables with month names as column headers that cannot be joined on "Month" until unpivoted — directly correspond to common spreadsheet-import problems in enterprise data.

Industry relevance. The authors include researchers from Microsoft and Microsoft Research, and the work builds on real Power BI project files and references Power BI and Tableau as the leading vendors. The released code and data are at https://github.com/Hu-Chuxuan/bi-agent.

Future Directions

• Broadening the tool set. The paper states that the tools designed for BI-Agent are not meant to be exhaustive and can be extended with additional data-management primitives inside the same LLM tool-call loop.

• Extending beyond the benchmark's scope. The authors position their problem as complementary to traditional BI rather than a replacement for mature enterprise BI systems, leaving open how far the approach scales toward the full enterprise BI lifecycle, since current BI assumes data has already been cleansed and organized into a curated model.

• Out-of-domain transfer. The paper reports gains on out-of-domain data analysis tasks such as Spider 2.0, raising the question of how broadly the trajectory-synthesis post-training framework transfers beyond BI.

• Improving the weakest steps. The findings that LLMs fail at join prediction and at recognizing and executing table-reshaping transformations point to specific capabilities — holistic table-structure understanding and correct generation of reshape logic — that remain open problems; the paper states that a visualization is not required as output, so producing visualizations that match user-authored dashboards is also left open.

Target Audience

This paper is most useful to researchers and practitioners working on LLM agents for data, data-management and database researchers interested in applying classical algorithms as agent tools, and industrial teams building analytics or BI copilots. It also serves benchmark designers interested in how to construct and validate end-to-end data benchmarks from real user artifacts, and engineers evaluating whether post-training small models can replace large frontier models in high-volume, cost-sensitive analytics workloads.

Authors’ abstract

Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorly on BI-Bench, with less than 50% accuracy. To address their limitations, we design a tool-augmented BI-Agent that decomposes BI workflows into subtasks on structured data, such as search, join, and transform, and orchestrates specialized data management methods across BI stages. Furthermore, we develop a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further post-trained using both supervised fine-tuning (SFT) and reinforcement learning (RL). BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points. Our results highlight the importance of combining tool-augmented reasoning with domain-specific post-training in complex BI workflows, and point to promising directions for future research.