Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics

AI summary

Overview

Research area: AI agents for data analytics — specifically, automatic tool construction for LLM-based coding agents that answer analytical questions over heterogeneous data lakes.

Technical level: Intermediate. The paper assumes familiarity with LLM agent loops (shell + file editor), benchmark evaluation, and the notion of step budgets, but its core ideas are explained without heavy formalism.

Scope: The paper introduces TEAL, an unsupervised framework that evolves a reusable library of executable tools and usage skills for a coding agent over a data lake, plus TEAL-Q, a synthetic question generator that lets evolution proceed when real user questions are unavailable, and evaluates both on KramaBench and LakeQA with four Worker LLMs under varying step budgets.

What This Paper Is About

Coding agents can answer a question over a data lake by hunting down the right files, cleaning them, and joining them — but they throw all of that work away and start over on the next question. The obvious fix is to build reusable tools, yet building them automatically is hard because verifying a tool normally requires knowing the correct answer, and in data-lake analytics even one gold answer requires an expert to solve the task first. The paper's goal is to evolve genuinely useful tools and instructions for using them without any labels, using only the data lake itself and the agent's own recorded behavior.

Key Contributions

1. TEAL, a label-free tool-evolution framework. A library consisting of tools (some backed by precomputed data artifacts like SQLite catalogs and value indexes) plus a natural-language skill file is evolved offline for a fixed Worker agent, then frozen and handed to the Worker at test time. Proposals come from two complementary channels: an Explorer that inspects the data lake with no access to questions or trajectories, and a Critic that diagnoses wasted steps and errors in the Worker's trajectories.

2. Rerun-based verification without ground-truth answers. A proposed change is kept only if the Worker actually calls the changed tool on a rerun, still finishes the task without extra steps, and — whenever the answer changes — an Analyst agent judges that the trajectory evidence supports the new answer over the old one. No stage uses labels.

3. TEAL-Q, synthetic question generation. When real questions are scarce, TEAL-Q drafts a question from a cluster of related files, hardens it over up to three rounds (e.g., adding joins or hard-to-parse files), builds an executable pipeline that reproduces the computed answer, and rewrites the question in natural language. Only the questions are passed to TEAL; the answers serve only to keep questions well posed.

4. An empirical study across Workers, benchmarks, and budgets. Four Worker backbones (Gemini-3.8-Flash, Gemini-3.1-Pro, Sonnet-5, Opus-4.8) on KramaBench and LakeQA under step budgets of 10 to 30, including comparisons against existing agentic methods and against ISE, a prior execution-only tool-evolution method.

Main Findings

• Tight budgets are where tools pay off. At a 10-step budget with Sonnet-5, TEAL gains 22.6 points on KramaBench and 30.4 points on LakeQA; at 15 steps it gains 9.3 and 21.7 points. Gains hold for all four Workers on both lakes at 10 and 15 steps, and in five of the eight panels TEAL at 15 steps is on par with or above the baseline at 30 steps.

• Evolve tight, deploy loose. Sonnet-5's LakeQA library evolved at 20 steps and deployed unchanged at 30 steps scores 68.3, above both the baseline (59.6) and a library evolved directly at 30 steps (64.2). Under a loose budget the Worker solves most tasks inline, so tools rarely change outcomes and feedback is weak; tight budgets expose visible failures that the Critic can target.

• Verification beats execution-only acceptance. Against ISE, which keeps any tool that runs without error, TEAL leads by 5.7 points on KramaBench at 15 steps. On LakeQA, ISE falls 10.0 points below the baseline — its search_file_content tool passed a single successful run but had 121 of its 126 test-time calls time out, consuming the step budget.

• The baseline is already strong, and TEAL still improves it. At 15 steps the baseline (63.23 on KramaBench) beats DS-Guru (39.73) and Smolagents (46.69), and at 30 steps (75.71) it beats LakeQA Agent (55.00) and LakeQA Agent + RAG (58.75). TEAL reaches 72.54 on KramaBench at 15 steps and 78.53 at 30, and 57.50 / 68.33 on LakeQA at 15 / 30 steps.

• Tools and skills must be evolved together. Dropping the Explorer costs a little (71.59 vs 72.54 on KramaBench at 15 steps) and also removes the artifact-backed tools; dropping the evolved skills costs much more (68.69), because the Worker no longer knows how to invoke the tools.

• Unsupervised verification matches oracle labels. Giving TEAL ground-truth answers during evolution yields 72.73, comparable to the label-free 72.54. Giving full ground-truth solutions yields 75.55 and more targeted tools (e.g., merge_entity_tables versus merely standardizing state names).

• Synthetic questions work. With 50 TEAL-Q questions per lake and a 15-step budget, Sonnet-5 reaches 72.18 on KramaBench (+8.95 over baseline) and 56.25 on LakeQA (+20.42), comparable to evolution on real questions (72.54 and 57.50). The same holds for all four Workers.

• Libraries transfer across harnesses. A library evolved in OpenHands, deployed unchanged, improves accuracy under Google ADK (53.95 to 70.20), Prime Agent (64.85 to 67.43), and Codex CLI (66.22 to 70.57).

• What the tools actually do. In the LakeQA library evolved with Gemini-3.1-Pro at a 20-step budget, query_lake_catalog is called in 97.1% of 240 test episodes; in KramaBench's library (Gemini-3.1-Pro, 10-step budget) the top tool resolve_concept_to_datasets is called in 55.3% of 159 episodes. Most used tools answer "where does the data live" or fix files that standard readers parse incorrectly.

Methodology in Plain English

The researchers separate two roles. A Worker is an ordinary coding agent with a shell and a file editor that answers questions under a step budget. A team of LLM agents then improves the Worker's toolkit offline, in batches of five training questions:

• Stage 0 runs the Worker and records each trajectory plus a summary of which tools were used, which were ignored, and where the Worker burned many consecutive steps without a tool call.
• Stage 1 sends an Explorer to survey the data lake as a domain expert would — never seeing a question — and propose lake-wide tools such as search over a precomputed catalog. A Toolsmith writes the code and unit-tests it. This runs once every 4 batches.
• Stage 2 has a Critic read the trajectories and the current tools to propose fixes for broken tools, promotions for ignored tools that could have helped, and new tools for gaps.
• Stage 3 has a Skill Writer update a plain-language document telling the Worker when and how to use each tool.
• Stage 4 reruns the Worker on the questions that motivated the change and keeps it only if the Worker used the tool, still finished, and any answer change is backed by the trajectory evidence (analyst judgment, not a gold answer). Otherwise the change is reverted after at most one round of repair.

For synthetic questions, TEAL-Q iteratively drafts, hardens, pipeline-checks, and rewrites questions from file clusters, so evolution can start before any real user question exists. Evaluation uses KramaBench (six domains merged into one lake of 1,742 files; 51 training and 53 test tasks) and LakeQA's tables-only variant (50 training and 80 test questions over 4,180 tables, each requiring one to five reasoning hops), with OpenHands as the harness and Opus-4.8 driving all evolution roles.

Why This Matters

Research impact. The paper reframes agent self-improvement as a data-management problem — build catalogs, indexes, and views for agents the way database systems build them for queries — and shows that budget pressure can substitute for labels as an evolution signal. It also makes a methodological point: accepting a tool because it executes cleanly is actively harmful, as the ISE reproduction demonstrates.

Real-world applications.

• Enterprise data lakes where analysts repeatedly ask related questions against the same messy tables, spreadsheets, and CSVs.
• Deployments with no labeled evaluation set, where TEAL-Q lets tool building begin from seed questions only.
• Agents with hard latency or cost ceilings, where converting multi-step exploration into a single tool call is the difference between answering and timing out.
• Portable tool libraries shipped across agent harnesses, since a library evolved in one harness improved accuracy in three others.

Industry relevance. The result that a library evolved at 20 steps outperforms one evolved at 30 steps when both are deployed at 30 is a concrete, cheap recipe for practitioners: train under tighter constraints than you deploy under. The reported limitation — longer prompts and more per-step reasoning — is the practical cost teams would need to budget for.

Future Directions

• Reducing the overhead of evolved libraries. The prompt grows at every step (more input tokens) and the Worker reasons more per step, plausibly about tool selection; the authors suggest better cache reuse of repeated prompt content and internalizing skills into the Worker.
• Deciding what to evolve. Artifacts varied across runs (catalogs, value indexes reaching 6 GB, full-text indexes, manifests, decoded archives), so what to build and when is left open.
• Better feedback signals without labels. Since tighter budgets improved feedback quality and ground-truth solutions still outperformed the label-free setting (75.55 vs 72.54), richer unsupervised signals remain an open target.
• Extending beyond data lakes. The question of which other agent domains have the property that "tools rarely fill a gap" and "correctness cannot be credited without labels" — the two assumptions the paper argues break prior tool-evolution methods — is untested.

Target Audience

Researchers and engineers working on LLM agents, agentic self-improvement, and tool use; data-infrastructure teams building natural-language analytics over lakes and warehouses; and benchmark designers interested in evaluation under resource budgets. Readers need comfort with agent loop mechanics and benchmark methodology, but the paper's framing is accessible to graduate students entering the area.

Authors’ abstract

Coding agents can answer analytical questions over a data lake by discovering, cleaning, and joining the relevant data. However, they discard this work after each question and redo it from scratch for the next one. Tool evolution, which automatically builds and refines reusable tools across tasks, is a natural fit for this repetition. Since a ground-truth label for a data-lake question requires an expert to solve it first, we consider the unsupervised setting. We propose TEAL (Tool Evolution for Analytics over data Lakes), which evolves a library of tools and usage skills for a given Worker agent without labels. Two agents propose tools from complementary evidence, an Explorer from the data lake itself and a Critic from Worker trajectories. A proposed change is kept only if the Worker adopts the tool on a rerun without extra steps and any changed answer is supported by trajectory evidence. We also introduce TEAL-Q, which generates synthetic questions for tool evolution when real ones are unavailable. We evaluate on KramaBench and LakeQA with four Worker LLMs under step budgets, which cap the number of steps the Worker may take per question. With a tight budget of 10 or 15 steps, TEAL improves every Worker on both benchmarks, lifting Sonnet-5 at 15 steps by 9.3 points on KramaBench and 21.7 points on LakeQA. A tighter budget also yields better feedback for evolution. For example, Sonnet-5's LakeQA library evolved at 20 steps and deployed unchanged at 30 steps scores 68.3, above both the baseline (59.6) and a library evolved directly at 30 steps (64.2). Additionally, a library evolved on synthetic questions performs comparably to one evolved on real questions. These results show that useful tools can be evolved without labels, and hopefully inspire future work on what to evolve and how to obtain feedback in unsupervised agent evolution.