Data World: Scaling Data Environments from Scratch

AI summary

Overview

Research area: AI agent training and evaluation — specifically, synthetic data-environment generation for LLM-based data-analysis agents (agents that write code to inspect files, compute quantities, and produce analytical reports).

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning post-training (policy gradients, group-normalized advantages), agent evaluation harnesses, and benchmark design, but its central idea is explained in plain terms.

Scope (one sentence): The paper introduces Data World, a multi-agent engine that synthesizes entire data-analysis worlds — files, tasks, and grading rubrics — from explicit generative mechanisms, and uses it to build both a 100-task benchmark (DataWorld-Bench) and a 2,135-task RL training set that improves a 35B-parameter model on held-out worlds and three external benchmarks.

What This Paper Is About

Training AI "data agents" requires large numbers of realistic file-based environments plus reliable ways to check whether the agent's analytical conclusions are correct. Real data is expensive, often private, and legally restricted, while open-ended analytical findings rarely have a known ground truth — so graders either rely on expert-written reference answers or on whatever a strong model happened to discover while exploring the files, which may miss valid explanations entirely. The authors' core insight is to invert the usual order: specify the mechanisms that generate the world before generating the files, so that the true underlying state is known, and then deliberately degrade that truth into imperfect, analyst-visible files while auditing that the intended findings remain recoverable.

Key Contributions

1. A mechanism-first synthesis framework. Data World formalizes a world as a four-tuple — a dependency DAG, a canonical state, a data environment, and a set of recomputable phenomena — and expands this through a world contract, layer-by-layer state generation, observation projection, and independent audit, with a stated requirement that target phenomena stay recoverable from imperfect observations without making analysis trivial.

2. DataWorld-Bench and a world-disjoint training set. From 1,302 candidate worlds the authors retain 1,087: 50 held-out worlds producing 100 test tasks across 16 domains (curated by 15 annotators for question realism), and 1,037 training worlds producing 2,135 RL tasks with no world overlap with the test set.

3. A measured computation–insight gap. Fine-grained rubrics that split data computation from insight discovery reveal that all ten evaluated frontier models are notably better at computing than at reaching the targeted conclusions.

4. RL-zero training without distillation. Training on synthesized tasks alone (no task-solving demonstrations from stronger models) yields DataWorld-35B-A3B, which improves over its Qwen3.6-35B-A3B base by 7.06 points on DataWorld-Bench, 14.28 on DAComp, 20.90 on DABstep-Hard, and 12.13 on JobBench.

Main Findings

• The benchmark is still hard for frontier models. The best average score is 74.43 (GPT-6-Astra) and the best insight-discovery score is 67.95 (Qwen-3.8-Max-0902). GPT-6-Astra leads overall and in data computation (87.80); GPT-6.1-Sol is second in computation (85.59) and third overall (73.30).

• Every model shows a computation–insight gap. The gap ranges from 4.87 to 21.35 points across the ten models, indicating that strong data processing does not automatically produce targeted analytical conclusions. The authors caution that criterion coverage and scoring differences may contribute.

• More interaction is not automatically better. GPT-6-Astra leads overall with the fewest rounds (28.7), while MiniMax-M3 takes the most rounds (93.8) yet scores lowest (49.75). Kimi-K3 uses the fewest total and response tokens (97.5k and 48.0k).

• Avoiding wrong conclusions is distinct from finding the right ones. GPT-6.1-Sol and GPT-6-Astra have the highest shares of trajectories with zero triggered negative rubric items (86% and 83%), while Gemini-3.8-Flash (57%) and MiniMax-M3 (59%) trigger them most often. Both GPT models trigger fewer negative items than Qwen-3.8-Max-0902 yet score lower on insight discovery.

• Domain difficulty varies. After removing each model's mean, bioinformatics, operations, and energy are relatively easier, while transportation, environment, and education are harder. No detectable cause-versus-consequence route difficulty difference was found at the achieved resolution.

• Training transfers to unseen worlds and external benchmarks. DataWorld-35B-A3B scores highest among the evaluated same-base models on all four benchmarks, including a 7.06-point gain on the world-disjoint DataWorld-Bench. By contrast, the retrained DataMind-35B-A3B improves external benchmarks but leaves DataWorld-Bench essentially unchanged (37.36 versus a base of 37.64, a change of -0.29).

• Gains extend beyond data analysis. JobBench covers 35 occupations, making it the least task-aligned suite; the 12.13-point gain there supports transfer to broader file-based professional reasoning. Scores, rounds, and response tokens all rise together, which the authors read as stronger long-horizon reasoning and tool interaction.

• General capabilities are largely preserved. MMLU-Pro accuracy is unchanged at 84.73%, LiveCodeBench-Hard drops slightly from 71.14% to 70.29%, and SimpleQA-Verified F1 rises from 22.20 to 24.00.

• Audit pass rates. Test worlds must pass every audit check; training worlds have a 95.98% DAG-edge pass rate.

Methodology in Plain English

The workflow proceeds in stages, each run by a separate agent session that shares no conversation state and communicates only through files on disk.

1. Sketch the causal skeleton. Personas from PersonaHub are used to prompt an LLM to construct synthetic dependency DAGs — graphs in which each node is a cluster of variables and each edge is a directional influence (monotonic, threshold, saturation, lagged, or interactive).

2. Write a world contract. A contract agent materializes the graph into a specification covering the world's entities and timeline, the generative mechanism for each edge (functional form, heterogeneity, temporal behavior, confounders), global invariants such as identifiers, units, and shared events, the imperfect "observation systems" through which the truth will be seen, and the final deliverables including the recomputable phenomena to be checked later.

3. Generate the truth layer by layer. Nodes are stratified by longest-path depth, and one agent session per layer writes and executes a deterministic script under a fixed random seed to produce that layer's canonical state. Upstream state is strictly read-only. Two safeguards apply: all nodes in a layer are committed atomically, and the layer agent empirically verifies each incoming edge's declared association pattern, functional form, time lag, and heterogeneity before proceeding. Each stage leaves a machine-checkable ledger plus a narrative self-report.

4. Degrade the truth into files. An observation map produces the analyst-visible directory. This is not arbitrary noise but explicit mechanisms: the same entity spelled differently across systems, missing values, delays, duplicates, unit drift, categorical encodings, schema drift, and cross-file references through unlabeled foreign keys — with no data dictionary provided.

5. Audit independently. A separate auditor recomputes the world from its artifacts, writes its own scripts to re-check every edge relationship, invariant, and target phenomenon, and additionally writes a world guide from an analyst's perspective. The auditor cannot modify the environment or truth and can only issue pass/fail.

6. Write tasks and rubrics. Each task is anchored to one target node and follows either a cause route (tracing upstream factors) or a consequence route (tracing downstream implications). To spread coverage, nodes are partitioned into depth strata and the maximal in-degree and out-degree node in each stratum is selected, yielding up to 2K tasks per world. A question-writer agent can read the contract, scripts, ledgers, canonical state, and world guide, but must confirm the targeted phenomena are derivable from the visible files before writing criteria. Rubrics separate data computation from insight discovery, with positive items for supported work and negative items for demonstrably wrong calculations or contradicted conclusions. Unlisted-but-valid insights get neither credit nor penalty.

7. Score with truth available. The evaluator sees the audited truth and scores reports item by item, normalizing net rubric points to [0, 1].

All construction stages use GLM 5.3 in the Claude Code harness. Evaluation of the ten frontier models uses the mini-swe-agent harness, with a single judging pass by Gemini-3.8-Flash over archived reports.

For training, Qwen3.6-35B-A3B undergoes Group Sequence Policy Optimization in an RL-zero setting: only the model's own rollouts and rubric rewards are used. To cut reward-scoring cost, the base model itself acts as the rubric judge — meaning the judge is also a weaker scoring model than the evaluation judge, a deliberate tradeoff.

Why This Matters

Impact on research. The dominant approach to task synthesis explores existing files and then writes rubrics based on what was discovered; the paper argues this couples supervision to the curator's or mining model's imagination. Specifying mechanisms first, then auditing recoverability, provides a route to scalable supervision where the criteria are checkable against a generative record. It also reduces pretraining-contamination risk, since new worlds and answers are synthesized rather than reused from public datasets. The result is a distinct complement — not a rival — to source-based benchmarks such as DiscoveryBench, BLADE, InsightBench, and InsightEval.

Real-world applications:

• Business analytics training — synthesized multi-table worlds model sales, promotions, staffing, and operations, so agents can be trained to explain metric changes without touching proprietary records.
• Enterprise spreadsheet and reporting workflows — the paper positions Data World for terminal-based analysis of interdependent files, as distinct from pipelines that simulate tool APIs.
• Healthcare and other sensitive records — the privacy and intellectual-property problem is a stated motivation; Synthea-style mechanism simulation is the closest prior precedent cited.
• Professional and occupational reasoning — the JobBench result (35 occupations) suggests the training signal carries into non-analyst file work.

Industry relevance. Environment supply is described as a bottleneck for agent post-training. An engine that produces environments, tasks, and rubric-based rewards in one pipeline addresses that bottleneck directly, and the RL-zero result matters commercially: gains were obtained without paying for trajectory distillation from a stronger teacher model. The model itself (DataWorld-35B-A3B) is a 35B-parameter mixture-of-experts derived from Qwen3.6-35B-A3B, an open-weight-scale footprint.

Future Directions

• Component-level ablations. The authors explicitly note that tight coupling and regeneration cost prevented controlled variants, so the framework's individual pieces — the audit, the observation projection, the route selection — are not isolated. Decomposing these is the most obvious next experiment, even as the authors argue such variants are unlikely to help.

• Closing the realism gap. Human preference curation selected for question realism but does not establish that tasks match the distribution of real user requests, particularly their ambiguity, implicit context, and evolving goals. Aligning synthesized tasks with observed user-query distributions is open.

• Better coverage and fairer grading of open-ended insights. Mechanism-grounded rubrics do not exhaust all valid findings, unlisted valid insights earn nothing, and LLM-based judges may introduce bias. Reducing judge bias and crediting alternative-but-valid conclusions remains unsolved.

• Real versus synthetic complementarity. The paper does not compare which source of environments is more effective, nor does it test whether the observed computation–insight gap can be narrowed by training specifically for insight discovery rather than averaged across both rubric sections.

Target Audience

Researchers and engineers working on agent post-training, synthetic environment generation, and data-analysis agent benchmarks will get the most from this paper — particularly those building RL pipelines who need checkable rewards for open-ended tasks. It is also useful for evaluation designers interested in rubric construction and contamination-resistant benchmark synthesis, and for practitioners at organizations where proprietary data cannot be shared but agent capabilities still need training. Readers wanting fully controlled ablation evidence or guidance on task realism for actual product settings should treat the results as framework-level rather than component-level.

Authors’ abstract

Data agents use code to process heterogeneous files and derive analytical insights. Scaling their training faces costly environment curation, privacy and intellectual-property restrictions, and difficulty verifying open-ended findings without known generative truth. Our core insight is to specify the world behind the files before generating the files themselves. We introduce Data World, a multi-agent engine that synthesizes environments, tasks, and rubrics from explicit mechanisms. Independent audits check target recoverability from observed files; rubrics support evaluation and RL rewards. Human preference-based curation for question realism selects 100 tasks across 50 held-out worlds and 16 domains for DataWorld-Bench. The engine also yields 2,135 training tasks from 1,037 disjoint worlds. Evaluation of ten frontier models shows that the benchmark remains challenging, particularly for insight discovery, with a consistent computation-insight gap. RL-zero training on the synthesized tasks produces DataWorld-35B-A3B from Qwen3.6-35B-A3B without trajectory distillation. Under matched evaluation, it improves over its base by 7.06 points on DataWorld-Bench, 14.28 on DAComp, 20.90 on DABstep-Hard, and 12.13 on JobBench, a broader occupational benchmark.