WorkFactory: Turning Professional Intents into Executable Work for Scalable Agent Training

AI summary

Overview

Research area: AI agent training and synthetic data construction for "working agents" — language-model agents that carry out professional workflows in digital environments and produce workplace deliverables.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, agent harnesses, tool-calling trajectories, and benchmark Elo ratings, though its central idea (make sure a task's inputs actually support the requested output) is explained concretely.

Scope (1 sentence): WorkFactory is a pipeline that turns professional intents into fully specified, executable task environments, then collects teacher trajectories from them to fine-tune the WorkPilot family of Qwen models at 4B, 27B, and 35B-A3B scales.

What This Paper Is About

Training agents for professional work requires many diverse tasks plus reliable expert demonstrations of how to do them. When a synthesized task is missing information — or contains contradictory information — the teacher agent cannot complete the work cleanly, and the resulting training trajectory is weak supervision. WorkFactory's goal is to construct task environments where every premise the output depends on is available and consistent, while the actual calculations and judgments are still left for the solver to perform, and then to verify this by having a probe agent attempt the task before an independent teacher does.

Key Contributions

1. Information support graph for task construction. A framework that plans backward from the intended output, representing what each result depends on (V_info), which carriers hold it (the user request, reference artifacts, external resources, future deliverables, or hidden internal construction requirements), and the support and location relations between them — so dependencies are explicit before any files are materialized.

2. Execution-based repair with work preservation. An isolated probe solver attempts the task using only solver-visible inputs and reports blocking defects; a refiner edits the request and reference files while preserving file count, types, and dependency structure, and each edit must pass construction checks plus an LLM-based "work-preservation" review against the original intent.

3. A large professional task corpus and the WorkPilot model family. 112,686 tasks spanning 612 occupations and 23 major occupational groups, with trajectories from GLM-5.3 used to fine-tune Qwen3.5-4B, Qwen3.6-27B, and Qwen3.6-35B-A3B, all improving over their bases on four professional-work benchmarks.

4. A systematic data-scaling study (2K–110K tasks) characterizing diminishing returns across model scales, execution-length dynamics, tool-failure rates, teacher choice, and cost–performance trade-offs.

Main Findings

• All three scales improve over their base models on all four benchmarks. WorkPilot-35B-A3B goes from 879 to 1565 Elo on GDPval-AA, from 28.91 to 62.35 on JobBench, from 47.12 to 56.53 on Workspace-Bench, and from 7.14 to 21.18 on OmegaUse-OfficeVal. WorkPilot-27B goes from 975 to a locally estimated 1612 Elo on GDPval-AA, and WorkPilot-4B from 661 to 1375.

• A 27B model lands in frontier territory. WorkPilot-27B's local GDPval-AA estimate of 1612 Elo is 12 points above DeepSeek-V4.1-Flash and 34 points below GLM-5.3, placing it within the range of the compared frontier models.

• Small models stay competitive at 4B. WorkPilot-4B exceeds Apodex-1.1-mini on GDPval and OmegaUse-OfficeVal (11.77 vs 10.98 on the latter).

• Diminishing returns are model-dependent. For Qwen3.6-35B-A3B, the first 5K tasks contribute 447 Elo points, while scaling from 75K to 110K adds only 13. By 30K tasks, the 35B-A3B and 27B models have captured 88% and 94% of their respective gains en route to the 110K checkpoint; the 4B model gains 546 Elo by 30K tasks. On JobBench, 35B-A3B moves from 58.80 at 50K tasks to 62.35 at 110K.

• Execution length expands then contracts. Mean execution length for 35B-A3B rises from 24.87 turns at baseline to 162.88 at 2K tasks, then falls to 85.90 at 110K while Elo keeps improving. The share of GDPval-AA trajectories hitting the 250-turn limit falls from 21.9% at 2K to 2.7% at 110K.

• Tool reliability improves unevenly. Code-execution failures drop from 16.0% at baseline to 6.3% at 110K. Web-search failures rise to 22.4% at 2K then fall to 10.4% at 110K, while web-fetch failures stay around 40%–48%.

• Information-level planning fixes some defects and worsens others. In a 5K-task controlled comparison, broken derivations fall from 24.86% to 6.86% and underspecified requests from 7.22% to 0.62%, but leakage/shortcut rises by 1.44 points and self-inconsistency by 1.18 points.

• Execution-based repair reduces defects across all eight categories. Cross-file inconsistency falls from 25.74% to 4.84%, execution-feasibility defects from 6.64% to 0.48%, and leakage from 3.36% to 0.84%.

• Cleaner environments produce better supervision. Training on trajectories from each condition yields 1245 Elo (artifact-level planning), 1293 (+48, information support graph), and 1326 (+33, adding repair) — an 81-point cumulative gain that tracks the defect reductions.

• Teacher choice matters on the same task set. On the 30K subset, GLM-5.3 trajectories produce a student at 1483 Elo [1455, 1510], versus 1248 Elo [1220, 1274] for DeepSeek-V4-Flash-0731; both beat the 879-Elo base.

• The models sit on the cost–performance frontier. Estimated inference cost per GDPval-AA task is about $0.012 for WorkPilot-4B, $0.031 for WorkPilot-35B-A3B, and $0.396 for WorkPilot-27B, all non-dominated among the models compared.

Methodology in Plain English

Step 1 — Invent professional intents from three sources. The team builds a library of task intents by (a) expanding a public occupational hierarchy into fine-grained roles and realistic situations, (b) combining documented skills when one skill's output can feed another, and (c) working backward from real professional deliverables (instruction backtranslation) to infer plausible goals. An LLM reviewer filters candidates for coherence, feasibility, professional appropriateness, and plausibility.

Step 2 — Plan backward from the deliverable. For each intent, a planner starts at the requested output and asks what calculations and judgments it needs, then what information those need. The result is a graph of information units, each tagged by role (fact, rule, intermediate value, deliverable, requirement) and by status (supplied, externally acquired, solver-derived, or internal to construction). The key discipline: premises must be provided, but intermediate and final results must be left underived.

Step 3 — Assign information to carriers. Each information unit is bound to a place — the user request, a reference file, an external resource, an output file — with access metadata. Internal construction requirements stay hidden and must never carry task-specific facts the solver would need.

Step 4 — Build the actual files. The graph becomes a construction plan (request u, artifact specs A, acyclic dependencies D), and a worker reuses, adapts, or synthesizes artifacts. Programmatic checks compare planned versus generated file counts and types; LLM review checks content coverage and that the request references all provided files. Failing tasks get up to three automatic retries, then are discarded.

Step 5 — Probe and repair. A probe solver attempts the task using only what a solver would see, with no access to the graph, the plan, or prior diagnoses. It reports blockers grounded in inspected sources and tool observations, distinguishing them from solver mistakes. A refiner then edits the request and existing files (for example, adding a missing USD 0.15-per-carton freight term) without performing the work for the solver. Revisions must pass construction checks and a work-preservation review; passing tasks get a fresh probe. The loop ends when no blocker remains or the budget runs out.

Step 6 — Collect teacher trajectories. Retained tasks go to an independent teacher (GLM-5.3) under the same information boundary, using the Stirrup and Claude Code harnesses. Rollouts are filtered for repeated reasoning, repeated tool calls, and weak deliverables; the 5% longest rollouts get extra pairwise rejection sampling. Retained rollouts become masked autoregressive training sequences.

Step 7 — Train and evaluate. All fine-tuned models share one SFT configuration. Evaluation covers GDPval (GDPval-AA protocol with the Stirrup harness), JobBench, Workspace-Bench-Lite, and OmegaUse-OfficeVal, with three rollouts per model–task pair; GDPval Elo is official where available, otherwise a local Bradley–Terry estimate judged by Gemini 3.1 Pro Preview and calibrated to the v2.1 scale.

Why This Matters

Impact on research. The paper reframes synthetic agent data as an information-completeness problem rather than a task-novelty problem. Its graph-plus-probe loop offers a reusable recipe for making environments self-consistent, and its component analysis is an unusually direct link between measured environment defects and downstream Elo. The scaling curves — including the execution-length hump and the persistent 40%–48% web-fetch failure band — give the field concrete targets for future data curation.

Real-world applications:

• Generating large, occupation-diverse training sets for enterprise agents that must work inside spreadsheets, PDFs, and document toolchains.
• Auditing existing agent benchmarks for missing evidence, broken derivations, or leaked answers, using the same probe-diagnosis categories.
• Producing verifiable deliverable-based evaluation tasks for regulated professional domains where answers must be traceable to supplied sources.
• Cost-aware deployment: the paper's per-task cost estimates let teams pick among 4B, 27B, and 35B-A3B variants by budget.

Industry relevance. The corpus statistics reflect realistic office work — an average of 4.15 provided reference files per task at 1.56 MiB average size, 53 deliverable file extensions and 317 reference file extensions, and trajectories averaging 57.36 assistant turns, 62.95 tool calls, and 136.18K tokens. That profile matches enterprise agent workloads far better than short QA-style tasks.

Future Directions

• Multimodal professional work. The GLM-5.3 teacher accepts text only, so WorkPilot is trained and evaluated text-only despite Qwen backbones having native vision components; collecting visual-evidence trajectories is left to future work.
• Reinforcement learning on top of SFT. The authors position RL over these professional tasks and their execution feedback as the next stage beyond supervised fine-tuning.
• Closing the web-tool gap. Web-fetch failure rates remain near 40%–48% across the scaling sweep, an unresolved reliability issue distinct from code execution, which improved substantially.
• Understanding teacher effects. The GLM-5.3 versus DeepSeek-V4-Flash-0731 gap on identical tasks shows trajectory-generation choices strongly shape training benefit; what makes a teacher good for a given task set remains an open question.

Target Audience

Researchers and engineers building agent post-training pipelines — particularly those responsible for synthetic task generation, environment design, and trajectory curation at scale. It is also useful for benchmark designers who need vocabulary for diagnosing why a task is unsolvable, and for applied teams deciding whether a 4B, 27B, or 35B-A3B model best fits a professional-work deployment budget. Readers without background in agent harnesses and SFT will find the supplier-selection and shipping-carton examples the most accessible entry point.

Authors’ abstract

Scaling supervised training for working agents requires diverse professional tasks that yield reliable expert demonstrations. However, missing or inconsistent task information can obstruct teacher execution and weaken the resulting supervision. We propose WorkFactory, a scalable framework for constructing coherent and executable professional tasks. Starting from professional intents, information-level planning builds an information support graph to guide task instantiation, explicitly representing the dependencies and locations of information required for the intended outputs. Probe executions then identify support failures and guide repairs while preserving the intended work. We construct 112,686 tasks spanning 612 occupations and use the collected teacher trajectories to fine-tune Qwen models at 4B, 27B, and 35B-A3B scales, yielding the WorkPilot family. All three models improve over their respective base models across four professional-work benchmarks, with WorkPilot-35B-A3B improving from 879 to 1565 Elo on GDPval-AA v2.1. Scaling analyses reveal model-dependent diminishing returns, with execution length first increasing and then decreasing. Controlled component analyses show that planning and repair reduce probe-diagnosed task environment defects and improve downstream performance. The data and models are available at https://github.com/IQuestLab/WorkFactory.