OfficeTown: Generative Organizations for Evolving Agent Intelligence

AI summary

Overview

Research area: AI agents, specifically the simulation of professional organizations as a source of evaluation tasks and training data for generalist tool-using agents. Technical level: Advanced. The paper assumes familiarity with agent harnesses, executable benchmarks, supervised fine-tuning, and on-policy distillation, though the core idea is explained clearly enough for a non-specialist reader. Scope: The paper introduces OfficeTown, a framework that simulates a persistent, permissioned professional organization from a one-sentence description and harvests the work it produces into a 406-task benchmark (OfficeTown Bench) plus training resources used to improve Qwen3.5-9B.

What This Paper Is About

Most agent benchmarks treat tasks as isolated instances, but real professional work emerges from ongoing activity: deliverables get reviewed and revised, decisions spawn follow-up requests, and materials accumulated in one project become inputs to the next. OfficeTown instead models the system that continuously produces such work, rather than hand-authoring individual tasks. The goal is to show that this organizational simulation yields both a challenging benchmark for frontier models and useful training data that improves agents on external professional-work benchmarks.

Key Contributions

1. A generative organizational simulation framework. From a natural-language description of an organization's purpose and activities, OfficeTown builds roles, reporting relationships, resources, and authority boundaries, then simulates asynchronous work in which professionals coordinate, delegate bounded assignments to assistant agents, and review returned deliverables. Organizational state is persistent but not globally observable: each member has a permission-filtered view determined by role, ownership, sharing rules, and confidentiality.
2. An expert-informed benchmark built from that ongoing work. OfficeTown Bench contains 406 tasks and 4,836 evaluation criteria drawn from 23 simulated organizations across eight professional domains, with 71 contributing domain experts providing feedback on task realism, clarity, input sufficiency, feasibility, and rubric correctness and coverage.
3. Training resources harvested from the same simulation. The authors package 2,202 complete execution trajectories for supervised fine-tuning (SFT) and 828 task–environment pairs for on-policy distillation (OPD).
4. Evidence of transfer. Frontier-model evaluation shows substantial remaining headroom on OfficeTown Bench, and training Qwen3.5-9B on OfficeTown data improves it on external benchmarks including GDPval, JobBench-Easy, APEX-Agents, and Terminal-Bench 2.1.

Main Findings

• Frontier models leave meaningful headroom. On the same 406 tasks with high reasoning effort, GPT-6 Astra leads with 74.4% Pass@1, followed by Grok-4.7 at 58.4% and Claude Opus 5.5 at 50.7%. Even the best model leaves roughly one quarter of tasks incompletely delivered.
• Partial fulfillment and complete delivery diverge. Claude Opus 5.5 passes more tasks than GPT-5.6 Terra (50.7% versus 39.4%) despite a lower Weighted Score (67.5 versus 76.1), so high average rubric fulfillment does not imply reliable full delivery.
• Domain strengths vary by model. Grok-4.7 exceeds GPT-6 Astra on assurance (70.2% versus 63.8%) despite a lower overall pass rate.
• Critical requirements are frequently missed. GPT-6 Astra satisfies 88.8% of critical criterion weight on average, yet misses at least one critical (P0) requirement on 84 of 404 eligible tasks (20.8%). The miss rate rises to 53.3% for GPT-5.6 Terra and 78.3% for DeepSeek-V4-Flash, while Claude Opus 5.5 and Grok-4.7 miss on 41.4% and 33.3% of eligible tasks.
• Turn budget matters, model effort less so. In the full records, Astra reaches a Weighted Score of 40 within 16 turns, followed by Claude at 18, DeepSeek-V4-Pro at 22, DeepSeek-V4-Flash at 24, and Kimi-K2.5 at 42. DeepSeek-V4-Flash's Low and High settings finish close (62.1 versus 61.9 in Weighted Score), whereas execution frameworks differ more: OfficeTown Executor reaches 61.9, Pi 48.6, and OpenCode 40.7. The authors state these are not token-matched comparisons.
• OfficeTown data transfers to external benchmarks. For Qwen3.5-9B on 79 shared GDPval tasks, the rubric score rises from 65.75 (Base) to 70.30 after SFT and 73.65 after SFT + OPD, a 7.90 percentage-point gain. Weighted score on GDPval rises from 40.41 to 44.87 to 51.48; JobBench-Easy Pass@1 from 9.40 to 16.16 to 19.43; APEX-Agents success from 42.31 to 51.22 to 63.64; Terminal-Bench 2.1 Pass@1 from 1.48 to 3.45 to 4.68. OfficeTown Bench Weighted Scores for the same stages are 26.20, 29.20, and 31.33.
• The principal–assistant design outperforms tested baselines. With GPT-5.5 at xhigh reasoning effort throughout, OfficeTown Harness has the highest reported point estimate in all five external columns: PRDBench DEV 89.67, PRDBench DEBUG 92.62, GDPevo 74.70, Terminal-Bench 2.1 87.4, and research code 46.40, versus Codex (85.87, 89.04, 64.51, 83.7, 40.88), MetaGPT (84.86, 86.51, 63.44, 81.1, 41.15), and CAMEL (76.74, 78.93, 63.45, 85.1, 39.95). On research code, OfficeTown scores 46.40 versus MetaGPT's 41.15 with 21.8% fewer reported tokens, while using 3.49 times Codex's tokens.
• Materials accumulate over simulated time. One Juniper organization produced 27 initial files and 470 subsequent logical outputs over 181 simulated days, with revisions and exact copies deduplicated; the authors note these counts track material accumulation, not task quality or success.

Methodology in Plain English

The authors start from a short natural-language brief describing what an organization does. OfficeTown expands that brief into departments, roles, reporting lines, and seed materials. Roles are grounded in O*NET and ESCO occupational descriptions, and each professional gets a profile plus skill packages. Seed materials mix retrieved or adapted public sources with synthetic organization-specific records, with source references preserved.

The organization then runs as a simulation on a shared simulated calendar. Professionals are activated asynchronously by requests, messages, meetings, and assistant deliveries. They exchange permitted materials, record agreed work as requests with outcomes, responsible people, supporting materials, and deadlines, then delegate bounded assignments to assistant agents. Assistants work inside a sandboxed, permission-filtered workspace where other people's private files are not mounted and shared documents are read-only. On delivery, the principal inspects the actual artifact and decides to accept, request revision, or seek more information. Work records, commitments, and unresolved decisions persist separately from conversation context, so refreshing a model session does not erase pending work.

To turn activity into evaluable tasks, the authors take a work request together with the materials available to its responsible professional at assignment time. This snapshot preserves temporal and permission boundaries while excluding later solutions and evaluator-only information. Each benchmark attempt starts in a fresh workspace initialized from those fixed inputs, and the agent can inspect and edit files, run commands, and use provided interfaces. Rubrics are drafted by a language model from task requirements, checked by a separate model-based review, and refined with expert feedback rated Critical (P0), Major (P1), Minor (P2), or Optional (P3). For training, the authors distill complete trajectories from DeepSeek-V4-Flash for SFT, then continue with on-policy distillation in OfficeTown environments using Qwen3.8-Flash-Next as teacher, and evaluate the resulting Qwen3.5-9B on external benchmarks.

Why This Matters

Impact on research. The paper reframes benchmark construction: instead of hand-authoring isolated tasks, it models the system that generates work, so tasks, context, and review feedback come from a single continuing process. The finding that models can show high average rubric fulfillment while still missing critical requirements suggests aggregate scores alone understate reliability gaps. The transfer results also indicate that simulation-derived data can supplement external benchmarks rather than only measuring performance on themselves.

Real-world applications:

• Generating training and evaluation data for agents that must produce file-based deliverables in spreadsheets, documents, and code.
• Stress-testing agents under realistic permission and confidentiality boundaries, where reporting relationships do not automatically grant file access.
• Studying review-and-revision workflows, since accepted deliverables feed colleagues' work and rejected ones return with comments.
• Probing capability in economically relevant domains such as assurance, clinical work, design, enterprise operations, finance, industrial work, legal work, and software.

Industry relevance. Organizations deploying tool-using agents need environments that reflect scoped access, persistent artifacts, and multi-step review. OfficeTown's sandboxing, save-accept-share distinctions, and filesystem validation offer a concrete model for containing agent actions, and the harness comparison gives practitioners a reference point for delegation architectures. The ethics statement is explicit that OfficeTown is a research project name, not a Microsoft Office product, and that the simulated organizations are examples rather than authoritative representations of occupations or institutions.

Future Directions

• Extending the simulations to more professional domains and institutional settings, with richer interactions and longer-running workflows.
• Simulating collaboration across organizations, not just within a single one.
• Investigating how persistent history, scoped access, and review specifically shape which tasks get generated and what agents learn from them.
• Improving simulation fidelity and efficiency, and validating simulated work against real professional practice.

Target Audience

Researchers and engineers working on agent benchmarks, agent training data, and multi-agent or principal–assistant architectures will get the most from this paper. It also suits practitioners building sandboxed, permission-aware agent deployments in enterprise settings, and readers interested in organizational or social simulation as a data-generation strategy. A background in LLM evaluation and fine-tuning helps, since the training and harness sections assume it, but the organizational framing and benchmark design are accessible to a broader technical audience.

Authors’ abstract

AI agents increasingly use files and software tools to produce professional deliverables. Such work draws on materials, decisions, and feedback accumulated across assignments. We introduce OfficeTown, a generative simulation framework for professional organizations in which tasks and their context arise from continuing organizational work. From a natural-language description of an organization's purpose and activities, OfficeTown initializes an organization whose members coordinate work, delegate bounded assignments to assistant agents, and review their deliverables. Artifacts, decisions, and unresolved matters persist and shape subsequent work, while responsibilities and permissions determine what each member can see and do. We curate work requests, their context, and review feedback into executable resources for evaluation and learning. We construct OfficeTown Bench with 406 tasks and 4,836 evaluation criteria from 23 simulated organizations across eight professional domains, incorporating domain-expert feedback. Frontier-model evaluations reveal substantial remaining headroom. Supervised fine-tuning and on-policy distillation on OfficeTown data substantially improve Qwen3.5-9B on external professional-work benchmarks, including GDPval. Supplementary evaluations show that the principal--assistant harness achieves higher scores than the single- and multi-agent baselines tested. These results highlight the value of organizational simulation in generating work-grounded data and environments for the continued development of generalist agents.