Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments

AI summary

Overview

Research area: Evaluation of Compound AI Systems / LLM agents in simulated enterprise environments (machine learning and agent benchmarking).

Technical level: Intermediate. The paper is readable without deep agent-framework expertise, though familiarity with planning strategies (ReAct, Chain-of-Thought), retrieval, and fine-tuning methods (SFT, DPO) helps.

Scope: The paper introduces EnterpriseBench, a 500-task benchmark plus a simulated enterprise sandbox and an automated task-generation pipeline, and reports how five LLMs perform on it under four planning strategies across two agent frameworks.

What This Paper Is About

Enterprises keep their data fragmented across email, chat, code repositories, CRMs, wikis and ticketing systems, and governed by access controls tied to organizational hierarchy. Most existing agent benchmarks do not model this fragmentation or these permissions, so it is unclear how well LLM-based agents actually work in an enterprise setting. The authors build a synthetic enterprise sandbox, generate 500 persona-specific tasks inside it, and measure how far current agents get.

Key Contributions

1. EnterpriseBench, a 500-task benchmark spanning HR, IT, Software Engineering, Business Operations and Sales, with tasks grouped into search (65%), CRUD (30%) and unanswerable (5%), and an average complexity of 3 tools required per task.
2. A simulated enterprise sandbox populated with synthetic and collected data across ten application/data-source types, including chats (3000 instances), enterprise mail (4500), code workspace (1000), CRM records (30195), enterprise internal overflow posts (5000), HR management system records (1260), and others, with employee personas drawn from a 4:3:2:1 distribution of Associates, Team Leads, Managers and Directors per department.
3. A persona-based task generation pipeline that combines domain and persona sampling, expert-curated goal templates seeded from O*NET 29.2, entity extraction, subgoal decomposition, and an iterative validation/rephrasing loop.
4. A dynamic Role-Based Access Control simulation with permissions assigned by organizational level (Levels 9 through 14), task requirements, data sensitivity and cross-departmental relationships, validated by human experts.

Main Findings

• Even the strongest agents fall far short: state-of-the-art models reach only 41.8% task completion on EnterpriseBench, which the authors highlight as the headline gap.
• ReAct beats no-planning and CoT: across both LangChain and DSPy, ReAct-style planning outperforms no planning and Chain-of-Thought. Under LangChain with the GPT-4 evaluator, GPT-4o scored 0.29 with no planning, 0.27 with CoT and 0.32 with ReAct; o1-mini scored 0.31, 0.28 and 0.35 respectively.
• o1-mini is the best-performing model tested: it leads in both frameworks, reaching 0.38 with ReAct under DSPy (GPT-4 evaluator) and up to 0.41 under the Gemini-2.5 Pro evaluator.
• Open-source Llama models lag substantially: Llama-3.1-8B scored 0.14 with ReAct under LangChain (GPT-4 evaluator), and Llama-3.3-70B scored 0.21.
• Gold planning shows a large headroom: supplying the gold plan yields roughly 40% to 50% improvements over ReAct, with the best figure being 0.63 for o1-mini with DSPy under the Gemini-2.5 Pro evaluator and 0.62 under GPT-4.
• Humans still outperform agents, but slowly: a human-as-agent study reached 70% accuracy versus 31% accuracy for the o1-mini ReAct LangChain agent under human evaluation, but took 8 minutes 30 seconds per task versus 50 seconds for agents.
• Small fine-tuned models can approach large ones: Qwen3-8B reached 27% accuracy with SFT and 29% with SFT+DPO, compared with 0.27 for GPT-4o with CoT, as reported in the paper's comparison table.
• Failure modes are dominated by planning and grounding errors: in an error analysis of 100 tasks using the o1-mini ReAct LangChain agent at 31% accuracy, the counts were Task Decomposition (20), Wrong Tool Selection / Wrong App Selection (18), Partial Factual Coverage (14), Search-based Answer Hallucination (8), Final Step Execution (7) and Context Retrieval (2).
• Expert validation filtered the benchmark: ten experts rated task realism, and 80% of tasks were judged correct, realistic and aligned with enterprise applications; the remainder were discarded.
• Frameworks differ in gains: DSPy with optimization-based few-shot prompting reached 0.62 (o1-mini, gold planning, GPT-4 evaluator) whereas LangChain topped out at 0.51 for the same model and setting.

Methodology in Plain English

The authors first assemble an enterprise-like world. They collect publicly available data, preprocess it, map entries to employee personas based on experience, skills and roles, and rephrase entries using enterprise-specific metadata. Conversations and emails are generated following an earlier methodology and grounded in that curated data. Employees are assigned roles and level-based permissions, and enterprise platforms such as GitHub are restricted to designated technical teams and their management chain.

Tasks are then generated automatically. The pipeline picks a domain and a persona, retrieves that persona's relevant context, chooses an expert-curated goal template, extracts entities, decomposes the goal into subgoals in a tool-aware setting, defines a task structure, and produces the final task. A validation and rephrasing loop repeats until the task passes a human-designed checklist, taking about 1 minute 20 seconds per task.

Agents are given the task and a set of tools described in a JSON file. For search tasks, tools return results by employee ID or semantic matching; for CRUD tasks, tools perform create, read, update and delete operations. Tool outputs are filtered by the access-control mechanism. The agent decomposes a task into subgoals, turns them into solvable steps, selects tools and iterates to produce an answer.

Evaluation uses Prometheus-2 with GPT-4 and Gemini-2.5 Pro to produce a rubric-based score from 1 to 5. For CRUD tasks, the authors call read() to verify execution before scoring. They also run human evaluation, with three experts serving as annotators averaged together. Context retrieval uses an ID-based retriever for structured text, Colpali for PDF documents, and query-to-SQL retrievers for tabular content. Training experiments used two NVIDIA GPUs with 80 GB each for SFT and DPO, additional 8 GB GPUs for retrievers, and API-based LLM inference.

Why This Matters

Impact on research: EnterpriseBench is positioned as the first benchmark that simulates data fragmentation and access-control hierarchies together, rather than studying them in narrow domains such as database management or CRM. Its paper reports expert validation (as does CRMArena among the compared benchmarks), and it offers an automated generation pipeline that lets others create more tasks with limited manual input.

Real-world applications:

• Pre-deployment validation: organizations can test agentic systems against a simulated enterprise before pointing them at sensitive production data.
• Access-control testing: the Level 9–14 permission scheme lets developers check whether an agent improperly surfaces or acts on data the requester should not see.
• Cross-functional workflow automation: tasks such as creating a repository and notifying a manager require coordinating identity resolution, tool calls and message generation.
• Domain-specific model selection: the domain-wise results indicate GPT performs well in HR and IT tasks, Claude excels in coding tasks, and o1-mini outperforms others in several non-technical domains, which can inform model choice.
• Small-model deployment: the Qwen3-8B results suggest that fine-tuned small models can be competitive on narrowly defined enterprise tasks, an attractive option where deployment cost and data locality matter.

Industry relevance: the work comes from Fujitsu Research and targets the practical question of whether LLM agents can be trusted with day-to-day enterprise operations. The gap between 41.8% completion and the human figure of 70%—alongside the 50 seconds versus 8 minutes 30 seconds trade-off—frames both the opportunity and the risk for enterprise adoption.

Future Directions

• Closing the planning gap: task decomposition was the single largest failure category (20 of the analyzed cases), so better trained planners rather than general-purpose few-shot prompting are needed.
• Grounding and hallucination control: the authors suggest improved agent memory management for search-based hallucination, and constrained decoding or function calling to ensure required structured fields such as employee IDs, policy names and dates are always produced.
• Better retrieval: context retrieval errors point to the need for retrievers that go beyond similarity matching to align with task intent.
• Scaling realism and reducing cost: the limitations section notes the dependency on an initial set of real enterprise data, the need for human experts to verify intermediate generation steps, remaining generation errors, reliance on current reasoning-model capabilities, and the absence of large-scale, terabyte-level data generation.
• Improving final-step synthesis: incorporating step validation or structured reasoning mechanisms to prevent miscombining intermediate results, which caused the Final Step Execution failures (7).

Target Audience

Researchers and engineers building or evaluating LLM agents and Compound AI systems, especially those working on enterprise automation, tool use, planning, and access-controlled retrieval. It is also useful for enterprise architects and product teams deciding whether current agents are ready for deployment, and for anyone designing agent benchmarks that must capture organizational structure rather than isolated tasks.

A note on reported figures: the abstract describes the benchmark as spanning "software engineering, HR, finance, and administrative domains," while Section 3 and the figures describe five domains (HR, IT, SWE, Business Operations, Sales). The text states that Qwen3-8B was evaluated at 27% (SFT) and 29% (SFT+DPO) on 1.2k samples with 1.2k preference pairs, while the accompanying Table 4 caption states the comparison was on 200 samples; the paper does not reconcile these. Several appendix tables (Table 7, and the tool inventory Table 8) are truncated in the available content, so their values are not reported here. Hyperparameter tables (Table 12 and Table 13) are also not included in the available content.

Authors’ abstract

Enterprise systems are crucial for enhancing productivity and decision-making among employees and customers. Integrating LLM based systems into enterprise systems enables intelligent automation, personalized experiences, and efficient information retrieval, driving operational efficiency and strategic growth. However, developing and evaluating such systems is challenging due to the inherent complexity of enterprise environments, where data is fragmented across multiple sources and governed by sophisticated access controls. We present EnterpriseBench, a comprehensive benchmark that simulates enterprise settings, featuring 500 diverse tasks across software engineering, HR, finance, and administrative domains. Our benchmark uniquely captures key enterprise characteristics including data source fragmentation, access control hierarchies, and cross-functional workflows. Additionally, we provide a novel data generation pipeline that creates internally consistent enterprise tasks from organizational metadata. Experiments with state-of-the-art LLM agents demonstrate that even the most capable models achieve only 41.8% task completion, highlighting significant opportunities for improvement in enterprise-focused AI systems.