Measuring Agents in Production

AI summary

Overview

Research area: AI agent systems in industry — empirical study of deployment practices, engineering methods, and organizational constraints.

Technical level: Intermediate. The paper is an empirical study rather than a technical contribution, so it requires no specialized mathematical background, but familiarity with LLM agents (prompting, fine-tuning, RAG, evaluation) helps contextualize the results.

Scope: The first systematic, first-hand study of how LLM-based agents are actually built and operated in production, based on 20 in-depth interviews and a 306-practitioner survey yielding 86 deployed systems across 26 domains.

What This Paper Is About

Despite rapid claims about LLM-based agents in finance, healthcare, education, and enterprise workflows, there is almost no public data on how working production agents are technically built. Most companies treat their architectures and evaluation methods as proprietary, so the research community lacks visibility into what actually works. MAP addresses this gap by gathering primary evidence directly from practitioners behind deployed or piloted agents, characterizing their decisions along 17 design dimensions and identifying where real practice diverges from published research.

Key Contributions

1. First large-scale empirical study of production agents. The authors conducted 20 in-depth interviews with deployment teams and surveyed 306 practitioners, filtering to 86 systems in production or pilot phases serving real users.

2. Quantitative characterization across 17 design dimensions. The paper reports concrete distributions for model selection, post-training, prompt construction, agent architecture, frameworks, evaluation, and operational constraints — data that is typically never disclosed.

3. Thirteen concrete findings organized under four research questions (applications, methods, evaluation, challenges) plus three cross-cutting themes: deployment diverges from research, reliability is achieved through system-level design, and human-in-the-loop is a deliberate architectural choice rather than a temporary crutch.

4. An agenda of underexplored research directions derived from observed gaps, including sample-efficient post-training robust to model upgrades, agent-specific inference scheduling, and correctness-aware system design.

Main Findings

• Productivity drives adoption. 80% of practitioners cite increased productivity and 72% cite reduced human task-hours; harder-to-quantify goals like risk mitigation (12%) are far less common. Where teams evaluated alternatives, 83% preferred agents over non-agentic software or human execution.

• Simple, controllable methods dominate. 68% of deployed agents execute at most 10 steps before human intervention (47% fewer than 5), 70% rely on prompting off-the-shelf models rather than weight tuning, and 79% construct prompts manually or with light LLM assistance.

• Closed-source frontier models dominate. 17 of 20 case studies use proprietary models; open-source is adopted mainly to address cost or regulatory constraints, not capability.

• Latency tolerance is surprisingly high. 66% of systems allow minute-scale or longer responses, and 17% set no explicit limit. Asynchronous background automation of human workflows is the dominant use case, with only 5 of 20 cases requiring real-time responsiveness (mostly voice agents).

• Human users are the primary consumer. 92.5% of deployed agents serve humans rather than other systems; 52% serve internal employees, 40% external customers, and only 8% non-human systems. Internal-first deployment is a deliberate risk management choice.

• Human-in-the-loop evaluation dominates. 74% rely on human verification; 52% use LLM-as-a-judge, almost always paired with human review; 42% use rule-based checks. 75% evaluate without formal benchmarks, relying on A/B testing, production monitoring, and expert feedback.

• Custom implementations beat frameworks at scale. 85% of case studies build in-house with direct API calls rather than third-party frameworks, citing flexibility, simplicity, and security.

• Reliability is the top bottleneck. 38% rank core technical performance (reliability, robustness, scalability) as their top priority — far above governance (3%) or compliance (17%). Teams mitigate through read-only modes, sandboxing, wrapper APIs, role-based access, and bounded autonomy rather than algorithmic fixes.

• Model upgrades create fragility. Some teams run legacy models alongside newer ones because prompts and scaffolds lock onto specific model behaviors, undermining the assumption that newer models monotonically improve agent performance.

• Security is handled implicitly. 69% of systems retrieve confidential data, but security is largely achieved through deployment constraints rather than dedicated mechanisms.

Methodology in Plain English

The researchers combined qualitative and quantitative methods. First, they conducted 20 semi-structured interviews (30–90 minutes each) with engineering teams, expanding their sample iteratively via snowball sampling to cover diverse sectors, organization sizes, and deployment scales. Each interview followed a consistent protocol and was independently coded by at least three researchers using grounded theory techniques, with disagreements resolved through peer debriefing. Second, they ran a 47-question online survey with dynamic branching, distributed through conferences, meetups, a MOOC, and professional networks between July and October 2025. From 306 valid responses, they filtered to 86 systems explicitly in production or pilot. Domain classification used semantic aggregation plus three independent human annotators. Categorical comparisons report 95% confidence intervals via bootstrap resampling. The authors explicitly acknowledge limitations: geographic concentration in the Americas, participation bias toward their own networks, and a fast-moving field that may shift some fine-grained patterns.

Why This Matters

The paper is distinctive because it makes proprietary deployment knowledge public, giving researchers an empirical baseline for what production agents actually look like — and revealing a substantial gap between research priorities and industrial reality.

Impact on research: Counterintuitive findings like bounded autonomy, minimal post-training, and relaxed latency requirements challenge several prevailing narratives in the agent literature. The paper reframes human-in-the-loop not as a temporary limitation to be engineered away, but as a durable design principle in reliable systems.

Real-world applications:

• Healthcare and insurance workflow automation, where agents prepare claims and obtain approvals asynchronously while tolerating minute-scale latency.
• Customer support and sales assistance, where agents augment human personnel and output quality matters more than speed.
• Enterprise SRE and incident triage, where agents generate recommendations in read-only mode for engineer review.
• Business process automation in finance, HR, and corporate services, where long-tail deployments are expanding beyond coding and math.

Industry relevance: For teams planning agent deployments, MAP offers a realistic picture of the architecture choices that correlate with production success. For tool and framework builders, it explains why 85% of production teams bypass existing frameworks. For model providers, it highlights that upgrade fragility is a real operational cost.

Future Directions

• Sample-efficient post-training and upgrade-robust learning. Since model brittleness under provider updates discourages fine-tuning, research could focus on methods that transfer learned behaviors or preserve performance when base models change.

• Inference-time scaling for latency-tolerant agents. Because two-thirds of systems permit minute-scale responses, evolutionary search, test-verification architectures, and batched agent inference scheduling become viable research directions.

• Reliability and correctness as first-class research goals. Continuous correctness verification during development and runtime calls for advances in observability, failure detection, and recovery, plus training methods responsive to system-level specifications.

• Efficient multi-model coordination. With 59% of systems coordinating multiple models for cost, modality, or migration reasons, routing and orchestration strategies are underexplored.

• Reusable benchmark curation and synthetic data. The convergence on golden question-answer sets across diverse domains suggests opportunities for principled, transferable evaluation construction.

• Direct auditing of deployed systems. The authors describe this as active follow-up work to complement self-reported interview and survey data.

Target Audience

This paper benefits AI systems researchers studying agent architectures, reliability, and evaluation; ML engineers and technical leads designing or operating production agent deployments; product and platform teams deciding whether to adopt frameworks versus custom implementations; model providers and infrastructure vendors seeking to understand real deployment constraints; and policy or governance researchers interested in how current production agents handle sensitive data and safety through operational rather than algorithmic means.

Authors’ abstract

LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first systematic study of Measuring Agents in Production, MAP, using first-hand data from agent developers. We conducted 20 case studies via in-depth interviews and surveyed 86 deployed systems practitioners across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and their top development challenges. Our study finds that production agents are built using simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability (consistent correct behavior over time) remains the top development challenge, which practitioners currently address through systems-level design. MAP documents the current state of production agents, providing the research community with visibility into deployment realities and underexplored research avenues.