Cost, Latency, Deployment, Monitoring, and Incidents

The planner spawned many searches, retries, and model calls. Quality was acceptable. What broke the product was everything the quality score did not contain: tail latency, vendor rate limits, and cost per completed report all exceeded the budget.

Neither half of that failure was unforeseeable. Both halves have been measured and published. Take latency first. A component that is slow for one request in 100 sounds harmless. Send one user request out to 100 such components in parallel and 63% of requests come back slow. Dean and Barroso worked that out in 2013. An agent that fans out inherits a tail it never sees in its own metrics.

Cost is the same story. A Princeton team ran the 164 HumanEval problems five times each in 2024 and priced the runs. LATS with GPT-4 reached 88.0% accuracy at $134.50 of API cost. Their trivial warming baseline reached 93.2% at $2.45. An agent that costs roughly fifty times more is not reliably the more accurate one.

Big-bang release, Progressive release, and Per-task routing differ mostly in how much blast radius you accept at once. One of those blast radii has a published measurement, with a root cause attached.

On 19 July 2024, Windows machines crashed. The cause was a Rapid Response Content update, and CrowdStrike published its own root cause analysis on 6 August 2024. A new IPC Template Type defined 21 input parameter fields. The integration code that invoked the Content Interpreter supplied it only 20 input values. Then a Template Instance introduced a non-wildcard matching criterion for the 21st input. The Content Interpreter read past the end of the input array — an out-of-bounds memory read — and the system crashed. Microsoft estimated 8.5 million Windows devices affected. That is what one off-by-one field count buys when the content goes to all hosts at once.

The fix, in the vendor's own words, is the Progressive-release column written out as a procedure: “Staged deployment mitigates impact if a new Template Instance causes failures such as system crashes, false-positive detection volume spikes or performance issues. New Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected. Each ring is designed to identify and mitigate potential issues before wider deployment.” — CrowdStrike, mitigation 6, “Template Instances should have staged deployment”.

So the three options are not three tastes. Big-bang release replaces the old system for all traffic. Simple routing, a large unknown blast radius, a poor default — and 8.5 million devices is the shape of that unknown when it resolves. Progressive release means shadow, canary, staged authority and monitored expansion. It limits exposure, it needs comparison infrastructure, and it is what the vendor committed to after the fact rather than before it. Per-task routing selects workflow or model by risk and complexity. It controls economics, it makes the router itself a critical component, and it earns its keep at scale.

Production agent performance emerges from the full trajectory: model prefill and generation, tool latency, retries, parallelism, state access, approvals, and downstream queues.

A cheaper call can cost more per outcome, and a dearer one often buys nothing. Both have been measured on a public benchmark. A Princeton team ran the 164 HumanEval problems five times each in 2024 and reported total API cost beside accuracy. LATS with GPT-4 reached 88.0% accuracy at $134.50. The trivial warming baseline reached 93.2% at $2.45: higher accuracy, for roughly one-fiftieth of the cost. Reflexion and LDB cost over 50% more than that baseline.

The objection there is to the reporting, not to the agents: “Agents differ drastically in terms of cost. For substantially similar accuracy, the cost can differ by almost two orders of magnitude. Yet, the cost of running these agents isn’t a top-line metric reported in any of these papers.” A team that reports accuracy alone cannot tell the $134.50 configuration from the $2.45 one. That is the same blindness that lets a twelve-minute research agent pass review.

So the optimization target is cost and time per acceptable outcome. Deployment must version the runtime, prompts, tools, policies, memories, graders, and environment assumptions together. Otherwise there is no unit to attribute a cost change to.

Operational ceilings are published, and they are not generous. AWS Step Functions gives an HTTP task 60 seconds as a hard quota and stops an Express workflow after five minutes. A Standard workflow may run a year, but only through 25,000 execution-history events. Over a full year that is one event every 21 minutes, so an agent appending several events per step exhausts the count in days rather than months. Azure Durable Functions takes the other position: an orchestration instance can last seconds, days or months, or be configured never to end. Neither choice is free, and both belong in the cost model.

Operating an agent means versioning a release unit: model, prompt, tools, policies, state schema, graders, and dependencies. That unit is then held to limits. A latency budget allocates time across planning, tools, approvals, retries, and response. A cost budget covers tokens, tool fees, compute, storage, human review, and failure recovery. Monitoring tracks outcome, policy, state, latency, cost, and error indicators. The fifth node is the incident system: detection, containment, rollback, evidence, correction, and learning. Monitoring and the incident system should not answer to one owner, or to one test.

What a release unit with one hole in it costs is on the public record. Knight Capital deployed new code to eight SMARS servers and reached seven of them. The SEC's order describes the gap: “During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.” On 1 August 2012, processing 212 small retail orders, the system routed millions of orders over 45 minutes, obtaining over 4 million executions in 154 stocks for more than 397 million shares. Knight lost over $460 million. The SEC imposed a $12,000,000 civil money penalty. Seven of eight servers held the intended version. The release unit was the eighth.

Monitoring and incidents are two duties in statute too, and they run on different clocks. The EU AI Act requires providers of high-risk AI systems to establish and document a post-market monitoring system, based on a post-market monitoring plan that forms part of the Annex IV technical documentation. That is Article 72, and it is a standing obligation, owned continuously. Article 73 then sets the incident clocks. Serious incidents go to market surveillance authorities not later than 15 days after becoming aware, and not later than 10 days in the event of the death of a person. The widest cases move faster still: “Notwithstanding paragraph 2 of this Article, in the event of a widespread infringement or a serious incident as defined in Article 3, point (49)(b), the report referred to in paragraph 1 of this Article shall be provided immediately, and not later than two days after the provider or, where applicable, the deployer becomes aware of that incident.” A monitoring plan filed in Annex IV documentation and a two-day reporting deadline cannot be discharged by the same rota.

The production plan is written for an agent you intend to launch, not for a hypothetical one. It has to show where cost lands when the agent fails quietly, so that healthy-looking numbers are not hiding what customers absorb. And it has to state what one successful, policy-compliant task costs to run.

First, define the release artifact: version every behavior-changing model, tool, policy, schema, and grader. The eighth SMARS server is the standing argument for treating this as one artifact rather than five deployments. Second, allocate budgets: task-level limits for time, calls, tokens, spend, and review, with the orchestrator's own published ceiling written in beside them. Third, choose rollout stages: offline eval, shadow, advisory, staged, canary, and gradual authority, promoted ring by ring with bake-in between rings.

Fourth, build operational dashboards that track outcomes, policy failures, loops, latency, cost, and unresolved tasks. This is the step most often written as an instruction rather than a specification. A mean is what hides the tail: “If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds. If your users depend on several such web services to render their page, the 99th percentile of one backend can easily become the median response of your frontend.” — Rob Ewaschuk, 2016, in Google's Site Reliability Engineering. The prescription is concrete. Collect request counts bucketed by latency, a histogram with roughly exponentially spaced boundaries — 0–10 ms, 10–30 ms, 30–100 ms, 100–300 ms — instead of means. A mean latency panel for an agent is a panel that cannot show you the twelve-minute task.

Fifth, rehearse incidents: test pause, credential revocation, rollback, state repair, notification, and postmortem, against the deadlines the incident actually carries.

A production agent should have a cheaper and safer fallback path. If the agent exceeds its budget or confidence boundary, the product still needs a valid way to serve or defer the task.

The operator on call does not take a quiet dashboard as proof that the system is working. The failures that hurt users are the ones nobody is counting. The component holding its 10ms median while 63% of assembled requests run over a second. The accuracy score that is identical at $134.50 and at $2.45. The eighth server that no second technician was required to review. What the operator reports instead is a single figure: cost per successful and policy-compliant task, with the tail and the review burden attached to it.