Observability, Tracing, and Systematic Agent Debugging

The model repeated a value returned by a stale cache. The trace recorded neither cache age nor source version. So the incident was filed as a generation error, and the wrong component was changed.

Nothing in the transcript could have separated the two readings. The words on screen are identical whether the model invented the status or faithfully reported what a stale tool handed it. The difference lives entirely in fields the run never recorded.

That misclassification is not exotic, and it is not a failure of attention. Agent failures have been catalogued: 14 distinct modes in 3 categories — system design issues, inter-agent misalignment, and task verification. Mert Cemri and 12 co-authors built that taxonomy, MAST, the first Multi-Agent System Failure Taxonomy, by annotating agent traces. The model saying something untrue is one family among many. The practice section takes the taxonomy apart. The point here is narrower. A trace that could not tell the categories apart is what assigned this incident to one.

One run leaves a Task trace, a Model span, a Tool span, a State span and an Outcome span. A failure can be localized only where the five line up. Four of those lanes no longer have to be invented locally. The names are already published.

OpenTelemetry's GenAI agent-span conventions define create_agent, invoke_agent — in a client and an internal variant — invoke_workflow and plan. gen_ai.operation.name is a Required attribute on all of them. gen_ai.provider.name is Required on create_agent and on the invoke_agent client span. The conventions are still at Development status. Their definition of invoke_workflow reads: “Represents an operation that executes a coordinated process composed of multiple agents or other operations involving generative AI.”

A second table lies on top of that one. Microsoft Foundry's 2026 agent tracing documentation publishes multi-agent semantic conventions developed with Cisco Outshift, built on the OpenTelemetry conventions plus W3C Trace Context. They add an execute_task span; the invoke_agent child spans agent_to_agent_interaction, agent.state.management, agent_planning and agent_orchestration; and the execute_tool attributes tool.call.arguments and tool.call.results.

Line the lanes up against those two tables and the mapping is direct. The Task lane is execute_task: goal, tenant, user, authority, run and environment identifiers hang off it. The Model lane is invoke_agent, carrying gen_ai.operation.name and gen_ai.provider.name along with input context metadata, model configuration, output, latency and usage. The Tool lane is execute_tool, and tool.call.arguments and tool.call.results are the two fields the case above was missing. The State lane is agent.state.management: memory writes, checkpoints, approvals, external mutations.

The fifth lane is the one neither table names — postconditions, grader results, business impact, incident labels. That is precisely why the State span and the Outcome span should be owned apart, and tested apart. A published span name costs nothing and makes a trace legible to a tool nobody on the team wrote. A private one costs nothing today and everything at the first cross-vendor incident.

Agent observability captures the causal surface of a run: task contract, context assembly, model requests, tool calls, state transitions, approvals, latency, cost, and terminal outcome. Debugging uses that evidence to localize the earliest material divergence — not the loudest symptom, the earliest one.

More logs do not guarantee observability. Events need stable identifiers, privacy controls, semantic fields, and links across model, tool, and business state. A stale cache read and a fabricated order status produce the same sentence in a log file and different values in a tool span. The distinction is a property of the schema, not of the volume.

Debug one failed trajectory end to end, from the contract at run start to the component you would change.

Start by reconstructing the contract: the requested outcome, the constraints, and the authority the run held when it began. Then find the first divergence, comparing expected and observed state step by step rather than reasoning backwards from the visible symptom at the end. Then classify the source. Classify it against a taxonomy that has been validated outside your team.

One exists. In 2025 Mert Cemri and 12 co-authors analysed 150 annotated traces, agreed with each other at kappa = 0.88, and built MAST, the first Multi-Agent System Failure Taxonomy. The paper is Why Do Multi-Agent LLM Systems Fail? Its abstract reports: “This process identifies 14 unique modes, clustered into 3 categories: (i) system design issues, (ii) inter-agent misalignment, and (iii) task verification.” The same authors released MAST-Data: 1,600+ annotated traces across 7 popular multi-agent frameworks.

The kappa = 0.88 is the part worth pausing on. It means two people looking at the same failed trace mostly agreed on what had gone wrong. That is the property an incident label needs, and the property ad-hoc labels never have. A team that files its incidents under those 14 modes can compare its failure profile against 1,600+ annotated traces from 7 frameworks. A team that invents fresh labels each quarter cannot compare its profile even with its own from last quarter.

Then test the hypothesis by replaying with one controlled change rather than many simultaneous edits. A replay with three edits confirms nothing about any of them. Finish with a regression artifact: a task, an assertion, a monitor, or a contract test that makes the same failure fail loudly next time. The exercise is complete when the run can be replayed from versioned evidence and the classification survives someone else reading the same trace.

The best debugging metric is not trace volume. It is the percentage of material failures that can be localized to a testable cause and converted into a durable regression check.

The distance between those two metrics has been measured. TRAIL — Trace Reasoning and Agentic Issue Localization — is a 2025 release of 148 human-annotated agent execution traces drawn from GAIA and SWE-Bench tasks. Its README describes the dataset as “a benchmark dataset of 148 annotated AI agent execution traces containing 841 errors across reasoning, execution, and planning categories”. The traces existed. The errors were inside them, already annotated. Localization was still hard. Deshpande and five co-authors report in the abstract that “Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL.”

Hold that 11% next to any dashboard whose headline number is traces collected. Collecting the evidence and closing the cause are different achievements. Only the second one removes a failure from the backlog. The team that runs agent observability and debugging owns the trace store's exposure as well as its coverage. What it should be able to demonstrate is one failed run, replayed from versioned evidence, localized to one component, and closed by a regression check that would catch it again.