Annotrace: A Workbench for Step-Level Annotation of Agent Traces

AI summary

Overview

Research area: Evaluation and observability of LLM agents, specifically tooling for structured, step-level human annotation of agent execution traces (with a human-AI teaming component).

Technical level: Intermediate. The paper is a systems/tooling contribution; it assumes familiarity with LLM agents, tool calls, and annotation concepts such as Cohen's kappa, but the architecture is described in largely conceptual terms.

Scope: The paper presents Annotrace, a workbench that ingests six agent trace formats, converts them into a unified nested step representation, and supports editable question-based annotation schemas plus LLM-proposed schemas and pre-annotations, together with agreement measurement between any two annotators.

What This Paper Is About

Studies of why LLM agents fail depend on people labelling execution traces by hand, and nearly every study builds its own disposable tool for the job. Existing observability platforms record traces but are not built for structured labelling, while general-purpose annotation tools label text without any notion of a step, a delegation, or a tool call. Annotrace aims to close that gap with a single workbench that preserves execution structure, lets annotators define what to label as editable questions rather than a fixed taxonomy, and brings a model into the loop to propose schemas and answer the same questions the human is asked.

Key Contributions

1. A unified nested step model across six trace formats. Adapters convert OpenTelemetry (GenAI), LangSmith, Langfuse, chat messages, Who&When, and a Coordinator format into Step objects carrying named fields, an actor, a kind, and a nesting depth. Format detection reads file content using a marker no other format uses (for example, run_type together with dotted_order for LangSmith), so the user need not know what they were given.
2. Schemas expressed as editable questions, not a shipped taxonomy. A schema is a list of questions, each with a prompt, user-typed labels, the step kinds it applies to, and optionally a condition on an earlier answer (the paper's example: ask "what went wrong?" only when "severity" is "Critical"). Conditions are authored in the interface rather than in a file, and the three included schemas are examples a user may delete.
3. A model in the loop that is asked exactly what the human is asked. The model reads ten randomly sampled runs and proposes a starting schema (at most three questions, four labels each, with house rules), then answers a trace using questions generated from the same object. Every answer records whether a person or the model produced it; a suggestion never overwrites a human decision, and editing one makes it the annotator's. Any of 46 models may be selected, so a cheap model can draft and an expensive one check.
4. Agreement measurement between any two annotator agents, human or AI. Two annotation sets are compared question by question, reporting raw agreement, Cohen's kappa, and the specific steps they disagree on; either side may be the open session. Kappa is reported because raw agreement flatters an uneven schema.

Main Findings

• Exact reproduction of a prior manual effort: On a 40-run multi-agent question-answering corpus, the coordinator adapter reproduced all 235 Thinking–Action–Response unit keys of a previous hand-built effort, on 40 of 40 runs, so earlier work still aligns with the new representation.
• High agreement between two models: Two models labelling the same six runs agreed on 91% of 22 answers at kappa = 0.82.
• Low cost and latency: A schema proposal takes about four seconds and annotating a run three to five seconds, at a fraction of a cent per call.
• Renaming a label does not destroy work: Renaming keeps and marks the answers that used the old name rather than discarding them.
• Trace-level success can hide step-level faults: In the paper's HotpotQA example, a run answers correctly, so a trace-level scheme calls it a success, yet its steps record eight wasted search cycles and a verification that the worker's own reasoning contradicts. The decisive fault sits inside a recovered cycle.
• Granularity is exposed rather than chosen: Where a worker's reasoning is a single text field of interleaved think, search, information, and answer tags, the adapter recovers Thinking–Action–Response cycles; a step containing finer steps is left unselected so nothing is labelled twice.
• A short first-run path: Figure 3 illustrates a visitor loading a run, typing one line about what they are looking for, accepting the proposed labels, and disagreeing with the model on a step, in under two minutes.

Methodology in Plain English

The authors built a tool rather than running a large empirical study. The pipeline has three parts. First, an adapter layer inspects an uploaded trace file, guesses its format from tell-tale markers, and rewrites the run as an ordered, nested list of steps that keeps delegation depth and separates out cycles hidden inside a single text field. Second, the annotator defines the labelling task as a list of questions with their own answer labels, optionally gated on earlier answers, editable directly in the interface. Third, a language model is used twice from the same question object: once to propose an initial schema after reading ten sampled runs, and once to answer a trace as a second annotator. Human and model answers are tracked separately so the human stays in control.

To check the approach, the authors tested whether their coordinator adapter recreated the exact set of annotation units from an earlier manual study, and measured agreement between two models labelling the same traces using raw agreement and Cohen's kappa. They also report timing and cost per call for schema proposal and annotation.

Why This Matters

Impact on research: Agent-failure research (MAST, Who&When, AgentErrorTaxonomy, AgentRx, as cited) currently ties annotation to a particular analysis objective and trace format, which limits reuse across studies. A shared step representation plus user-defined questions makes annotations comparable and earlier datasets re-usable, and putting agreement measurement alongside the annotation workflow makes reliability a first-class reported quantity rather than an afterthought.

Real-world applications:

• Debugging production agent deployments recorded in observability platforms, where a successful outcome can still contain redundant or contradictory steps.
• Quality assurance and regression review of multi-agent assistants, such as retrieval or question-answering agents that delegate sub-questions to workers.
• Safety and audit review of tool-calling agents, where a reviewer needs to mark the exact step where a constraint was violated.
• Dataset creation for training or evaluating automated failure detectors, using model pre-annotations as a starting point that humans then correct.

Industry relevance: The work sits directly between two established categories: observability platforms (LangSmith, Langfuse, Arize Phoenix, with OpenTelemetry conventions for representation) and general annotation tools (Label Studio, doccano, INCEpTION). By supporting the OpenTelemetry GenAI format and the two named commercial platforms as first-class inputs, it targets teams that already have traces and need structured labels on them.

Future Directions

• Human-in-the-loop agreement evidence: The paper reports agreement between two models; agreement between humans, and between a human and a model, on the same traces is not reported and is the natural next measurement for the "synergistic human-AI" framing.
• Scale of validation: The reproduction check covers a 40-run corpus with 235 units, and the two-model agreement check covers six runs and 22 answers; whether these properties hold on larger and more varied corpora is not established in the paper.
• Schema reuse and sharing: Since no taxonomy ships and each user writes their own questions, how schemas should be shared, versioned, or standardized across studies remains open.
• Anchoring effects of model proposals: Because the model proposes the starting schema and then answers first, whether its suggestions bias what human annotators notice or label is a question the current evaluation does not address.

Target Audience

Researchers and practitioners who need to inspect, label, or diagnose LLM agent execution traces: agent-evaluation and interpretability researchers building failure taxonomies, teams operating multi-agent systems who need step-level review of recorded runs, and engineers building annotation or observability tooling who want a concrete design for unifying heterogeneous trace formats under a user-defined, LLM-assisted labelling workflow. Readers looking for a controlled user study or a broad benchmark comparison will not find one here; the contribution is the workbench and its format-agnostic step model.

Authors’ abstract

Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $κ= 0.82$.