What Did the Agent Actually Do? Evidence-Grounded Oversight for Long-Horizon Agents

AI summary

Overview

Research area: AI agent oversight and evaluation — specifically human-in-the-loop monitoring of long-horizon coding agents, combining benchmark construction (software engineering) with a training-free evidence-organization method.

Technical level: Advanced. The paper assumes familiarity with agentic execution traces, repository-level code tasks, retrieval/localization metrics, and LLM-as-a-judge evaluation.

Scope in one sentence: The paper introduces AgentMonBench (three software-engineering subsets totaling 300 evaluation instances) and the Evidence-Grounded Behavior Graph (EBG), a training-free method that organizes source-linked evidence into behaviors and relations so a monitor model can identify consequential agent decisions warranting user verification and localize supporting evidence.

What This Paper Is About

As coding agents take on long-horizon tasks, users shift from making each decision themselves to overseeing autonomous execution, yet the volume and fragmentation of agent activity makes it hard to know which decisions deserve scrutiny. The paper asks which consequential decisions in long-horizon agent tasks warrant user verification, and what evidence helps users understand their implications. To answer this, the authors build a benchmark covering two oversight dimensions — alignment between requirements and behavior, and awareness of consequential autonomous decisions — and propose a way of restructuring the agent's visible context so a monitor can connect observed behavior back to source evidence.

Key Contributions

1. A monitoring formulation for consequential decisions requiring user verification, centered on identifying specific choices that affect task goals, constraints, or outcome interpretation and connecting them to evidence in the agent's context.
2. AgentMonBench, a software-engineering benchmark with three complementary subsets: SpecGAP (consequential repository behaviors left unspecified by incomplete requirements), SilentSwap (semantic deviations from documented requirements even when existing tests pass), and FeedbackTrace (autonomous choices warranting verification, with targets derived from subsequent real user feedback).
3. The Evidence-Grounded Behavior Graph (EBG), a training-free method that organizes source-linked evidence through behaviors, scopes, and explicit relations, then renders task-oriented evidence views for monitor judgment.
4. Empirical validation across eight models and five families, showing EBG generally improves decision identification and evidence localization over the evaluated baselines, plus a Codex Harness integration on five real-world research tasks.

Main Findings

• Current LLMs remain limited at consequential-decision monitoring: the authors report "considerable room for improvement in identifying consequential decisions and grounding them in supporting evidence, even with structured context."
• EBG generally improves monitoring performance: EBG improves on the Base baseline across most settings and achieves higher scores than RepoGraph in most repository comparisons. For example, GPT-5.6 Terra goes from Base SF1 0.275 / LF1 0.309 to EBG SF1 0.392 / LF1 0.462, and Kimi K3 goes from Base LF1 0.497 to EBG LF1 0.603.
• Gains vary by monitor and task: the authors state that the benefits "vary across monitors and tasks" rather than being uniform. DeepSeek-V4.1-Flash, for instance, shows EBG SF1 0.372 versus Base 0.376 and EBG QQ 0.807 versus Base 0.832 — slight decreases — while its localization and downstream metrics improve.
• Evidence retrieval and decision understanding are distinct: better evidence retrieval can coincide with weaker decision interpretation, meaning monitoring requires connecting evidence to the meaning and consequences of a decision, not just locating the material.
• Base performance degrades as context grows: averaged over GPT-5.6 Luna, Kimi K3, and GLM-5.3, FeedbackTrace EHR falls from 0.667 to 0.422 and SilentSwap LS from 0.557 to 0.429 as input size increases, while SpecGAP LF1 declines more modestly.
• EBG improves average localization at every input scale: all nine average comparisons favor EBG. In the High input-token group, absolute gains are 0.078 on SpecGAP LF1, 0.115 on SilentSwap LS, and 0.108 on FeedbackTrace EHR, with all three monitors improving on each metric.
• Evidence-localization gains persist across input scales and tested hyperparameter settings, per the abstract and Section 5.
• Real-world application with Codex Harness: on five adapted research-task cases, Codex without the harness does not disclose the key issues for user verification; with it, Codex identifies and discloses them. Cases include disclosing a missing optimization priority, disclosing a changed randomization protocol and restoring per-problem seeds, disclosing validation-data leakage, disclosing that live API connectivity remained unverified, and disclosing unequal search budgets that undermined a claimed gain.
• Annotation quality: double-review agreement was reached on 447 of 484 conditions (92.4%) for SpecGAP, 468 of 500 substitutions (93.6%) for SilentSwap, and 93 of 100 sessions (93.0%) for FeedbackTrace.

Methodology in Plain English

Benchmark construction. The authors start from existing artifacts. SpecGAP and SilentSwap are built from DeNovoSWE, which supplies task documents, repositories, and runnable tests. For SpecGAP, they delete selected requirement conditions from the task document, then check that the remaining text does not reveal the deleted conditions and stays coherent; each deleted condition is traced back to repository evidence using two independent mapping methods (condition-by-condition and holistic), and disagreements are resolved by human reviewers. For SilentSwap, they keep the task document unchanged but introduce five semantic substitutions into the implementation — changes that break no existing test but do change the documented behavior — validated with a dedicated semantic test that must go from pass to fail, both individually and in combination with the other four substitutions. For FeedbackTrace, they take real Python software-engineering sessions from SWE-chat, keep at most one feedback event per session, and use the later user feedback to define the verification target while giving the monitor only the preceding interaction history.

EBG mechanics. EBG has two stages. First, a graph builder turns the visible context into a Structured Behavior Graph of Evidence, Behaviors, Scopes, and explicit relations — this stage is deterministic, uses no LLM calls, and only touches visible context. Behaviors bundle evidence references by role: Condition–Operation–Result for repositories, Demand–Action–Response for trajectories. Second, a presenter renders the graph into task-oriented views: for repositories, requirements guide directory ranking, matching scopes become seeds, each seed's incident edges form a one-hop subgraph, components are partitioned with one reading root each, and the view combines them with matching document passages and source code; for trajectories, behaviors are grouped by Task Scope in original order, with the last visible behavior labeled Current and the rest History. The monitor then reads the task requirements and these views to produce findings.

Evaluation. Eight monitors spanning five model families are tested: GPT-5.6 Luna, GPT-5.6 Terra, GPT-5.6 Sol, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, Kimi K3, Claude Sonnet 5, and GLM-5.3. Each is compared against a Base baseline (repository content or original execution trace) and, on SpecGAP and SilentSwap, against an adapted RepoGraph. Metrics range from 0 to 1 and mix rule-based scoring (SpecGAP LF1, SilentSwap LS, FeedbackTrace EHR) with LLM-as-a-judge scoring (SpecGAP SF1 and QQ; SilentSwap LC and CC; FeedbackTrace VPA and ESS), with GLM-5.2 and Qwen3.7-Max judging independently and their scores averaged.

Why This Matters

Impact on research. The paper reframes agent evaluation around a supervisory question — which decisions need a human to look — rather than around task success alone. It supplies a benchmark whose FeedbackTrace targets come from real delayed user feedback that the monitor never sees, and whose SpecGAP and SilentSwap conditions isolate specific failure modes of oversight: unstated requirements and silent semantic drift that passes tests. It also positions structured evidence organization as a lever separate from model capability, since EBG is training-free and deterministic in construction.

Real-world applications (drawn from the paper's five Codex Harness cases and stated use cases):

• Research replication pipelines: catching a fallback code path that silently changes per-problem randomization and restoring per-problem seeds.
• Machine-learning experiment review: spotting validation-derived samples entering training and disclosing the resulting score gain as leakage rather than improvement.
• Benchmark and optimizer comparison: detecting unequal search budgets so a reported gain cannot be attributed solely to the optimizer.
• Data analysis and reporting: flagging omitted criteria, such as an excluded-record or scoring rule, that change what a result means — the paper's illustrative examples include an omitted scoring rule that can reverse model rankings and a metric swap from exact match to semantic similarity.
• Tool and infrastructure validation: surfacing that API checks passed using cached responses, leaving live API connectivity unverified.

Industry relevance. The work targets the practical workflow where teams delegate long-horizon coding work to agents and need to know where to intervene. It evaluates monitors across commercial and open-weight models, integrates into an existing coding harness (Codex) through hooks, skills, and tools, and frames three review types — ambiguity review, adjustment review, and result review — that map onto plan approval, mid-execution change control, and result sign-off. Code and data are released publicly, with links to a GitHub repository, a Hugging Face dataset, and a project website.

Future Directions

• Closing the decision-understanding gap: the paper reports that better evidence retrieval can coincide with weaker decision interpretation, raising the question of how to make monitors reason about consequences rather than only locate supporting material.
• Reducing residual failures: the authors state that current LLMs still have substantial room for improvement on this monitoring task, and provide failure attribution and future directions in Appendix G.
• Broader evaluation settings: ablation results on EBG components, sensitivity to expansion depth and interaction budget, and disclosure detection with false alarms are reported in appendices E.2, E.4, and E.5 — indicating room to probe robustness and error modes further.
• Extending beyond the current scope: AgentMonBench is a software-engineering benchmark, and the Codex Harness demonstration covers five research-task cases; whether the formulation and EBG carry over to other long-horizon domains is not established in the reported results.

Target Audience

Researchers and practitioners working on agent oversight, agent evaluation, and human-AI interaction, particularly those building or auditing long-horizon software-engineering agents. It is also relevant to benchmark designers interested in deriving evaluation targets from real user feedback, to teams deploying coding agents who need a principled way to decide what to surface to users, and to engineers who want a training-free method they can layer onto an existing agent harness.

Authors’ abstract

As agents take on long-horizon tasks, users shift from making individual decisions to overseeing autonomous execution. Yet the volume of agent activity and the fragmentation of supporting evidence make it difficult to determine which decisions warrant user verification. We study monitors that identify consequential decisions and locate evidence to help users assess their implications. We introduce AgentMonBench, a software-engineering benchmark comprising three subsets that cover two complementary dimensions: alignment between requirements and behavior, and awareness of consequential autonomous decisions for verification. To support these judgments, we propose the Evidence-Grounded Behavior Graph (EBG), a training-free method that groups source-linked evidence into behaviors and organizes their relationships into a graph. EBG presents task-oriented views of this graph to help monitors interpret behavior in context. Experiments across eight models show that EBG improves decision identification and evidence localization in most settings compared with direct access to the original context. Further experiments show that EBG's evidence-localization gains persist across input scales and hyperparameter settings, while real-world applications illustrate its practical value for human oversight.