EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation

AI summary

Overview

Research area: Evaluation and auditing of LLM-based computer-use agents — benchmarks, environment perturbation design, and verifiable-reward infrastructure for agentic reinforcement learning.

Technical level: Intermediate. The paper is readable with a general ML background, though the statistical protocol (paired sign tests, multiplicity correction) and the certification vocabulary require careful reading.

Scope: EnvOS is a framework for building, certifying, executing, and independently auditing evaluation environments in which an application's authoritative state changes mid-task, demonstrated on two prospective workflows (clinic booking and insurance-claim approval) plus separate evaluator-corruption challenges.

What This Paper Is About

Computer-use agents operate in software whose records change while they act — an appointment moves, a policy is revised — but benchmarks usually treat application state as frozen once an episode begins. When an agent fails in a dynamic setting, the failure may belong to the agent, or to a change that fired at the wrong moment, an update the model never received, or a grader that scored the episode wrongly. EnvOS's goal is to make those three possibilities separable, by certifying each state-changing test case before any model runs and by re-checking each verdict afterward from archived evidence.

Key Contributions

1. A phenomenon–perturbation formulation. A phenomenon is an application-independent class of changes to authoritative state; an executable perturbation is one instance of it, written as ρ = (δ, τ, c, g) — a state operator, a trigger, a disclosure channel, and a once-only guard. This keeps the world-state change separate from any model-visible notice.

2. A certification and evidence pipeline. Before evaluation, candidate instances are screened on intervention validity, observation consistency, feasibility, behavioral discrimination, and scorer robustness, with each check returning pass, violation, or monitor error (an unassessable check never counts as a pass). After rollout, separate code reconstructs verdicts from the archive, and sealed corruption challenges test the evaluator itself.

3. An evidence-bounded empirical study of two controlled stale-information workflows (clinic booking, insurance-claim approval), analyzed per model–workflow block rather than pooled into a cross-model ranking.

4. Reusable verifiable-reward environments. Because instances are seeded and procedurally generated and outcomes are scored from committed application state, the certified environments and their verifiers are presented as candidates for reinforcement learning from verifiable rewards — with the corruption challenges showing where such verifiers could mis-reward an episode.

Main Findings

• Clinic booking, Claude Opus 5: success is 0/16 when nothing extra is shown at confirmation (OFF-A), 2/16 when the agent is reminded to re-check the source (CUE-A), and 15/16 when the updated location is displayed directly (LIVE-A). The gain of LIVE over the reminder holds after correcting for all tests (p = 0.00293).

• Insurance-claim approval, Opus 5: the model re-checks on its own and succeeds on 10/10.

• Insurance-claim approval, Claude Sonnet 5: success rises from 1/10 to 9/10 when the current policy is shown again at approval, a gain significant in its pre-registered test but not after correcting for all tests.

• Model- and interface-dependence: whether an agent caught the change varied with both the model and what the interface disclosed, so the authors decline to combine results into a controlled model ranking.

• Evaluator challenge, blind corrupted episodes: on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean.

• Evaluator challenge, externally generated set: EnvOS caught 25 of 39 corrupted cases, against 4 for simple log checks.

• Null and control behavior: the claims blocks used 0 and 0 replacements under the registered reserve; in CLM-BS the arm run first and the arm run second each succeeded on 5 of 10 pairs, which the authors note cannot exclude a period effect.

• What the design cannot show: because the same state change is injected in every condition of a principal contrast, the experiments estimate within-configuration review-information effects, not the effect of making a static environment dynamic.

Methodology in Plain English

The authors start from a workflow an agent must complete — book a clinic session, approve an insurance claim — and an external event that changes a record the agent has already seen. They enforce a strict ordering: the old value must be served first, then a single eligible request triggers the update, then the agent acts. The state change and the disclosure to the model are logged as separate events, because a page having been served does not prove the model received it, and delivery does not prove comprehension.

Before letting any model run, they script through the environment: does the update fire at the right request and only once, does it leave unrelated records alone, is the task still completable, can a deliberately deficient scripted policy be distinguished from a competent one, and does the scorer catch known corruptions and reject good archives? Only configurations passing these checks are admitted. During a model episode, the policy runs in a separate OS account with restricted access, receives screenshots and text through a controlled semantic-browser interface, and acts through a fixed schema under a fixed action budget (48 actions for clinic, 40 for claims). The testbed is a synthetic workplace with 14 containerized infrastructure services including clinic and insurance applications plus mail, calendar, and database services.

Scoring is validity-first: an episode is checked against registered validity rules before its outcome counts, so a broken intervention or a missing model input is not charged to the agent. Outcomes are read from committed application state, not from the agent's verbal report. A second, separately implemented checker re-derives the verdict from raw archived events, and any term the archive cannot support is left unresolved.

The comparisons are pre-registered paired tests: each generated instance is run under every condition, only discordant pairs enter an exact two-sided sign test at α = 0.05, and Newcombe 95% intervals are reported for the paired difference. A twelve-slot paper-wide Holm family is reported as sensitivity analysis, but because it was finalized after the claims outcomes existed, the authors label its adjustment of the claims and historical tests retrospective.

Why This Matters

Impact on research. The paper reframes a benchmark number as a chain of separately checkable claims: the intervention fired, the observation arrived, the task was feasible, the scorer discriminated, the verdict is reconstructable. That decomposition is directly transferable to any benchmark with stateful environments, and it supplies a concrete failure taxonomy — mutation firing early or twice, missing promised observation delivery, scorer mislocalization, valid archives wrongly rejected.

Real-world applications.

• Enterprise workflow agents (expense, procurement, HR ticketing) where policy limits and approval rules are revised while an agent is mid-process.
• Healthcare scheduling and referral systems, where a session's location or slot changes after a patient-facing confirmation.
• Insurance and claims processing, where an approval amount must be computed against the policy in force at the moment of approval.
• Agentic RL training pipelines, where a silently failing intervention or an over-permissive checker corrupts the reward signal rather than merely the reported score.

Industry relevance. Teams shipping computer-use agents need to distinguish "the model is bad" from "the harness is bad" before they trust a regression number or a training reward. EnvOS offers a concrete admission gate, a logging discipline that separates serve from delivery, and an adversarial audit of the grader — plus published evidence that simple log checks caught only 4 of 39 externally generated corruptions while the full pipeline caught 25.

Future Directions

• Broadening beyond one scaffold. The prospective instances are generated within a single synthetic workplace scaffold; the authors explicitly note this is not evidence of portability to unrelated production applications.
• Reducing shared-archive exposure. "Independent" reconstruction means separate code paths written by the same team over the same archive, so agreement cannot rule out errors in the underlying evidence. The evaluator challenges are the authors' answer to the residual risk, and they report misses rather than eliminating them.
• Separating the mechanisms behind the effect. Rechecking is observed behavior, not a randomized intermediate variable, so the study cannot isolate mediation through rechecking from contradiction, salience, or reduced verification effort. Whether the effect generalizes across task populations and repeated model sampling is also untested.
• Turning certified environments into training signal. The paper positions the seeded, procedurally generated, verifier-graded environments as verifiable-reward candidates, and the corruption challenges as a map of where such verifiers could mis-reward an episode — leaving the actual RL training results unreported in the supplied content.

Target Audience

Benchmark and evaluation engineers building stateful agent test suites; agentic RL researchers who need trustworthy programmatic reward checkers; product and safety teams deploying computer-use agents in policy- or record-driven enterprise workflows; and methodologists interested in pre-registration, validity-first scoring, and evaluator audits for AI evaluations.

Authors’ abstract

Computer-use agents built on large language models (LLMs) work in software that can change while they act: an appointment moves, a policy is revised, a record is updated. Testing agents under such changes is harder than it looks. When an agent fails, the cause may be the agent, but it may also be a change that did not fire as intended, an update the agent never received, or a grader that scored the episode wrongly. We introduce EnvOS, a framework for building and certifying environments that change mid-task. Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence. We apply it to stale information, where a value the agent has already seen changes before the agent acts on it. In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown; the gain over the reminder holds after correcting for all tests (p = 0.00293). In insurance-claim approval, Opus 5 re-checks on its own and succeeds on 10/10, whereas Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests. Whether an agent caught a change thus varied with the model and the interface. We also stress-test the grading itself: on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks. Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.