OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

AI summary

Overview

Research area: Evaluation and reward modeling for computer-using agents (CUAs) — AI systems that operate web browsers, mobile apps, and desktop software by clicking, typing, and running commands.

Technical level: Advanced. The paper assumes familiarity with vision-language models, reinforcement learning reward signals, benchmark design, and judging/LLM-as-a-judge methodology.

Scope in one sentence: The paper builds OSReward, a cross-platform human-gold benchmark of 1019 computer-use trajectories for measuring how reliably VLM judges verify task completion, and releases OS-Shepherd, an open reward model trained on a 100K-sample corpus derived from the study's findings.

What This Paper Is About

Computer-using agents produce long trajectories of screenshots, actions, and reasoning, and something has to decide whether each trajectory actually completed its task — this verdict drives agent evaluation, training-data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can supply those verdicts at the scale required, so the field has quietly adopted vision-language models as judges, without ever systematically testing whether those judges are reliable. This paper builds a purpose-made benchmark to measure judge reliability across four platforms, shows that current judges share a systematic leniency bias, and closes the gap with an open, low-cost reward model.

Key Contributions

1. OSReward, a standardized cross-platform benchmark. Built from scratch on dedicated data infrastructure rather than reused rollouts: 1019 human-gold trajectories across web, mobile, Ubuntu, and Windows, executed by agents from four model families on human-verified instructions. It includes two nested variants — OSReward-Hard (284 challenge cases) and OSReward-Multi (440 successful trajectories with fine-grained alignment and efficiency labels).

2. The broadest evaluation of VLM judges for computer use to date. 27 vision-language models are benchmarked under one fixed protocol, spanning closed frontier judges, closed efficient tiers, large open-weight models, and small open VL models, with several run in thinking variants.

3. OS-Shepherd-100K, an open training corpus. More than 300K judge instances on cross-platform trajectories are curated into nearly 100K reasoning-annotated trajectory judgments, labeled without new human annotation by applying the findings of the judge study, with failure types annotated and analyzed.

4. OS-Shepherd-9B and OS-Shepherd-35B, open reward models. Trained with a two-stage recipe that builds accurate judging first and then directly targets the false-success failure mode, delivering rewards at 30–60× lower cost than the frontier.

Main Findings

• Even frontier judges miss the bar a training reward needs. On the full OSReward set, Claude-Opus-4-8 comes closest to the roughly 90% accuracy working bar at 89.7%; GPT-5.5 and Claude-Opus-4-6 sit just behind at 89.5%. The field spans about twenty points down to the small open VL models.

• Performance collapses on hard cases. OSReward-Hard drops every judge by 20–43 percentage points. The best judge loses a full twenty points to 69.7% — level with a constant always-fail predictor on this 30/70 success/fail split — the mean judge falls to 52%, and the lenient tail falls lower still. On balanced accuracy, such a constant predictor falls to 50%.

• A systematic leniency bias is shared across all model families. Plotting judges at (fail recall, success recall) shows a large lenient cluster with high success recall and low fail recall, plus a smaller strict cluster. Over-accepting an incomplete task accounts for two-thirds of all errors and is the leading error mode of every single judge, at no less than 48% of its mistakes. Pooled across judges, over-accepts outnumber over-rejects three to one, narrowing to roughly two to one for the strongest judges.

• Judges read the agent's narrative, not the screen. Dropping the per-step thought and action text costs 7.2 percentage points on average and flips 22.7% of verdicts — three times the churn of any visual change. Dropping only the chain-of-thought while keeping actions is far milder at 1.8 points and 11.6% flipped. This is the mechanism behind the leniency bias: a confident closing success claim fools a judge that leans on the narrative.

• Visual settings barely matter for aggregate accuracy. Swapping the five trailing screenshots for the last three, or for the first plus the last two, moves binary accuracy by less than half a point; removing the red click marker does not hurt at all. Sweeping trailing screenshots from one to sixteen produces two to three points of wander with no trend, with each judge near its own best at N=5 to 9. Each setting still flips 5–7% of individual verdicts, though.

• Model-side knobs do not buy reliability. Extra thinking helps weakest judges by a few points and almost nothing at the frontier. Re-running the same judge at T=0.7 flips 6–9% of verdicts, so visual-setting churn sits at or below sampling noise while text-ablation churn is far outside it. Top judges agree strongly (pairwise Cohen's κ ≈ 0.71) and herd on the same hard trajectories, so a top-3 majority vote beats the best single judge by about a point at several times the cost; an oracle accepting any pooled judge's correct verdict reaches 99%.

• Failures are structured by platform and type. Windows is the hardest platform to judge and mobile the easiest. Failure types that hinge on reading the screen (perception, then action) are markedly harder to catch than planning-and-reasoning failures — which dominate the data, being tagged on 86% of failed runs.

• Reliability can be bought but not afforded. Claude-Opus-4-8 costs roughly $100 to judge the full set once and holds 69.7% on OSReward-Hard; GPT-5.5 costs $45 for 67.3%. The best sub-$3 judge drops to 57.0% and the cheapest tier falls below 50%. On the full set the trade-off looks benign — about 3 points for a 42× price cut — but not on the hard set.

• Quality grading lags far behind outcome judging. On OSReward-Multi, the best judge falls from around 90% on the binary verdict to the low sixties. GPT-5.5 leads on macro-recall (alignment 58.7, efficiency 68.2, multi 63.5) and 66.7 AUC. The AUC–macro-recall gap shows judges rank levels better than they score them — discrimination exists but emitted thresholds are miscalibrated, worst on alignment, where judges default to the top rating.

• OS-Shepherd narrows the gap. OS-Shepherd-9B reaches 86.1% on OSReward and 60.2% on OSReward-Hard; OS-Shepherd-35B-A3B reaches 85.6% and 62.7% respectively, with balanced accuracy of 61.9 and 64.3 on the hard set. The models match commercial judges at 30–60× lower cost.

• The existing picture was already suspect. A pilot study finds even the best VLM judges disagree with existing benchmarks' own verifiers on roughly a quarter of desktop verdicts.

Methodology in Plain English

The researchers first argue that reusing trajectories from existing agent benchmarks would be a mistake: you could not tell whether a judge's error came from the judge or from a flawed run, and the existing gold labels inherit the noise of hand-written verifiers. So they build everything themselves, end to end.

They construct four environments — live websites with hardened Chromium browser workers, a Windows machine with about twenty everyday applications, an Ubuntu machine with about thirty applications and a typed pool of real files, and an Android emulator — and initialize them like lived-in machines with user profiles, seeded databases, browsing history, and distractor content hiding in the background. Annotators explore these environments and write instructions grounded in what the machines actually offer. Roughly 1500 candidates are authored and about 800 survive peer screening; every instruction is checked by an annotator other than its author.

Agents from four model families (Claude, Gemini, Kimi, Qwen) then execute each instruction, one to three backbones per instruction, so the capability spread yields genuine successes and genuine failures. An automatic pre-filter removes runs broken by anti-bot blocks, network failures, or frozen executions, so a fail reflects the agent failing, not the environment.

Labeling is deliberately heavy. Three annotators independently read every screenshot, thought, and action and issue a verdict; unanimous verdicts are final and disagreements escalate to a meta-review where two senior reviewers deliberate and issue the judgment. One strict rule applies throughout: an answer the agent did not obtain or verify through the environment counts as a fail even if it happens to be correct. Successes are additionally scored on alignment and efficiency; failures are tagged against a taxonomy spanning reasoning-and-planning, action, perception, and memory errors. Altogether the annotation costs roughly 800 human hours.

For the judging experiments, every one of 27 models runs under one fixed protocol: read the last N states with per-step reasoning and action text (N=5 by default), no task-specific harness, no tools, no step-level supervision, and emit a success/fail verdict. Accuracy is decomposed into success recall and fail recall, with balanced accuracy as their mean so the 43/57 class mix cannot inflate the score. The authors then perturb one component at a time — screenshots, text, marker, decoding, ensemble — to see what actually moves a verdict.

Finally, they turn the diagnoses into training data. Following the ablation that text history matters and the finding that judges herd, they select trajectories for large-scale training and label them without new human annotation, assembling OS-Shepherd-100K from their own collection pipeline plus filtered public corpora. On this corpus they train a 9B and a 35B reward model with a two-stage recipe, the second stage aimed directly at the false-success mode the study exposed.

Why This Matters

Impact on research: The paper argues that the whole computer-use pipeline — evaluation, data curation, and reinforcement learning — consumes a reward signal, and that the field has been consuming one without knowing how reliable it is. By separating judge error from trajectory error through purpose-built environments and multi-stage human gold labels, it gives the community a clean measuring stick instead of benchmark-inherited confounds. It also reframes the open problem: de-biasing a specific lenient failure mode is higher-value than ensembling or prompt tuning.

Real-world applications:

• Reinforcement learning for computer-use agents. Reliable verdicts at millions of calls per training run is exactly what a frontier judge's per-call cost rules out, and what the OS-Shepherd models are designed to supply.
• Trajectory data curation and mining. Filtering agent-generated corpora for genuine successes requires judges that do not accept a narrated-but-unfinished run.
• Desktop and mobile automation quality assurance. Teams deploying CUAs need to know whether a run truly changed the environment, not whether the agent said it did.
• Benchmark construction for GUI agents. Standardized cross-platform judging supports reproducible task evaluation where hand-written verifiers do not exist.

Industry relevance: The cost-accuracy frontier is explicitly framed as an economic constraint: the judges that hold roughly 70% on hard cases cost $45–100 to judge the full set once, which does not scale. The released corpus, model checkpoints, and open-weight 9B/35B models make a training-scale reward signal affordable to academic and smaller-scale groups, not just frontier labs.

Future Directions

• Confidence-weighted, soft-label reward modeling. Because judges herd and a majority vote cannot identify which judge to trust on which trajectory — while an oracle over the pool reaches 99% — the authors argue for soft-label, confidence-weighted reward modeling over hard votes, and leave it to future work.
• Explicit completion verification in the judging prompt. Since over-accepting an incomplete task is the dominant error and the text-history ablation names the mechanism, the authors suggest prompting the judge to verify completion explicitly may help more than ensembling, and leave this to future work.
• Improving quality grading, especially alignment. No judge grades alignment or efficiency well, and judges default to the top alignment rating; closing the AUC–macro-recall gap on this axis is an open problem. The 35B OS-Shepherd model recovers part of the gap while the 9B stays near the constant-level baseline.
• Transferring de-biasing to unseen benchmarks. The paper reports that held-out evaluations confirm the de-biasing transfers, and that a generalization study measures judge–verifier disagreement at scale, but the design of reliable CUA reward at scale remains an ongoing question.

Target Audience

Researchers and engineers working on computer-using agents, GUI and web agents, reinforcement learning from model-based reward, and LLM-as-a-judge evaluation. It is most useful to those building or auditing reward models and trajectory-filtering pipelines, benchmark designers who need to know the pitfalls of reusing existing rollouts and verifiers, and practitioners deciding whether a proprietary judge's cost is justified against an open-weight alternative. Readers should be comfortable with recall/precision-style metrics, balanced accuracy, AUC, and how reward signals are consumed in RL training.

Authors’ abstract

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.