Remote Labor Index: Measuring AI Automation of Remote Work

AI summary

Overview

Research area: Machine learning evaluation and AI labor economics — specifically, benchmarking the ability of AI agents to complete real, economically valuable remote freelance work.

Technical level: Intermediate. The core idea and metrics are accessible to non-specialists, while the Elo/Bradley-Terry scoring details and qualitative failure taxonomy require some familiarity with AI benchmarking.

Scope: The paper introduces and analyzes the Remote Labor Index (RLI), a 240-project benchmark of real freelance deliverables used to measure how much of the remote labor economy current AI agents can actually automate.

What This Paper Is About

AI systems have improved rapidly on knowledge and reasoning benchmarks, but it has been unclear whether those gains translate into the ability to perform economically valuable work. The authors build RLI, a benchmark of end-to-end remote freelance projects sourced from real marketplaces, where each project includes a client brief, input files, and a gold-standard human deliverable. The goal is to measure AI automation capability against real commissioned work, rather than against simplified synthetic tasks.

Key Contributions

1. The Remote Labor Index (RLI) itself: a benchmark of 240 real-world projects spanning 23 categories of work from the Upwork taxonomy (out of Upwork's 64), each grounded in an actual economic transaction with a human gold-standard deliverable.
2. A standardized evaluation protocol and open-source platform: because automated grading is not yet feasible for these artifacts, every AI deliverable is judged manually by trained evaluators using a holistic 3-point scale, with a web platform that natively renders dozens of file formats.
3. Four complementary metrics for automation: Automation rate, Elo, Dollars earned, and Autoflation — the latter two explicitly quantifying the economic value and cost reduction associated with AI completion.
4. An empirical baseline of frontier agents near the floor: a pre-release evaluation of six AI agents showing the best automation rate is 2.5%, plus a qualitative taxonomy of failure modes and a set of projects where AI matched or exceeded humans.

Main Findings

• Automation rate is near the floor. The highest-performing agent was Manus at 2.5%, followed by Grok 4 at 2.1%, Sonnet 4.5 at 2.1%, GPT-5 (CLI) at 1.7%, ChatGPT agent at 1.3%, GPT-5 (CUA) at 0.8%, and Gemini 2.5 Pro at 0.8%. All models solve less than 3% of tasks.
• Elo rankings show relative progress but all models fall short of humans. Scores were Manus 509.9, Grok 4 468.2, ChatGPT Agent 454.3, Sonnet 4.5 441.7, GPT-5 (CLI) 436.7, GPT-5 (CUA) 431.6, and Gemini 2.5 Pro 411.8, against a canonicalized human baseline Elo of 1,000. A difference of 400 in score corresponds to 10:1 odds of winning.
• Dollars earned are a small fraction of the total project value. Against a maximum possible $143,991, Manus earned $1,720, Sonnet 4.5 $1,280, GPT-5 (CLI) $1,180, Grok 4 $858, GPT-5 (CUA) $858, ChatGPT agent $520, and Gemini 2.5 Pro $210.
• RLI projects are large, difficult, and economically grounded. Mean human completion time was 28.9 hours with a median of 11.5 hours — more than 2x previous comparable benchmarks. Average project cost was $632.6 with a median of $200. Projects represent over 6,000 hours of real work valued at over $140,000. Some projects cost over $10,000 and took over 100 hours.
• Failure modes cluster into four categories (not mutually exclusive): poor quality (45.6%), incomplete (35.7%), corrupted files (17.6%), and inconsistencies across deliverable files (14.8%). Representative failures include an 8-second video where 8 minutes was requested, child-like geometric drawings, and digital floor plans that do not match supplied sketches.
• Successes were concentrated in a narrow slice of work. Across all tested models, AI matched or exceeded human baselines mainly on audio editing, mixing and production tasks, image generation, report writing, and code for interactive data visualization.
• Evaluation is reliable but not perfect. Inter-annotator agreement was 94.4% for the automation rate metric. Ternary inter-annotator agreement for Elo was 56.9%, versus a 33.0% random chance, with a 5.9% probability of hard disagreements. Average evaluation time was 11.4 minutes for automation rate and 17.4 minutes for Elo.
• Scaffolding matters. GPT-5 performed better with a command-line (CLI) scaffold than a computer-use (CUA) scaffold on both Elo (436.7 vs 431.6) and automation rate (1.7% vs 0.8%), suggesting current models do not fully exploit computer-use environments.

Methodology in Plain English

The authors collected real freelance projects rather than authoring tasks themselves. They started from the 64-category Upwork taxonomy, filtered down to 43 eligible categories by removing work that required physical labor, waiting to evaluate, or could not be rendered on a web platform, then posted job listings and hired freelancers to sell samples of their prior work. That yielded 207 projects from platform sourcing, 7 from long-tail sourcing, and 33 from independently identified digital work whose authors granted permission and supplied timing and pricing information. In total they recruited 358 freelancers with verified Upwork accounts who averaged 2,341 hours worked, 89 prior jobs, and $23,364 in total Upwork earnings, and who were paid between $15 and $200 per project (average $41) across 550 initial projects. Multiple rounds of review, cleaning, and standardization reduced this to the final 240 projects, of which 230 form a private test set and 10 are released publicly.

To evaluate, agents were given the brief and input files with no mandated execution environment, but received an "evaluation compatibility prompt" listing supported file formats. Six agents were tested — ChatGPT agent, GPT-5, Claude Sonnet 4.5, Grok 4, Gemini 2.5 Pro, and Manus — using a Scale AI computer-use scaffold where supported and OpenHands otherwise. Human evaluators compared each AI deliverable to the human gold standard on a 3-point holistic scale (failure / acceptable / exceeds), with majority voting across three independent evaluations. A separate pairwise Elo evaluation compared two AI deliverables per project on project-completion and overall-quality dimensions, with the human deliverable shown as a reference. Elo scores were fitted with Bradley-Terry utilities using 100 bootstrap samples for 95% confidence intervals, with each model pair compared on at least 10 projects (median 25).

Why This Matters

Impact on research: RLI argues that progress on narrow agent benchmarks (software engineering, basic computer use) is a poor proxy for economically valuable automation. It provides a benchmark explicitly constructed to track that gap, and an evaluation design and metrics vocabulary (Autoflation in particular) other researchers can build on.

Real-world applications:

• Workforce and policy planning: governments and labor economists can use a grounded automation rate to forecast which categories of remote work are actually at risk.
• Freelance marketplaces: platforms can estimate which job categories AI can currently serve and at what effective cost.
• Enterprise procurement of AI agents: companies can calibrate expectations about what frontier agents can deliver end-to-end today, rather than extrapolating from research benchmarks.
• Model and agent developers: the failure taxonomy (corrupted files, incomplete work, quality, inconsistency) gives concrete targets for scaffold and tooling improvements.

Industry relevance: The paper comes jointly from the Center for AI Safety and Scale AI, and its headline result — the best agent automating only 2.5% of real freelance projects while earning $1,720 of $143,991 available — is a direct counterweight to claims that current agents are close to replacing remote knowledge work.

Future Directions

• Extending coverage beyond the current scope. RLI excludes work requiring client interaction (such as tutoring), team-based work (such as project management), and other categories that failed the inclusion criteria, so an agent scoring 100% on RLI may still underperform humans on unmeasured work.
• Repeated measurement over time. The authors frame RLI as a tracking instrument for AI-driven automation and plot Autoflation over time as new models are released. Sustained measurement as new frontier models appear is the natural next step.
• Better scaffolds and vertical integration. The GPT-5 CLI-versus-CUA gap suggests more tightly integrated model-scaffold designs could unlock additional performance in computer-use settings.
• Scaling and hardening evaluation. The authors hypothesize that inter-annotator agreement on the automation rate will fall as AI deliverables become more complex, implying a need for more experienced evaluators, longer evaluation times, or new evaluation methods as deliverables improve.

Target Audience

AI evaluation and benchmark researchers; labor economists and policy analysts studying automation; product and strategy teams at AI labs and agent startups; and freelance-platform operators or enterprise buyers assessing how much remote work AI can currently absorb. Readers looking for a single defensible empirical number for "how much remote labor can AI automate today" will find the paper's 2.5% figure and its economic framing most useful.

Authors’ abstract

AIs have made rapid progress on research-oriented benchmarks of knowledge and reasoning, but it remains unclear how these gains translate into economic value and automation. To measure this, we introduce the Remote Labor Index (RLI), a broadly multi-sector benchmark comprising real-world, economically valuable projects designed to evaluate end-to-end agent performance in practical settings. AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2.5%. These results help ground discussions of AI automation in empirical evidence, setting a common basis for tracking AI impacts and enabling stakeholders to proactively navigate AI-driven labor automation.