The Pulse
Mercor Finds Benchmarks Cover Little of U.S. Work
Mercor estimates that 90% of U.S. working hours involve tasks with little or no coverage in its sample of 29 AI benchmarks. The company’s map links benchmark prompts to Labor Department occupations and tasks, while warning that its estimate
AI.info Team · Written with AI ·
About 90% of American working hours fall on tasks with little or no coverage in the AI benchmarks Mercor examined. The estimate does not mean AI cannot do that work; it means existing tests offer little evidence about how well models handle it.
Mercor’s new benchmark map compares 29 evaluations with the U.S. Department of Labor’s O*NET catalog, which lists 923 occupations and 18,797 tasks. “Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks,” the company writes.
Five prompts set the coverage threshold
Mercor counts an O*NET task as covered when its language-model judge links it to at least five prompts from the 29-benchmark sample. One prompt can map to multiple tasks. The company estimates the share of working hours tied to covered tasks by combining its coverage figures with prior estimates of how workers’ time is distributed across O*NET tasks.
The sample emphasizes agentic evaluations: tests that ask models to carry out tasks rather than answer questions or advise a user. Mercor also favored publicly available benchmarks with at least 25 tasks, economic relevance and unsaturated difficulty. It capped how many benchmarks could come from each job family, aiming to avoid the concentration of software and research tests.
Under that definition, only 208 of the 923 occupations have even one task represented by five or more prompts. Six of the 22 job families have no occupation that meets the threshold. Mercor says the gaps are especially visible in smaller, middle-income knowledge-work roles, including legal secretaries and court clerks.
Lawyers get tested; clerks largely do not
The map shows a concentration of tests in software engineering and financial analysis, while truck driving and gas-plant operation receive little representation. Tasks in occupations with larger wage bills also appear more often, even after Mercor accounts for whether tasks are considered exposed to AI.
AI-exposed tasks are about six times more likely to appear in the benchmark sample than tasks rated unexposed. Within occupations, tasks rated most important are roughly six to ten times more likely to be represented than those rated least important. Mercor’s example: benchmarks cover AI-exposed lawyers’ work far more heavily than similar exposure-rated tasks performed by legal secretaries or court clerks.
The company’s figures describe what its selected tests measure, not the full capabilities of current models or the amount of work AI has already changed. An occupation missing from the benchmark sample may still include tasks that AI can perform; the map simply cannot support a strong estimate of model performance on them.
Mercor calls the estimate provisional
Mercor says the 29 evaluations do not form a complete or fully representative sample of public, unsaturated, economically relevant benchmarks. An incomplete sample may push the estimated share of uncovered work upward, and the company says it is difficult to put a bound on that bias. Its selection rules also involve judgment: “economic realism,” for example, is hard to define precisely.
The company checked its task-matching process against human reviews of its own APEX benchmarks. Experts reviewed 1,013 of 1,022 prompts across APEX-1, APEX-Agents and APEX-Accounting, producing 1,763 judgments. Reviewers accepted the model’s first-choice task mapping in 76.7% of judgments; counting cases where the correct answer appeared elsewhere on the model’s list raised that figure to 85.4%. Those checks cover APEX prompts, not every benchmark in the full index.
Mercor also warns that its unsaturated-benchmark rule may exclude easier tasks, while including saturated tests could make coverage look more meaningful than it is if those tests no longer reflect real work. It says neither choice is ideal. A separate obstacle is physical work: the company cites an estimate that only 37% of U.S. jobs can be done entirely from home, leaving many tasks that current digital benchmarks do not readily capture.
Coverage is not the same as job impact
The map addresses a measurement problem, not whether AI will replace or augment particular workers. Mercor says current models are mostly used by people rather than to automate whole jobs end to end, so useful evaluations would also need to measure how much AI improves human productivity. The company says it has early work in that area but no method that scales.
For employers, workers and policymakers, the immediate implication is narrower: widely cited model scores may reflect a limited set of economically important tasks, while leaving much of ordinary work untested. Mercor’s 90% estimate will change as it adds benchmarks, but the present figure rests on 29 evaluations and a threshold of five prompts per task.