The Pulse
Toptal Launches ExpertCrowd and a 100-Task SWE Benchmark
Toptal announced ExpertCrowd alongside a software-engineering benchmark built from 100 tasks across 87 open-source projects and 11 programming languages. The benchmark’s results show varied performance across seven AI models, with Toptal ke
AI.info Team · Written with AI ·
ExpertCrowd had already generated more than $5 million in recurring revenue before Toptal announced the platform on October 13, 2026. The company says the service connects AI developers with specialized human experts for model training, data generation, annotation and evaluation. Toptal also released ExpertCrowd-SWE, a coding-agent benchmark built from 100 tasks spanning 87 open-source projects and 11 programming languages.
A specialist network behind the platform
Toptal says ExpertCrowd draws on a network of more than 30,000 domain experts in over 140 countries. The company identifies more than 2,000 of them as AI and big-data specialists, 12,000 as professionals with advanced degrees, and 1,000 as PhD-level AI specialists. Its platform supports multimodal annotation, customizable workflows and live project tracking, while an in-house sourcing system matches experts to projects.
The company says it developed that sourcing engine using 15 years of proprietary data on finding, screening and matching professionals. Toptal describes its AI research team as including more than a dozen PhDs and researchers from institutions including Harvard, EPFL and the University of Liège. The press release names Scale AI, Sterling Labs and Vetto among companies it says had worked with ExpertCrowd before launch.
100 software tasks, tested five times each
ExpertCrowd-SWE asks AI agents to resolve software issues in codebases they have not seen before. Engineers from Toptal’s network wrote the tasks and grading rubrics; the company says its AI researchers reviewed the methodology and results. Each model attempted each task five times under the same tools and a limit of 1,000 steps per attempt, producing 3,495 valid runs in total.
The benchmark page reports Claude Opus 5.5 at 67.3%, GPT-6 Astra at 63.7%, and Claude Fable 5.1 at 46.5%. Gemini 3.8 Flash scored 40.0%; Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 scored between 32.2% and 33.5%. Toptal says the gap between the two leading models was not statistically significant.
Those scores come with a qualification: the company says it selected the final 100 tasks from a larger set based on how difficult six of the models found them. Gemini was excluded from that selection because it lacked a full set of valid runs on some tasks. The results therefore describe performance on a deliberately challenging set, not a random sample of everyday software work.
Failures matter more than a single score
The benchmark’s task-level results show why an overall percentage can conceal weaknesses. On one task involving canceled or interrupted jobs in the Go library Asynq, Opus 5.5 and Grok 4.6 earned no credit across their runs, while DeepSeek V4.1 Flash averaged 75.6%. Toptal also reports that no task was fully solved by all seven models.
“The fact that our SWE benchmark results expose the most deficiencies relative to any other published benchmark on leading models such as Opus 5.5 and Fable 5.1 demonstrates our ability to push frontier AI forward,” said Taso Du Val, founder and CEO of Toptal.
The benchmark makes 24 tasks public and keeps 76 private, according to the company’s research page, allowing some results to be inspected while retaining a set that is less exposed to training-data contamination.
ExpertCrowd plans to move beyond coding
Toptal says ExpertCrowd is intended to help teams use benchmark results to target weaknesses with expert-created training data or reinforcement-learning tasks. The company says customers can retrain models themselves or use a managed engagement. It also says its next benchmark will focus on management consulting, with tasks involving documents, financial data and recommendations that may not have one correct answer.
The launch combines a commercial service with a benchmark whose scores Toptal says expose gaps in current coding agents. Its initial public dataset is a limited window into that test: 24 tasks can be reproduced and examined, while the remaining 76 are reserved from public release.