The Pulse
Real-SWE Puts Coding Agents on Private Enterprise Code
Specific Labs' Real-SWE benchmark evaluates coding agents on private production codebases and finds the top configuration resolved 38.8% of scored attempts.
AI.info Team ·
Specific Labs has released Real-SWE, a benchmark that asks coding agents to modify private production software rather than public repositories or synthetic exercises. Across 10 tasks, eight model-and-agent configurations and eight independent runs per task, the benchmark records 640 scored rollouts.
Fable 5.1 running through Claude Code ranks first with a 38.8% resolution rate. GPT-6 Astra through Codex CLI follows at 33.8%, while Gemini 3.8 Flash through Gemini CLI reaches 31.2%. Specific Labs presents the results as a test of whether agents can work inside the company-specific systems, conventions and business rules that define ordinary software engineering.
Specific Labs Moves Beyond Public GitHub Issues
Real-SWE uses tasks drawn from private production codebases licensed from real companies. The participating systems work in isolated sandboxes that can include Docker, Kubernetes, GitHub, PostgreSQL, MySQL, MongoDB, Redis, AWS emulators and business tools such as Linear, Slack, Intercom, Google Drive, email and ClickUp.
The benchmark's sample codebases include a Luma and Partiful competitor with more than 200,000 users and a top-100 App Store ranking, a consumer fintech platform that processes more than 100,000 bank statements, and enterprise AI sales platforms built around complex business workflows. Specific Labs does not name the companies or publish the private repositories.
Tasks cover work such as billing, tax calculation, customer migration, data storage, identity and analytics. One example asks an agent to correct invoice tax behavior across several business configurations, external tax-authority services and ledger requirements. The task requires more than changing a function: the agent must infer how the surrounding application handles environments, exemptions, addresses, settlement and reporting.
Agents Edit 11 Files, Then Miss the Requirements
A median Real-SWE instruction contains 1,742 characters, and the reference solutions edit 11 files. Specific Labs compares that figure with six files for FrontierCode and DeepSWE, two other coding-agent evaluations. The benchmark's authors say the wider edit surface reflects changes that cross application services, infrastructure and business tools.
Failure analysis points to incomplete understanding rather than simple syntax errors. Missed requirements form the largest failure category for several configurations: 67.2% of Grok 4.6's failed runs, 53.8% of Kimi K3's and 38.6% of GLM 5.3's. Agents also make unverified assumptions, wire otherwise reasonable fixes into the wrong part of a system or introduce regressions.
Gemini 3.8 Flash records the highest share of integration errors at 49.1% of its failed runs. Fable 5.1, the overall leader, posts a 34.7% integration-error rate among failed runs. The figures show why passing a narrow test is not enough when the requested change depends on several services and established company behavior.
Shorter Runs Do Not Lose the Contest
Real-SWE also tracks how long agents work. Rollouts lasting less than 10 minutes fail 70 times out of 98, or 71.4%. Longer runs fail 398 times out of 542, or 73.4%. The result does not show a performance advantage for simply allowing an agent more time.
Specific Labs estimates the cost of a rollout at between $2.50 and $6.96. Gemini 3.8 Flash is the least expensive configuration in the published table, while Fable 5.1 is the most expensive. Cost and accuracy do not line up: Fable leads the benchmark, but the cheaper Gemini configuration ranks third.
The benchmark's authors frame the test with a direct question: Can a coding agent actually do the work of a software engineer in the real world?
Siddhant Paliwal, a co-author of the Real-SWE report, and his co-authors write:
Our results show us that we're far from that reality.
Why Private Code Changes the Result
Public coding benchmarks give models a familiar setting: an open-source repository, a structured issue and a solution that can often be related to material available on the internet. Real-SWE removes much of that familiarity. Its tasks come from code and workflows that the participating models could not inspect during training, while the instructions leave agents to discover implementation details inside each environment.
Specific Labs says 99% of tokens in real-world enterprise systems sit outside the data available to frontier models. The claim is not a measurement of all enterprise software, but it captures the benchmark's central premise: production code contains local rules that are rarely documented in a standard issue description and cannot be solved reliably through pattern matching alone.
The company also says its verifiers are inspired by existing test suites or use those tests verbatim, with verifiers injected during grading. That approach keeps private repositories and reference solutions out of public view while still testing whether a submitted change preserves the behavior that the original engineers expected.
Real-SWE Tests the Part Public Scores Leave Out
Real-SWE does not establish that any model is incapable of useful software work. Fable 5.1 solves nearly four in 10 scored attempts under the benchmark's rules, and individual tasks show much higher resolution rates than the overall average. GPT-6 Astra, for example, resolves eight of eight runs on the benchmark's multi-region sweep task, while Gemini 3.8 Flash does the same on that task.
The broader result is less flattering. No configuration solves every task, and several agents fail repeatedly on billing migrations, API token metering, datastore measurements, tax jurisdiction and analytics-stream work. For companies evaluating coding agents, the published evidence points to a practical requirement: review must focus not only on whether an agent writes code that passes tests, but also on whether it found every business requirement and changed the live execution path.
Real-SWE's first leaderboard therefore measures a narrower but more demanding question than public benchmark scores: how often can an agent make a correct change inside software it has never seen, under rules it must reconstruct from the surrounding system? On the published results, the answer tops out at 38.8% for the Fable 5.1 and Claude Code configuration.