The Pulse
Cognition Adds GPT-6 Astra to Devin’s Autonomous Testing
Cognition has added OpenAI’s GPT-6 Astra to Devin Desktop, Devin CLI and Devin Cloud. The company says Astra improves Devin’s autonomous software testing with broader test coverage, clearer reports and video evidence.
AI.info Team ·
“The future of async development is verified.”
Ido Pesok, engineer, Cognition
Cognition is putting OpenAI’s GPT-6 Astra inside Devin’s testing workflow, giving its autonomous software engineer a new model for checking whether generated code works in real applications.
The company announced the integration on September 3. GPT-6 Astra is now available in Devin Desktop and Devin CLI, and Cognition says it is also part of the model mixture used by Devin Cloud.
GPT-6 Astra joins Devin across three products
Cognition says Astra can be used directly in Devin Desktop and Devin CLI, while Devin Cloud can select it as part of the system’s model mixture. The company presents the release as both a coding-model update and a testing improvement.
On FrontierCode 1.1, Cognition’s proprietary benchmark for real-world engineering tasks, Astra surpasses Anthropic’s Fable 5.1 and ranks within 0.4 points of Fable 5. Cognition also says Astra costs 64% less than Fable 5.1 in the configurations shown in its announcement.
FrontierCode 1.1 grades more than whether a model produces functioning code. Cognition says the benchmark measures quality and mergeability, making the result relevant to teams that care about whether an agent’s changes can enter an existing codebase without extensive cleanup.
Testing is the narrower target
The most specific claim concerns Devin’s testing capabilities. Cognition says GPT-6 Astra reaches state-of-the-art results on its internal testing benchmark, where it produces more comprehensive tests, clearer reports and more usable video evidence.
Devin’s testing mode is designed to operate an application rather than stop after writing or reviewing source code. The agent can create a test plan, launch the software in a cloud environment, interact with the interface, capture screenshots and record the run. Its final output includes a report that identifies important moments and indicates which assertions passed, failed or were left untested.
That workflow matters because an agent-generated pull request can appear plausible while still failing in the running product. Cognition’s earlier engineering account describes Devin testing changes by starting the application, clicking through key flows and checking whether the expected behavior appears. The company says its engineers have run 10 to 20 Devins in parallel, each with its own development server.
Why Cognition is routing a separate model to verification
Writing code and testing software require different kinds of model behavior. Code generation rewards repository navigation, planning and implementation. Testing requires the agent to read screenshots, track interface state, recognize whether a change produced the intended result and decide what action to take next.
Cognition’s Ido Pesok wrote that the company began routing the testing phase to different models because “testing leans on different strengths than writing code.” In the same account, Pesok describes failure modes that included testing unrelated parts of a product, getting stuck during setup and missing the behavior a pull request was meant to change.
To reduce those failures, Devin first writes a test plan grounded in the source code. The agent then annotates its timeline with setup notes, named tests and explicit pass, fail or untested assertions. Cognition says those annotations reduce the tendency to treat an unexpected result as a successful test.
The evidence problem for autonomous coding
Cognition’s release places GPT-6 Astra inside a broader attempt to make asynchronous software work easier to trust. Devin can already be triggered by events, automations, schedules and other Devin sessions. As more work arrives without an engineer watching each step, a pull request alone offers less information about whether the change behaves correctly in practice.
Video evidence does not eliminate the need for review. A recording can show that an agent followed a visible path without proving that it covered every important case, handled unusual inputs or avoided regressions outside the demonstrated flow. Cognition’s announcement does not publish the size, composition or score breakdown of the internal testing benchmark, so its claims about Astra’s testing performance cannot be independently reproduced from the release.
The public results do establish the product decision: Cognition is assigning GPT-6 Astra to a role that sits after implementation and before human approval. The company is betting that a model with stronger computer-use abilities can make Devin’s test reports more useful without requiring an engineer to manually replay every change.
What the announcement actually changes
GPT-6 Astra does not remove human review from Devin’s workflow. Cognition’s announcement says the model is available within the product suite and improves the quality of automated testing; it does not claim that Devin can certify its own code without oversight.
The immediate change is more practical. Devin can use Astra to exercise an application, produce a clearer account of what happened and attach visual evidence to the result. For teams already using Devin to generate pull requests, that shifts part of the review burden from inspecting code alone to examining a structured test run and deciding whether the evidence is sufficient to merge.
That distinction is the substance of Cognition’s release: GPT-6 Astra is not merely another model option in Devin’s editor. Cognition is using it to strengthen the point where autonomous coding meets verification, with the company’s own benchmark and internal testing results as the evidence it has published so far.