The Pulse
OpenAI’s GPT-6 Astra Scores 87.4% Across 20 Coding Languages
OpenAI says GPT-6 Astra achieved an 87.4% average test pass rate on MultiLangBench, which measures software-building across 20 programming languages. The company also reports improved COBOL migration results, while warning that production r
AI.info Team · Written with AI · fact-checked ·
OpenAI says its GPT-6 Astra model passed an average 87.4% of tests in a new coding evaluation spanning 20 programming languages, including COBOL, Fortran and Visual Basic. The company published the results on October 19, alongside a separate evaluation of models migrating COBOL programs to newer languages. The tests target two kinds of work: building software in less commonly used languages and preserving the behavior of existing business programs.
SQLite tests span 20 languages
OpenAI’s MultiLangBench asks models to build a SQLite-compatible database engine from scratch in a specified language. The model must organize a codebase, manage state, handle errors and add features without breaking existing behavior. Adapted tests from SQLite’s test suite check the output, giving OpenAI a common measure across languages.
At maximum reasoning effort, GPT-6 Astra recorded an average test pass rate of 87.4%, compared with 26.8% for GPT-5.6 Sol. OpenAI says Astra improved in every language tested, scoring above 83% in 19 of the 20; COBOL was the hardest, at 72%. Those results combine mainstream languages with less widely used ones, so the average is not a score for low-resource languages alone.
GPT-6 Luna offers a lower-cost alternative on the same benchmark. At maximum reasoning effort, it achieved a 40.3% average pass rate, with an estimated average API cost of $0.77 per sample, versus $3.48 for GPT-5.6 Sol. OpenAI also says it is releasing the benchmark setup, test suite and language-level results on GitHub, allowing developers to inspect the tasks and compare results for individual languages.
COBOL migration tests check behavior, not just syntax
In the second evaluation, models translate COBOL programs into C++, Java, Kotlin, Python or Rust. OpenAI tests whether the rewritten programs preserve observable behavior using hidden tests that require byte-for-byte matching outputs. A migration counts as successful only when every test passes.
GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting; OpenAI says the newer model cost 39% less on this evaluation. At higher reasoning effort, GPT-6.1 Sol reached 97.9%, close to the evaluation’s ceiling. The results are specific to OpenAI’s test and do not establish how the model would perform on any particular company’s codebase.
OpenAI flags the gap between a benchmark and a live system
The company frames the work around maintenance problems reported by developers and enterprise customers: incomplete documentation, departed original developers and business rules recorded only in code. In financial software, a rewrite can run successfully yet change results if it mishandles details such as COBOL’s fixed-point arithmetic. OpenAI’s test treats preserving those details as part of migration success.
OpenAI says the latest models are available through its API for coding tasks. But the announcement cautions that production migrations still require expert review and testing against a system’s business rules and dependencies. The benchmark measures progress on defined tasks; it does not remove the need to verify that a rewritten system behaves correctly in its operating environment.