COBOL migration from 5.4 to 86.6 percent — OpenAI's figures still lack independent verification
OpenAI claims its newest models can now build and migrate software in "low-resource" languages like COBOL, Fortran and Visual Basic with a leap in accuracy — but every number is the company's own, and independent testing has yet to happen.
OpenAI announced on October 19, 2026 that its latest model improvements deliver "stronger capabilities for building and migrating software in low-resource programming languages such as COBOL, Fortran, and Visual Basic" (OpenAI). The claims are large: a COBOL score said to have risen from 10.9 to 72.1 percent, and a COBOL migration evaluation in which the company reports 86.6 percent accuracy where its predecessor scored 5.4. All of the figures come from OpenAI's own evaluations, and as of the announcement, no independent verification exists.
How it is measured
OpenAI built two in-house evaluations. The first, MultiLangBench, tests whether a model can build a SQLite-compatible database engine from scratch in 20 different programming languages. The second is a COBOL migration evaluation that tests whether the model can move existing programs to another language "while preserving behavior," according to the company. The full methodological criteria for the migration evaluation have not been published in full in the announcement.
The bar is strict, and for good reason: OpenAI itself points out why the task is hard. COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested. A rewrite that rounds those values instead can run without errors "while producing different financial results." In other words, a migration that looks successful can change balances and interest calculations in banks' systems.
The numbers, as OpenAI itself presents them
On MultiLangBench, at maximum reasoning effort, OpenAI reports that GPT-6 Astra achieves an average test pass rate of 87.4 percent across the 20 languages, up from 26.8 percent for GPT-5.6 Sol. The improvements are said to cover every language tested:
- COBOL: 10.9 → 72.1 percent (still the hardest)
- OCaml: 25.6 → 85.8 percent
- Fortran: 21.4 → 83.5 percent
- Visual Basic: 27.5 → 89.4 percent
On the COBOL migration evaluation, the company reports that GPT-6.1 Sol achieves 86.6 percent accuracy at low reasoning effort, versus 5.4 percent for GPT-5.6 Sol — at 39 percent lower cost. At higher reasoning effort, accuracy is said to reach 97.9 percent, which OpenAI describes as near saturation of the evaluation.
In addition, the company states that GPT-6 Luna improves on every language tested at an estimated average API cost of $0.77 per sample, versus $3.48 for GPT-5.6 Sol — roughly 78 percent lower. The cost figures are likewise the company's own estimates.
Why legacy code is a big market
OpenAI justifies the push with COBOL's continuing role in societal infrastructure: according to the company, which cites external sources via links in the announcement, the language supports more than 40 percent of online banking systems, 80 percent of in-person credit card transactions and 95 percent of ATM transactions. The same source base, according to OpenAI, shows that Fortran is used in weather forecasting and safety analyses of nuclear reactors — these are also the company's rendering of external sources, not independently verified by AIMag.
CryptoBriefing, which covered the announcement, quotes an unattributed estimate that 800 billion lines of COBOL remain in production, in systems at banks, government agencies and large enterprises. The figure should be read as an estimate without a stated source, not as an established fact.
Historically, research benchmarks have shown low Pass@1 and compilation rates on COBOL tasks — from single digits to around 40 percent, according to CryptoBriefing's review. Against that backdrop, OpenAI's claim of 72 percent on COBOL and near saturation on migration is a dramatic jump — which is precisely why independent testing matters.
Competition and skepticism
OpenAI is not alone in this market. According to CryptoBriefing's secondary coverage, Anthropic released COBOL-related updates in February 2026, and IBM offers watsonx Code Assistant, a tool built to translate COBOL to Java. The same information has not been confirmed by a primary source.
CryptoBriefing also points to the core of the skeptic's case: OpenAI "has not published comparable benchmark figures for the latest update in this announcement, so independent testing will matter" (translated from English). In other words, no third party can yet confirm the numbers.
What can be verified now
One thing distinguishes this announcement from many other benchmark reports: OpenAI is releasing the evaluation itself. According to the company, the MultiLangBench evaluation setup and test suite are being published on GitHub, along with the results for each language. That means independent researchers can in principle run the evaluation themselves and test the claims of 87.4 and 72.1 percent.
OpenAI has also opened the floor to developer input on which languages should be prioritized next — an invitation suggesting the company sees this as an ongoing program, not a one-off update.
Open questions
Several things remain that the announcement does not answer. First, the lack of independent verification noted above. Second, the full methodological criteria for the migration evaluation are not available in the announcement, so exactly how "preserved behavior" is measured remains unclear for now. Third, it is unclear which of the mentioned models are available to customers through which products — the announcement ties GPT-6 Astra, GPT-6 Luna and GPT-6.1 Sol to the evaluations but does not explain how the capabilities reach end users.
The concrete question over the coming weeks is therefore not whether the numbers OpenAI presents are impressive — they are, on paper — but whether they hold up when others are allowed to run the same tests.