Skip to content
AI.info

The Pulse

Sakana AI researchers release Japanese finance benchmark in preprint

JFinBench uses 321 examples from six Japanese financial certification exams to test language models. Researchers found a gap between the highest-scoring proprietary and open-weight models, even when the models had access to tools.

Sakana AI researchers release Japanese finance benchmark in preprint

JFinBench tests language models on 321 examples drawn from six Japanese financial certification exams, covering accounting, tax, financial planning and banking. The benchmark, described in a Nov. 13 preprint by researchers at Sakana AI and the Institute of Science Tokyo, includes a process for converting new exam editions into evaluation material.

The researchers report that proprietary models Claude Fable 5 and GPT-6 Astra scored above 93% overall, while the strongest open-weight model tested, Kimi K3.1 Pro, trailed the top model without access to tools. Giving the models terminal and web access narrowed the gap but did not eliminate it.

Exams as test material

The exams were held between May and August 2026 and include the CPA short-answer exam, the Certified Public Tax Accountant exam, financial planning and financial counter services tests, and bookkeeping exams. Alongside multiple-choice questions, JFinBench includes computed answers, answer forms, essays and 12 questions that depend on images.

The researchers argue that country-specific benchmarks are needed because financial expertise depends on local rules and practices. They also note that older exam questions may reflect rules that no longer apply, and that published questions can enter training data, making scores potentially reflect memorization.

For grading, the team uses different methods for different answer formats. A language-model judge checks short-answer equivalence, awards partial credit for answer forms and assesses whether essays cover key points.

How the models scored

The researchers evaluated 12 recent language models. In the tool-free tests, Claude Fable 5 scored 95.8%, GPT-6 Astra scored 93.6% and Kimi K3.1 Pro scored 83.4%, according to the results. Scores are averages across three runs, with the six exam scores weighted equally in the overall result.

With terminal and web access, GPT-6 Astra scored 97.3% and Kimi K3.1 Pro scored 92.9%, the results show. Each tool-enabled task gives an agent a shell, internet access and six hours; the reported results use the best score for each model across the available agent harnesses.

The gap between the best proprietary and open-weight models narrowed from 12.4 to 4.4 points with tools, according to the researchers, while the ranking changed little. Their analysis of failures found that missing knowledge accounted for most errors, whereas calculation errors were rare.

Building and accessing the benchmark

The construction process starts by transcribing exam PDFs into a format that preserves their layout. A coding agent then writes exam-specific conversion code, and a person checks each resulting example against the source and sends corrections back to the agent.

The evaluation code and construction pipeline are in the Sakana AI GitHub repository. The repository says 264 of the 321 examples are hosted on Hugging Face under publishers' terms, with access gated behind acceptance of those terms and a login.

The remainder must be assembled from publishers' PDFs using the repository's reconstruction process. Exam questions, answer keys and images remain their publishers' property, and the repository asks users not to redistribute the dataset or generated tasks.

Sources

Related stories

Explore

More articles