UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian

AI summary

Overview

Research area: Natural Language Processing — multilingual code generation and benchmark evaluation for large language models, with a focus on a low-resource language (Ukrainian).

Technical level: Intermediate. The abstract is accessible without deep technical background, but familiarity with LLMs, prompting, and automated code evaluation helps.

Scope: The paper introduces and describes UA-Code-Bench, an open-source benchmark of 500 Ukrainian competitive programming problems used to evaluate 13 language models on code generation.

What This Paper Is About

Most benchmarks for evaluating large language models either translate tasks from English or test only shallow language understanding, which makes it hard to judge how well models actually perform in low-resource languages. The authors build a benchmark specifically for Ukrainian that targets a demanding task — competitive programming — rather than simple comprehension. The goal is to measure whether leading models can generate correct code when the problem statements are written in Ukrainian.

Key Contributions

1. A new open-source benchmark for Ukrainian code generation. UA-Code-Bench consists of 500 competitive programming problems drawn from the Eolymp platform, evenly split across five difficulty levels ranging from very easy to very hard.

2. A multi-model evaluation with rigorous correctness checking. Thirteen leading proprietary and open-source models were asked to produce Python solutions from a one-shot prompt, and their code was judged against hidden tests using the dedicated Eolymp environment.

3. A multi-dimensional performance analysis. Beyond overall scores, the paper reports how performance varies by difficulty level and assesses the uniqueness of solutions and their computational efficiency in terms of elapsed time and memory consumption.

4. Public release of the full pipeline. The benchmark data, parsing and preparation scripts, code generation scripts, and evaluation scripts are released on Hugging Face.

Main Findings

• Even the strongest models solve only about half the problems. The abstract states that top performers such as OpenAI o3 and GPT-5 solve only half of the problems, which the authors present as evidence of how difficult code generation remains in a low-resource natural language.

• Performance varies across difficulty levels. The paper provides a breakdown of how models perform at each of the five complexity tiers, though the abstract does not report the specific per-level figures.

• Solution quality was examined beyond correctness. The authors analyze whether models converge on unique or similar solutions and measure the time and memory their generated code consumes — the abstract does not give the resulting numbers.

• Broad model coverage. Thirteen leading proprietary and open-source models were compared under a consistent setup: a one-shot prompt and Python as the target language.

• Competitive programming is presented as a valuable evaluation lens. The authors argue that this style of benchmark is well suited to probing LLM capabilities, particularly for underrepresented languages.

Methodology in Plain English

The authors collected 500 programming problems from the Eolymp platform and organized them into five equal-sized difficulty bands. Each of the 13 models was given a problem statement in Ukrainian along with a one-shot prompt and asked to write a Python solution. Instead of grading the code by hand or with heuristics, they submitted the generated programs to the Eolymp judging environment, where the code had to pass hidden tests to count as correct. On top of pass/fail results, they compared performance by difficulty level and looked at how varied and how resource-efficient the generated solutions were.

Why This Matters

Impact on research: The work pushes back on the common practice of translating English benchmarks, offering a native-language evaluation of a genuinely hard task. It gives multilingual code generation and reasoning-focused research a concrete, reusable testbed in a low-resource language.

Real-world applications:

• Model selection for Ukrainian-language developer tools. Teams building coding assistants or IDE plugins for Ukrainian-speaking developers get evidence about which models can handle the language.
• Multilingual software education. Educators and platforms teaching programming in Ukrainian can see where automated help is reliable and where it fails.
• Localization and translation-versus-native-quality studies. The benchmark provides a way to test whether translated benchmarks overstate real ability in a language.
• Automated judging and submission pipelines. The released parsing, generation, and evaluation scripts are directly reusable for building similar benchmark harnesses.

Industry relevance: Companies deploying code assistants in non-English markets need to know the performance gap before promising capability. The finding that top-tier models solve only about half of these problems is a caution for anyone assuming frontier models are equally strong across languages.

Future Directions

• Extending the benchmark approach to other underrepresented languages and comparing results across them.
• Investigating reasoning-enhanced models and whether they close part of the gap observed here.
• Studying why difficulty level affects performance the way it does, and what failure modes dominate at higher tiers.
• Deepening the analysis of multilingual code generation — for example, how prompt language and problem phrasing influence correctness, uniqueness, and efficiency of generated solutions.

Target Audience

Researchers working on multilingual NLP, code generation, and LLM evaluation; benchmark designers who need a template for native-language, execution-verified tasks; practitioners choosing models for Ukrainian-language developer tooling; and anyone interested in how well frontier models generalize beyond high-resource languages.

Authors’ abstract

Evaluating the real capabilities of large language models in low-resource languages still represents a challenge, as many existing benchmarks focus on widespread tasks translated from English or evaluate only simple language understanding. This paper introduces UA-Code-Bench, a new open-source benchmark established for a thorough evaluation of language models' code generation and competitive programming problem-solving abilities in Ukrainian. The benchmark comprises 500 problems from the Eolymp platform, evenly distributed across five complexity levels from very easy to very hard. A diverse set of 13 leading proprietary and open-source models, generating Python solutions based on a one-shot prompt, was evaluated via the dedicated Eolymp environment against hidden tests, ensuring code correctness. The obtained results reveal that even top-performing models, such as OpenAI o3 and GPT-5, solve only half of the problems, highlighting the challenge of code generation in low-resource natural language. Furthermore, this research presents a comprehensive analysis of performance across various difficulty levels, as well as an assessment of solution uniqueness and computational efficiency, measured by both elapsed time and memory consumption of the generated solutions. In conclusion, this work demonstrates the value of competitive programming benchmarks in evaluating large language models, especially in underrepresented languages. It also paves the way for future research on multilingual code generation and reasoning-enhanced models. The benchmark, data parsing, preparation, code generation, and evaluation scripts are available at https://huggingface.co/datasets/NLPForUA/ua-code-bench.