LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?

AI summary

Overview

• Research area: Large language model evaluation, specifically competitive programming and code-generation benchmarks.
• Technical level: Intermediate. The benchmark methodology is accessible to anyone familiar with LLM evaluation; the algorithmic discussion of dynamic programming, segment trees, and contest subtasks assumes some competitive programming background.
• Scope: The paper introduces LiveOIBench, an offline, expert-curated benchmark of 403 Informatics Olympiad problems with official test cases and human contestant rankings, and uses it to evaluate 34 LLMs against elite human competitors.

What This Paper Is About

Existing competitive programming benchmarks for LLMs suffer from incomplete test suites that produce false positives, a shortage of genuinely hard problems, and dependence on online judge APIs that make large-scale evaluation slow and hard to reproduce. The authors build a benchmark directly from official Informatics Olympiad sources, where the problems, test cases, scoring subtasks, and human results are all released by the contest organizers, and then measure how close today's best models come to the humans who actually competed.

Key Contributions

1. A new dataset. 403 expert-designed problems drawn from 72 contests across 14 Informatics Olympiads held between 2023 and 2025, each averaging 60 official organizer-written test cases, with fine-grained subtask rubrics and official human contestant rankings attached.
2. An offline evaluation system. A self-contained local judge that removes reliance on external platform APIs, making evaluation reproducible, accessible, and cheap enough to run at scale.
3. Extensive benchmarking. Results for 34 proprietary and open-weight models across pass rate, relative score, human percentile, Olympiad medal counts, and Codeforces Elo.
4. Behavioral and contamination analyses. Studies of algorithmic strengths and weaknesses, reasoning-trace behavior, error patterns, programming-language effects, and four separate probes for data contamination.

Main Findings

• Top models still lose to elite humans. GPT-5 reaches the 81.76th human percentile with a 63.03% pass rate and a 2414 Elo, earning gold in 50% of contests. Since a gold medal requires consistently beating roughly the top 10% of contestants, no model clears that bar.
• Open-weight models are closing the gap. GPT-OSS-120B reaches the 60th percentile at standard reasoning effort and about the 72.88th percentile at high effort, only nine percentile points behind GPT-5. Seed-OSS reaches roughly the 54th percentile, and Qwen3-32B the 42nd.
• Extended reasoning is decisive. Non-thinking models almost all fall below a 10% pass rate. GPT-4.1, the best of them, performs at roughly the level of Qwen3-14B. Both parallel scaling (sampling more candidates) and sequential scaling (longer reasoning budgets) improve results, with diminishing returns for parallel sampling.
• Algorithmic weaknesses cluster in DP, trees, and segment trees. Models score above 70% on implementation, mathematics, prefix sums, sorting, and graph traversal, but GPT-5 drops to about 47% on dynamic programming, 56% on segment trees, and 38% on tree problems, which require creative state design and hierarchical invariants.
• Strong models reason more strategically. Reasoning traces split into analysis, planning, exploration, implementation, and verification. Weaker and stronger models are distinguished mainly by how much they pivot between alternative solution paths: stronger models allocate extra budget to analysis, implementation, and verification while keeping exploration stable, avoiding "underthinking."
• Later subtasks are harder. Splitting subtasks by position within a problem shows consistent performance decline, because early subtasks carry tighter input constraints and later ones require general optimized solutions.
• Pass rate alone is misleading. GPT-OSS-120B has a higher pass rate than Gemini-2.5-Pro (47.78% vs 44.46%) yet loses to it on human percentile and Elo, so the authors recommend reporting several metrics together.
• Little evidence of contamination. Model performance shows no correlation with problem release date relative to training cutoffs, near-zero correlation with model familiarity for task statements or official solutions, preserved scores under semantic rephrasing, and low source-code similarity to official solutions (median 0.11 for GPT-5, 0.12 for Grok-4-Fast). Performance does drop under adversarial masking of examples and tokens, indicating models rely on semantic understanding rather than memorized problem identities.
• Runtime errors persist even in the best models. Time limit, memory limit, and compilation failures fall sharply as reasoning improves, but runtime errors decline much less, plausibly because top models favor aggressive optimizations such as custom data structures and pointer arithmetic.
• Language effects are about execution, not strategy. On 132 USACO problems, C++ leads (53.26% relative score, 59.09% pass rate), followed by Java (50.78%, 56.82%) and Python (44.35%, 46.97%), while the distribution of reasoning behaviors stays similar across all three languages.

Methodology in Plain English

The authors first selected a set of internationally recognized Informatics Olympiads plus national contests from strong IOI countries that publish English problem statements. They wrote custom crawlers to pull statements, constraints, subtask definitions, test cases, official solutions, and contestant rankings directly from official sites, falling back to mirrors like CSES and LibreOJ when official pages were incomplete, and restricted collection to contests from 2023 onward so the benchmark stays fresh.

To verify quality, they ran the organizers' own official solutions against the crawled test cases to confirm the judge worked correctly. Because many statements are released only as PDFs, they converted them to Markdown with an automated tool and had Gemini-2.0-Flash check and correct the output, manually inspecting a sample of 40 tasks first. They enriched each problem with difficulty and algorithm tags from solved.ac and Luogu, and linked human contestants to their Codeforces profiles to obtain ratings spanning 2022 to 2025.

For evaluation, each model generates eight candidate C++ solutions per problem and the highest-scoring one is kept. Because subtask rubrics and human contestant results are both available, a model's total contest points can be converted into a percentile rank among the real human participants, a medal outcome, and an equivalent Codeforces Elo, rather than being reduced to a single pass/fail number.

Why This Matters

This paper shifts competitive programming evaluation from "did the code pass some tests" toward "how would this model have placed against the humans who competed." Because the test cases come from the organizers rather than being generated by rules or other LLMs, false-positive successes are much less likely, and because the judge runs locally, anyone can reproduce results without hammering a platform's servers.

Real-world applications:

• Model and tool selection. Teams choosing a coding assistant for algorithmic or performance-critical work get a human-calibrated difficulty signal instead of a saturated pass rate.
• Training signal design. The finding that models fail on DP, trees, and segment trees, and that runtime errors persist, points to concrete targets for fine-grained reward signals and curriculum data in reinforcement learning.
• Education technology. The subtask rubrics, official rankings, and difficulty tags support automated tutoring, partial-credit grading, and practice platforms for competitive programming students.
• Benchmark integrity auditing. The four-pronged contamination methodology (temporal, familiarity, rephrasing, code similarity) is reusable for any LLM benchmark that risks training-set leakage.

For industry, the benchmark is a practical upper-bound measurement for how far automated code generation can currently be trusted on hard algorithmic problems, and the gap to the 90th-percentile human threshold is a concrete target for the next generation of reasoning models.

Future Directions

• Targeted training for hard algorithm classes. The authors suggest curriculum-driven fine-tuning on synthetic graph, tree, and dynamic programming problems so models internalize hierarchical invariants and compositional reasoning.
• Reward signals beyond correctness. Current RL training rewards only passing solutions, leaving efficiency and memory management unoptimized; future work could add fine-grained rewards for runtime robustness and resource use.
• Better reasoning-budget allocation. Open question: how should models distribute reasoning effort across analysis, planning, exploration, implementation, and verification to avoid excessive pivoting and underthinking?
• Keeping the benchmark alive and broader. The crawling pipeline supports periodic updates from all 14 competitions, but extending coverage to more Olympiads, languages, and task types remains open.

Target Audience

LLM evaluation researchers and benchmark builders will find the construction pipeline and contamination methodology directly reusable. Model developers at both frontier labs and open-weight projects get a clear picture of where reasoning models still fail and why. Competitive programming educators and contest organizers benefit from the human-percentile framing and subtask-level scoring. General machine learning practitioners will find the reasoning-trace and error-pattern analyses useful even if they never touch an Olympiad problem.

Authors’ abstract

Competitive programming problems are increasingly used to evaluate the coding capabilities of large language models (LLMs) due to their complexity and ease of verification. Yet, current coding benchmarks face limitations such as a lack of exceptionally challenging problems, insufficient test case coverage, and reliance on online platform APIs that limit accessibility. To address these issues, we introduce LiveOIBench, a large-scale competitive programming benchmark featuring 403 expert-curated problems, averaging 60 official test cases each, drawn from 72 contests across 14 Informatics Olympiads held between 2023 and 2025. LiveOIBench has four key features: (1) expert-designed tasks with detailed subtask rubrics and extensive test cases; (2) direct comparison to elite human contestants; (3) continuous updates to reduce contamination risk; and (4) a fully offline, reproducible evaluation system. Benchmarking 34 popular general-purpose and reasoning LLMs, we find that GPT-5 achieves an 81.76th percentile, still falling short of top human contestants, while among the open-weight models, GPT-OSS-120B reaches only the 60th percentile. Reasoning-trace analyses indicate that robust reasoning models prioritize precise problem analysis over excessive exploration. Finally, analyses across release dates, task familiarity, and code similarity find minimal evidence of data contamination in our benchmark. Our leaderboard, code, and data are available at: https://liveoibench.github.io/.