Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models

AI summary

Overview

Research area: Machine learning — large language model reasoning, test-time compute scaling, and competitive programming evaluation.

Technical level: Advanced. The paper assumes familiarity with LLM inference pipelines, clustering, LLM-as-a-judge ranking, and competitive programming scoring conventions.

Scope: The paper introduces GenCluster, a reproducible test-time compute pipeline that lets the open-weight model gpt-oss-120b reach gold-medal-level scores on the IOI 2025 benchmark under the same submission limits imposed on human contestants.

What This Paper Is About

Proprietary systems such as OpenAI's o1-ioi and o3 have claimed gold-medal performance at the International Olympiad in Informatics (IOI), but the methods behind those results have not been fully disclosed. The authors ask whether a comparable result can be reached using only open-weight models plus a transparent, reproducible strategy. Their answer is GenCluster, which generates thousands of candidate C++ solutions per subtask, groups them by behavior on synthetic tests, ranks the groups through an LLM tournament, and then spends a limited submission budget carefully.

Key Contributions

1. GenCluster framework. A scalable test-time compute approach that generates large pools of parallel candidate solutions and then filters them through behavioral clustering and tournament-based ranking before submission.
2. First open-weight gold at IOI 2025. The paper reports the first gold-medal-level IOI 2025 result achieved with open-weight models combined with a transparent, reproducible test-time compute strategy.
3. Demonstrated scaling. Scores improve consistently as the number of generated candidates grows, suggesting a path toward surpassing gold-level performance.
4. Public release plan. The authors state they intend to release their code publicly at the NVIDIA-NeMo/Skills repository.

Main Findings

• Gold medal with gpt-oss-120b. Using 5000 generations per subtask and respecting the IOI limit of 50 submissions per problem, GenCluster reaches a total score of 446.75, which corresponds to a gold medal at IOI 2025 (rank 26 in the comparison table).
• Model comparison favors gpt-oss-120b. Among gpt-oss-120b, gpt-oss-20b, DeepSeek-R1-0528, and Qwen3-235B-A22B-Thinking, gpt-oss-120b scores highest by a significant margin and is the only model with the potential to reach gold with up to 5000 generations. The gpt-oss models show stronger gains as the generation count rises.
• Steady gains under the submission cap. The submitted score climbs from 332.27 at K=50 to 446.75 at K=5000, showing that scaling compute helps even when submissions are limited to 50 per problem.
• GenCluster beats alternative selection strategies. With 5000 solutions per subtask, GenCluster scores 446.75, versus 441.11 for the Score-Based variant, 406.49 for the Random-Rep variant, 314.22 for Cluster-Majority, 300.10 for Random, 299.87 for Cluster-Size, and 277.36 for Longest. Simple heuristics such as choosing the longest reasoning trace perform worst.
• Test-set size trades off purity against cluster count. Increasing the number of test cases improves cluster purity (measured by F1-score) but also raises the average number of clusters while lowering average cluster size, making selection harder under submission constraints.
• Tournament rounds saturate around 10. More games per cluster improve the final score, but the benefit largely plateaus after 10 rounds; a single game is clearly insufficient.
• Ranking is strong but not perfect. In 35 out of 39 subtasks, the best solution appears among the top 50 ranked clusters.
• Longer reasoning helps the gpt-oss family. gpt-oss-120b, gpt-oss-20b, and DeepSeek-R1-0528 keep improving up to their token limits, while Qwen3-235B-A22B saturates after roughly 48K tokens.
• Gap to the best human and to OpenAI. The top-ranked IOI 2025 contestant, Hengxi Liu, scored 591.23 (Gold); OpenAI's reported IOI 2025 result is 533.29 (Gold); GenCluster reaches 446.75 (Gold), and the silver medalist Maxim Tsoy scored 436.72. The authors note OpenAI has not disclosed the compute budget or methodology behind its result.

Methodology in Plain English

The method has four stages, applied separately to each subtask of each problem.

1. Parallel generation. For every subtask, the model generates K candidate C++ solutions from a single prompt. Because the generations are independent, they run in parallel. Candidates with unparseable or non-compiling code are discarded.

2. Behavioral clustering. The model also writes many test-input generators and many independent validators (programs that check whether an input satisfies the subtask's constraints). A generated test is kept only if at least 75% of the validators approve it. All candidate solutions are then run on the validated test inputs, and solutions producing identical outputs are grouped into the same cluster using hash values. Clusters that produce empty output on any test (for example, from a runtime error) are removed entirely.

3. Ranking with tournament. Each cluster is represented by its solution with the longest thinking trace. Clusters then play a partial round-robin tournament: each comparison pairs two cluster representatives and the LLM judges which solution is better, with presentation order randomized to reduce position bias. Clusters are ranked by number of wins. A variant replaces wins with average 0-10 score from the judge.

4. Submission. With only 50 submissions allowed per problem, the system starts with the final (usually hardest) subtask and cycles through clusters in ranked order, taking one solution at a time from each cluster and rotating within a cluster by reasoning length. Once a subtask reaches its maximum score, the remaining clusters for it are skipped.

Experimental setup. All code — solutions, generators, and validators — is written in C++ because that is the dominant IOI language. Each subtask used 100 test generators and 100 validators, yielding 100 validated test cases. The tournament used 10 games per cluster. Maximum generation lengths were 120K tokens for gpt-oss-120b, 120K for gpt-oss-20b, 64K for DeepSeek-R1-0528, and 120K for Qwen3-235B-A22B. Scoring used the official IOI graders with the same time and memory limits, and the benchmark, IOI-2025, mirrors the structure of the public IOI-2024 benchmark.

Why This Matters

Impact on research. Prior gold-level IOI and ICPC results came from closed proprietary systems with undisclosed methods. This work provides a fully specified, reproducible pipeline built on an open-weight model, giving the community a transparent baseline for studying large-scale test-time compute on verifier-limited reasoning tasks. It also isolates where the remaining headroom lies: selection and ranking, not generation.

Real-world applications:

• Automated software engineering assistants that must choose among many candidate implementations under a strict verification or review budget.
• Automated grading and code review systems that need to rank student or submitted solutions by likely correctness.
• Systems for generating and validating test data automatically when no reference test suite is available.
• Resource-constrained deployments that need strong model behavior without access to proprietary frontier APIs.

Industry relevance. The paper quantifies the cost of that capability: generating 5000 candidate solutions for the benchmark consumes roughly 7.3 billion tokens, and the ranking tournament consumes an additional 7.3 billion tokens. That makes the approach informative for anyone weighing inference-time compute spend against model quality, and it frames open-weight models as a viable route to competition-grade coding performance.

Future Directions

• Better ranking under submission caps. A substantial gap remains between the unconstrained best-of-K score and the submitted score, and it widens as K grows, so improved cluster ranking and selection is the clearest next target.
• More efficient test generation. Synthetic tests currently come from the same models being evaluated and can miss rare corner cases; developing more reliable or automated test coverage could sharpen clustering without inflating cluster counts.
• Reducing compute cost. The 7.3 billion tokens for generation plus 7.3 billion for judging suggest room for cheaper generation or judge strategies, especially for real-time or low-resource use.
• Replacing or improving the length heuristic. Reasoning length is used both to pick cluster representatives and to order solutions within a cluster, but the authors acknowledge it can reward verbose, confused, or incorrect traces.

Target Audience

Researchers working on LLM reasoning and inference-time scaling; competitive programming and code-generation benchmark developers; engineers building agentic or ensemble-based code generation systems; and anyone tracking the capability gap between open-weight and proprietary models on hard reasoning tasks. The paper's ablation tables and pipeline description make it especially useful for practitioners who need to reproduce or adapt a large-scale generation-and-selection pipeline.

Authors’ abstract

Competitive programming has become a rigorous benchmark for evaluating the reasoning and problem-solving capabilities of large language models (LLMs). The International Olympiad in Informatics (IOI) stands out as one of the most prestigious annual competitions in competitive programming and has become a key benchmark for comparing human and AI-level programming ability. While several proprietary models have been claimed to achieve gold medal-level performance at the IOI, often with undisclosed methods, achieving comparable results with open-weight models remains a significant challenge. In this paper, we present GenCluster, a scalable and reproducible test-time compute framework that attains IOI gold-level performance using open-weight models. It combines large-scale generation, behavioral clustering, ranking, and a round-robin submission strategy to efficiently explore diverse solution spaces under limited validation budgets. Our experiments show that the performance of our proposed approach scales consistently with available compute, narrowing the gap between open and closed systems. Notably, we will show that GenCluster can achieve a gold medal at IOI 2025 for the first time with an open-weight model gpt-oss-120b, setting a new benchmark for transparent and reproducible evaluation of reasoning in LLMs