Open Polymer Challenge: Post-Competition Report

AI summary

Overview

• Research area: Machine learning for materials science, specifically polymer informatics — applying ML to predict polymer properties for virtual screening and sustainable materials discovery.
• Technical level: Intermediate. The competition design and metric are math-light, but understanding the results requires familiarity with molecular representations (SMILES, graphs), molecular dynamics simulation concepts, and standard ML pipeline vocabulary (cross-validation, ensembling, distribution shift).
• Scope: A post-competition report on the Open Polymer Challenge, the first community-developed benchmark for polymer property prediction, covering its dataset, molecular dynamics data generation pipeline, results, winning approaches, and lessons learned.

What This Paper Is About

Polymer discovery has been slowed by the absence of large, high-quality, openly accessible datasets — decades of experimental effort between 1950 and 2018 produced only about 1,500 polymers with gas permeability measurements, where each major gas type has 400 to 800 reported values. The Open Polymer Challenge (OPC) was created to fill this gap by releasing a community benchmark of roughly 10K polymers annotated with five molecular-dynamics-simulated properties, and by asking participants to build models that predict all five at once under realistic constraints of small data, label imbalance, and heterogeneous simulation sources. This report describes how the competition was designed, how its data were generated and corrected, what the winning teams did, and what the results imply for future polymer datasets.

Key Contributions

1. The first ML competition and benchmark dedicated to polymer informatics. The OPC released a dataset of 10K polymers with five properties: thermal conductivity (TC), radius of gyration (R_g), density, fractional free volume (FFV), and glass transition temperature (T_g). The curated data contain 11,475 unique polymers, of which 9,625 have at least one labeled property.
2. A released data generation pipeline (ADEPT). Published at https://github.com/sobinalosious/ADEPT, this "Automated molecular Dynamics Engine for Polymer simulaTions" simulates more than 25 properties, including thermal conductivity, radius of gyration, and density. A test dataset was also released at https://www.kaggle.com/datasets/alexliu99/neurips-open-polymer-prediction-2025-test-data.
3. A documented, mid-competition data regeneration effort. After detecting data leakage via SMILES matching on the leaderboard, the organizers released the overlapping subset openly and generated new structures and new MD-based labels for both the public and private leaderboards, producing a superset of 3,000 candidate structures.
4. A distillation of best practices from hundreds of teams. The report analyzes top-200 solutions and identifies recurring successful strategies: domain-specific data curation, fingerprint- and descriptor-based feature engineering, and strong but relatively simple models such as gradient-boosted trees with rigorous cross-validation.

Main Findings

• The competition scaled far beyond the organizers' expectations. It ran on Kaggle from June 16, 2025 to September 15, 2025 — three months — and attracted more than 10,000 registrations, 2,600 participants, 2,000 teams, and 50,000 submissions from over 100 countries. Five prize-winning teams came from a wide range of backgrounds, including undergraduate and graduate students and software engineers with or without prior chemistry experience.
• Data leakage forced a redesign of the evaluation sets. Previously generated polymer data that had once been publicly visible was unintentionally reused by participants, even though it had been removed from GitHub before the competition, because third parties had already re-circulated it. This was detected through SMILES matching on the leaderboard.
• Data generation had to draw on multiple sources. For the new evaluation sets, the organizers sampled 100 structures from the PI1M database of one-million virtual polymers with no overlap with the competition data, used the PI1M generative model to create 2,000 previously unseen structures, and added 900 experimentally validated polymers lacking competition labels as a backup pool — 3,000 candidate structures in total. Two research groups simulated polymers in parallel: one for TC, density, and R_g; the other for T_g and FFV.
• Small data and imbalance were the defining technical constraint. The training set had 7,973 unique SMILES, all with at least one label. Per-property label counts in training were 511 for T_g, 7030 for FFV, 737 for TC, 614 for R_g, and 613 for density — a heavily uneven distribution. Simulation cost per sample ranged from 432 CPU hours (R_g, density) to 864 (FFV), 1440 (TC), and 2880 (T_g).
• Cross-group simulation consistency caused distribution shifts. About 20 of the 3,000 SMILES strings were valid in older RDKit versions but invalid in newer ones, and there was a unit mismatch (K <-> degrees C) in the new T_g values. Both were corrected.
• The T_g shift between leaderboards was not a unit conversion. Public leaderboard T_g had mean 102.9 and standard deviation 103.7; private leaderboard T_g had mean 179.8 and standard deviation 134.9. Fitting a linear transform y = ax + b gives a = 1.3 and b = 46, whereas degrees C to degrees F conversion would give a = 1.8 and b = 32. The paper states the shift instead traced to differences between bi-linear and hyperbolic fitting methods for extracting T_g from density-temperature curves.
• The competing fitting methods produced materially different T_g values. Hyperbolic fits were adopted for high-throughput evaluation because bi-linear fits require choosing an arbitrary linear range. Comprehensive checks after the competition showed many T_g values from hyperbolic fits are higher than those from bi-linear fits, and hyperbolic-fit T_g values depended on the constraints placed on the fit parameters.
• Fingerprints and descriptors dominated feature selection. All solutions in the top-10 used Morgan fingerprints. RDKit descriptors, MACCS, Atom Pair, Topological Torsion, Mordred descriptors, and ECFP were also used, in descending order of frequency. SMILES embeddings and graph representations were each used by only one of the top-10 solutions.
• Tree-based models beat neural networks under this data regime. LightGBM, XGBoost, CatBoost, and TabPFN appeared in all top-5 solutions. The 14th-place team observed an inverse correlation between model complexity and performance, and most top teams used lightweight ensembles such as target-wise averaging rather than deep fusion.
• Addressing distribution shift was decisive but fragile. The 1st place solution applied a calibrated offset proportional to the standard deviation (T_g <- T_g + sigma x 0.5644). The 3rd-place team used fold-wise regression calibration, improving scores by 5.5%. The paper notes these corrections depended heavily on post-hoc public leaderboard probing and manual calibration rather than systematic diagnostics.
• Aggressive augmentation backfired. Random stereoisomer or tautomer enumeration frequently led to overfitting, with the 3rd place team explicitly reporting its ineffectiveness for GNNs. Canonicalization and kekulization were the safer, widely adopted cleaning steps; 3 of the top-10 solutions instead used non-canonical SMILES forms to generate new training data.
• Targeted data filtering helped. The 1st place team filtered training data whose ensemble predictions exceeded an error threshold and removed out-of-distribution samples, specifically from the TC data, finding that removing polymers with TC values over 0.402 led to better results.
• Supplemental external data was widely used. Most teams incorporated additional T_g and density data sources, and the 1st place solution performed fast MD simulations with neural networks to calculate density, FFV, and R_g for 1116 polymers from PI1M.

Methodology in Plain English

The organizers built a benchmark around one central task: given a polymer structure, predict five simulated physical properties at once. Polymers were released as SMILES strings, a standard text encoding of molecular structure, with the equivalent graph representation also valid.

To produce labels, the team ran molecular dynamics simulations rather than experiments, because polymer synthesis is slow and complex. The ADEPT pipeline grew polymer chains from monomers using a random-walk polymerization algorithm with methyl termination, packed multiple chains into a periodic simulation box to form an amorphous cell, and then equilibrated the system through two stages — an initial relaxation with electrostatics off and a truncated Lennard-Jones cutoff, followed by an annealing stage with electrostatics re-enabled via PPPM Ewald summation. From these equilibrated systems the pipeline extracted density from NPT production runs, radius of gyration via per-molecule chunking and time-averaging, and thermal conductivity using non-equilibrium MD along the x-direction with hot and cold Langevin thermostatted slabs. A separate workflow, using Material Studio and the PCFF force field with a 21-step equilibration protocol, generated systems for FFV and T_g, with FFV evaluated by PoreBlazer 4.0.

Models were scored with a weighted Mean Absolute Error (wMAE) across all properties. The weighting factor combines two ideas: dividing by each property's value range so that large-scale properties do not dominate, and multiplying by an inverse-square-root term based on each property's number of labels so that rare properties receive appropriate weight.

Participants submitted predictions to a public and a private leaderboard. After leakage was detected, the organizers regenerated evaluation data in two rounds — one public, one private-only — and continued the competition. The report then analyzes submissions across the top-5, top-10, and top-200 tiers to extract common patterns.

Why This Matters

The OPC provides the first large-scale, openly accessible benchmark for polymer property prediction, addressing a scarcity that has held back ML progress in polymers relative to small molecules and proteins. By releasing the data, the test set, and the ADEPT generation pipeline, the organizers create reusable infrastructure rather than a one-off contest. The competition also surfaced problems that the field will face repeatedly: leakage from briefly public intermediate data, inconsistencies between simulation batches, force-field and post-processing sensitivity, and distribution shifts that cannot be curated away in advance.

Real-world applications:

• Separation membranes. Polymeric membranes are central to separation technologies that can greatly reduce the energy, carbon, and water intensity of many traditional thermally driven separation processes.
• Sustainable and energy-efficient materials. The abstract frames the challenge as a path toward discovering sustainable polymer materials and accelerating development of sustainable and energy-efficient materials.
• Virtual screening pipelines. Accurate multi-task property prediction is described as a core step in virtual screening for materials discovery, allowing candidates to be triaged computationally before synthesis.
• Gas separation polymers. The paper highlights gas permeability data collection as a bottleneck, noting that decades of experiments produced only about 1,500 such polymers.

Industry relevance: because experimental polymer characterization is slow and expensive, and simulation costs here run from 432 to 2880 CPU hours per sample per property, accurate surrogate models directly reduce the cost of screening candidate materials. The paper's finding that simple, well-tuned tree models with careful feature engineering outperform complex neural architectures under small-data conditions is practically useful guidance for industrial R&D teams with limited labeled data.

Future Directions

• Standardizing polymer datasets and simulation protocols. The report emphasizes the need for transparent pipelines and standardized polymer datasets, since annotations are sensitive to force field parameters and post-processing choices, and even small methodological differences produce meaningful shifts.
• Systematic distribution-shift diagnostics. Current corrections relied on post-hoc leaderboard probing and manual calibration. The paper frames replacing this with principled diagnostics as an open problem.
• Better T_g extraction methodology. The discrepancy between bi-linear and hyperbolic fits, and the dependence of hyperbolic fits on constraint bounds, remains unresolved and directly affected the benchmark.
• Scaling beyond a few thousand labeled polymers. The paper notes that thousands of polymers were considered here while total possible polymers number in the billions, and states that differences in data distribution are unavoidable at such scale — implying that generation and curation must scale together.

Target Audience

This paper is most useful to ML researchers working on molecular property prediction and multi-task learning under small-data and distribution-shift conditions; to materials scientists and polymer informaticians seeking a benchmark with released data and a reusable simulation pipeline; and to competition organizers and dataset builders who want a candid case study in leakage, cross-group simulation consistency, and leaderboard design. Practitioners in industrial polymer R&D looking for evidence on which model families actually work at this data scale will also find the best-practices section directly actionable.

Note: the paper content provided does not report the numeric wMAE scores achieved by the winning teams or the final leaderboard standings.

Authors’ abstract

Machine learning (ML) offers a powerful path toward discovering sustainable polymer materials, but progress has been limited by the lack of large, high-quality, and openly accessible polymer datasets. The Open Polymer Challenge (OPC) addresses this gap by releasing the first community-developed benchmark for polymer informatics, featuring a dataset with 10K polymers and 5 properties: thermal conductivity, radius of gyration, density, fractional free volume, and glass transition temperature. The challenge centers on multi-task polymer property prediction, a core step in virtual screening pipelines for materials discovery. Participants developed models under realistic constraints that include small data, label imbalance, and heterogeneous simulation sources, using techniques such as feature-based augmentation, transfer learning, self-supervised pretraining, and targeted ensemble strategies. The competition also revealed important lessons about data preparation, distribution shifts, and cross-group simulation consistency, informing best practices for future large-scale polymer datasets. The resulting models, analysis, and released data create a new foundation for molecular AI in polymer science and are expected to accelerate the development of sustainable and energy-efficient materials. Along with the competition, we release the test dataset at https://www.kaggle.com/datasets/alexliu99/neurips-open-polymer-prediction-2025-test-data. We also release the data generation pipeline at https://github.com/sobinalosious/ADEPT, which simulates more than 25 properties, including thermal conductivity, radius of gyration, and density.