Continuous SUN (Stable, Unique, and Novel) Metric for Generative Modeling of Inorganic Crystals

AI summary

Overview

Research area: Machine learning for materials discovery — specifically evaluation metrics for generative models that design inorganic crystals, plus reinforcement learning with metric-based rewards.

Technical level: Advanced. The paper presupposes familiarity with crystal structure representations (unit cells, space groups, Wyckoff letters, convex hulls), generative model benchmarks, and pseudometric theory (Lipschitz continuity, triangle inequality).

Scope: The paper defines continuous versions of uniqueness, novelty, and stability for generated crystals, unifies them into a "continuous SUN" (cSUN) score, proves their theoretical robustness properties, and benchmarks seven generative models on the MP20 dataset.

What This Paper Is About

Generative models for crystals are usually judged by three things: whether their outputs are unique (U), novel (N), and stable (S). All three are currently computed as yes/no answers controlled by arbitrary thresholds, so a crystal that is 0.11 [eV/atom] above the convex hull scores the same as one that is 1.0 [eV/atom] above it, and small jitters in atomic coordinates can flip a score. The authors replace these binary decisions with smooth, continuous distance- and energy-based scores, combine them into a single weighted metric called cSUN, and show that this gives finer-grained rankings of generated candidates.

Key Contributions

1. Continuous uniqueness and novelty (cU, cN): The authors replace the binary StructureMatcher comparison with continuous crystal distance functions — the Element Mover's Distance (d_elm) for composition, the L∞ distance between Average Minimum Distance vectors (d_am) for structure, and a weighted linear combination (d_elm+am) for both, with weights w_elm ≈ 0.78 and w_am ≈ 0.22 derived from the standard deviations of pairwise distance distributions on the MP20 test data.

2. A theoretical robustness analysis: The paper proves that the continuous distances satisfy isometry invariance and Lipschitz continuity (two requirements drawn from prior work) and, additionally, that the model-level uniqueness score is invariant to permutation of samples — a property the conventional Ū_smat violates because d_smat breaks the triangle inequality.

3. Continuous stability (cS): A monotonically decreasing function of energy above the convex hull that gives 1 for E_hull ≤ 0, decays linearly as 1 − E_hull/τ up to the threshold τ = 0.4289 [eV/atom] (the 99.9th percentile of the MP20 test set E_hull distribution), and 0 beyond.

4. The cSUN metric and its use as an RL reward: cSUN is defined as cS(x_i)^w_S · cU(x_i)^w_U · cN(x_i)^w_N, with tunable exponents that the binary SUN cannot support (since 0^x and 1^x are insensitive to x). The paper explores cSUN as a reinforcement-learning reward signal.

Main Findings

• Binary U and N mostly measure composition, not structure: Ū_smat − Ū_comp stays below 0.12 for every model tested, indicating that the widely used d_smat acts mainly as a compositional match checker. The same pattern holds for N̄_comp and N̄_smat.

• Ū_smat is not permutation invariant in practice: On a fixed set of 10k CDVAE samples shuffled with seeds 0–4, Ū_smat returned 0.9942, 0.9947, 0.9940, 0.9944, and 0.9943 — different scores for the identical set of crystals.

• Continuous metrics expose weaknesses that discrete metrics hide: CDVAE ranks highest in Ū_comp and Ū_smat but performs poorly in Ū_elm and Ū_elm+am, meaning its outputs are concentrated rather than dispersed in continuous space. DiffCSP++ ranks first in Ū_wyckoff but falls to fourth in Ū_am, showing that diversity in Wyckoff templates does not imply structural diversity in interatomic distances.

• Models beat the test set on novelty more often than on uniqueness: Most models surpass the MP20 test set in N̄, but fewer exceed it in Ū; only Chemeleon-DNG exceeds the test set in Ū_am.

• High novelty can signal implausibility: The five most novel CDVAE samples selected by cN_elm+am had cN values of 0.7757, 0.7711, 0.7689, 0.7682, and 0.7666, but E_hull values of 0.5338, 0.8257, 0.6765, 0.6657, and 0.2781 [eV/atom] (computed against the Materials Project with the MACE-MPA-0 force field). Crystal (a) fails the SMACT chemical validity filter; (b)–(d) are SMACT-valid but thermodynamically unstable; only (e) is both valid and moderately stable.

• Binary stability discards most samples: With binary S, 60–70% of generated samples are simply discarded across models (96.5% for CDVAE), whereas cS produces a smoother distribution and removes fewer samples.

• Justification for relaxing the stability threshold: Approximately 20% of experimentally synthesized crystals in the Materials Project (theoretical=False) have E_hull greater than the typical 0.1 [eV/atom] threshold.

• SUN's granularity problem: In Figure 8(a), roughly 3000 MatterGen samples share SUN = 1 and are indistinguishable from marginal candidates, whereas cSUN_elm+am positively correlates with SUN_smat while providing a detailed ranking.

• RL with cSUN reduces reward hacking: The adjustable weighting scheme in cSUN effectively mitigates reward hacking and avoids local minima when used as a reinforcement-learning reward signal.

Methodology in Plain English

The authors start from the standard pipeline: generate 10,000 crystals per model via de novo sampling, then score them. For the similarity metrics they swap out the yes/no StructureMatcher test for two smooth distances. One compares chemical compositions by treating each composition as a histogram of elements and computing the cheapest way to transform one histogram into the other, using chemical similarity between elements as the transport cost. The other compares structures by summarizing each crystal as a vector of average nearest-neighbor distances (the AMD vector) and taking the largest component-wise difference between two such vectors. Both are rescaled to [0, 1] via d = d'/(1 + d'), then blended into a single distance using weights chosen so the two components contribute equally.

For stability, instead of asking "is E_hull below 0.1 [eV/atom]?", they let the score slide linearly from 1 at zero energy above the hull down to 0 at a threshold of 0.4289 [eV/atom], chosen as the 99.9th percentile of the MP20 test-set distribution. The three continuous scores are then multiplied together with tunable exponents to form cSUN. The authors benchmark seven models (CDVAE, DiffCSP, DiffCSP++, MatterGen, Chemeleon-DNG, ADiT, Chemeleon2), all trained on MP20, comparing rankings under discrete and continuous distances and checking the stability of the metrics under sample shuffling. They also run a reinforcement-learning experiment in which cSUN serves as the reward.

Why This Matters

Evaluation metrics silently shape which generative models the field pursues. If a metric is threshold-dependent and order-sensitive, models can score well without producing genuinely useful crystals, and promising near-miss candidates are thrown away before anyone looks at them.

• Materials discovery screening: A continuous ranking lets researchers triage thousands of generated candidates by how stable, diverse, and novel each one is, rather than sorting them into pass/fail buckets of ~3000 indistinguishable "SUN = 1" hits.
• Model selection and benchmarking: Continuous scores distinguish models whose outputs are concentrated from models whose outputs are genuinely diverse — a distinction the binary composition-based metrics miss entirely.
• Reinforcement learning for inverse design: cSUN's tunable weights give a reward signal that can be rebalanced, which the authors report mitigates reward hacking and local minima.
• Identifying where current models fail: The continuous metrics reveal that today's crystal generators are better at novelty than at producing diversity comparable to the test data.

Industry relevance: Any organization using generative models for battery materials, catalysts, thermoelectrics, or semiconductors needs a trustworthy way to rank candidate crystals before committing expensive DFT or synthesis effort. Continuous, robust metrics directly affect the cost and hit rate of that funnel.

Future Directions

• Full cSUN-based model evaluation: The paper introduces cSUN and compares its distribution against SUN_smat, but the detailed modeling discussion of the truncated Section 3.3 leaves room for broader benchmarking across all seven models with multiple weight settings.
• Tuning the cSUN weights: Because w_S, w_U, and w_N can prioritize stability, uniqueness, or novelty, systematic study of how different weightings change model rankings and RL behavior is a natural follow-up.
• Direct reinforcement learning against cSUN: The authors demonstrate that cSUN works as a reward, but the paper only sketches this direction; scaling it to train generative models end-to-end and comparing against other reward formulations remains open.
• Beyond thermodynamic stability as a synthesizability proxy: The paper notes that no established in-silico method for synthesizability exists, and that E_hull is only an approximation — improved proxies, or validated relaxed thresholds such as τ = 0.4289 [eV/atom], would strengthen the S component.

Target Audience

Researchers and engineers working on generative models for inorganic crystals and materials discovery; benchmark designers who need robust, threshold-free evaluation metrics; and practitioners applying reinforcement learning to scientific design tasks. Readers without a background in crystallography (space groups, Wyckoff positions, convex hulls) or in metric theory (Lipschitz continuity, triangle inequality) will find the theoretical sections demanding, but the model-comparison results and the cSUN definition itself are accessible to anyone familiar with generative model evaluation.

Authors’ abstract

To address pressing scientific challenges such as climate change, increasingly sophisticated generative models are being developed to efficiently sample the large chemical space of potential functional materials. The proliferation of these models has necessitated the establishment of rigorous evaluation metrics. While uniqueness (U), novelty (N), and stability (S) of samples serve as standard metrics, their current formulations show several limitations. U and N rely on binary comparisons of crystals, rendering them dependent on heuristic thresholds, incapable of quantifying the degree of similarity, sensitive to atomic coordinate perturbations, and not invariant to sample permutation. Similarly, the binary assessment of S risks a premature exclusion of marginally unstable yet potentially novel candidates. These limitations are addressed by making the aforementioned metrics continuous. Furthermore, we integrate them into a unified metric ``continuous SUN" (cSUN), which offers a smoother score distribution and greater tunability than the conventional binary SUN metric. Experimental results demonstrate that our continuous metrics provide granular insights into sample distributions and facilitate the identification of the most promising candidates. Finally, the use of cSUN as a reward signal in reinforcement learning is explored, showing that its adjustable weighting scheme effectively mitigates reward hacking and avoids local minima.