Towards Defining, Measuring, and Leveraging Creativity in AI Applications

AI summary

Overview

Research area: Computational creativity and AI evaluation, specifically formal definitions and measurement of creativity in AI applications (chess and painting).

Technical level: Intermediate. The paper is written around psychological theory and formal postulates, and it includes mathematical notation (postulates, a multiplicative formula, embedding distances), but each concept is explained in prose, making it accessible to readers comfortable with basic algebra and machine-learning terminology.

One-sentence scope: The paper proposes eight formal postulates of creativity grounded in Boden's newness–value–surprise framework, derives a multiplicative creativity formula from them, and tests that formula on chess moves and paintings.

What This Paper Is About

Researchers have long disagreed on what creativity is: some define it only conceptually (listing criteria such as novelty and value) without formal mathematics, while others propose formulas without examining their assumptions or validating them empirically. This paper tries to close that gap by deriving a mathematical creativity score from psychologically motivated postulates and then checking whether the score actually behaves sensibly in two very different domains, chess and art. The same formula is used in both domains, but the way newness, value, and surprise are measured is adapted to each domain.

Key Contributions

1. A set of eight formal postulates of creativity. The postulates, grounded in prior psychological work and Boden's three criteria (newness, value, surprise), include boundedness, continuity, a "creativity property" (an object is creative only if all three components are positive), newness, pairwise separability of newness/surprise/value, and a non-continuity postulate specifying that newness and surprise are related but can change at very different rates.
2. A mathematical definition and closed-form creativity formula. Creativity is defined as a function of newness, value, and surprise with domain-dependent importance weights, instantiated as a multiplicative formula C = N^α · V^β · S^γ, which the authors contrast with additive formulations that would allow a creative score when one component is zero.
3. Concrete measurement procedures for each component. Newness is the minimum embedding distance to any pre-existing object; value is estimated from intrinsic value, auction price, or summed domain criteria; surprise is measured through probabilistic mismatch (e.g., Shannon Surprise) or observation mismatch depending on the domain.
4. Empirical validation in two applications. The formula is applied to individual chess moves and to paintings, with the chess side supported by a ground-truth labeling procedure built from expert annotations and three LLMs.

Main Findings

• The proposed formula satisfies the postulates by construction. The multiplicative form requires all three components to be positive, which the authors argue is violated by additive formulations such as C = N + V + S.
• Chess data and labeling pipeline. The cleaned Mega ChessBase 2025 dataset contains 94,565 games with 7,711,327 moves; the initial prior-knowledge set T consists of 10,842 games all played before 1960, updated yearly. Expert comments mentioning "creative" or equivalent language produced 17,000 candidate comments, which three LLMs (chatgpt 4o-mini, gemini-2.5-flash-lite, claude-3-5-haiku-20241022) filtered down to 639 labeled moves. Labeling used a consistency rule requiring pairwise cosine distance above 0.70 between CLIP embeddings of the LLM justifications and unanimous answers across the three models.
• The chess labeled set is imbalanced, and the paper argues this is expected. 78% of the 639 moves are labeled creative; the authors attribute this to experts commenting disproportionately on interesting, non-obvious moves rather than to a flaw in the annotation process.
• Newness is treated differently across domains. In chess, newness is binary (1 if a move has never occurred, 0 otherwise) because the board is finite and openings are well established; in paintings, newness is the real-valued minimum L2 distance between a painting's embedding and the embeddings of prior paintings.
• Value and surprise implementations differ by domain. Chess value comes directly from the Stockfish engine at a depth of 15; chess surprise uses an observation-mismatch lookup dictionary over board positions. Painting value uses maximum auction price adjusted to 2024 dollars via the Consumer Price Index; painting surprise uses probabilistic mismatch with a Gaussian Mixture Model fitted to prior artwork embeddings.
• Two model variants are used for chess. Since the chess creativity label is binary, taking logarithms of the multiplicative formula allows the weights α, β, γ to be learned by logistic regression (the "basic model"), and a SplineTransformer expansion of each numerical feature forms a "non-linear model."
• Robustness checks for painting embeddings. The paper reports trying multiple pre-trained embeddings—CLIP, DINOv2, and ViT—to test robustness of the method on paintings.
• Painting dataset scope. After cleaning, the auction data covers western art painters producing work between 500 AD and 2024, yielding 355,551 auction records with complete information; 2,802 paintings have multiple auction records, with a maximum of 4 auctions per painting.
• The abstract states both domains were validated, but the detailed painting results are not in the available content. The provided text is truncated during the description of the GMM-based surprise measure for paintings, so the numbers reporting how well the formula performs on paintings are not reported here.

Methodology in Plain English

The authors start from theory rather than data. They take Boden's widely cited claim that something is creative if it is new, valuable, and surprising, and translate it into mathematical postulates—statements the creativity score must obey, such as "a score is positive only when all three components are positive," "the components are independent of one another," and "newness and surprise do not move together in a smooth, predictable way." These postulates rule out certain formula shapes, and the authors settle on a multiplicative formula where each component is raised to a domain-specific exponent.

For measurement, each object (a chess position or a painting) is converted into a numerical embedding. Newness is how far that embedding sits from the closest object already known in the domain. Value is either an intrinsic measure or a practical proxy—Stockfish's evaluation for chess, maximum inflation-adjusted auction price for paintings. Surprise is either how unlikely the object is under a model of prior objects, or how far the object is from what a prior-knowledge dictionary would predict.

Validation proceeds differently in each domain. For chess, the authors mine expert commentary for statements about creativity, use three LLMs to confirm the comments really concern creativity, keep only cases where the models agree, and then check whether the creativity formula predicts those labels—learning the exponents through logistic regression. For paintings, they compute the same three components from auction and image data and examine whether the resulting creativity scores line up with the patterns one would expect across artists.

Why This Matters

Impact on research. The paper argues that much prior work on computational creativity either lacks formal grounding or lacks empirical validation, and that existing formulas have specific defects: components not tied to creativity theory, additive forms that let a non-new object be called creative, purely conceptual proposals with no measurement scheme, and measures such as Bayesian surprise that blur newness and surprise together. By tying its formula explicitly to postulates and testing it in two domains, the work offers a reusable template for defining and measuring creativity elsewhere.

Real-world applications:

• Evaluating AI-generated content—images, text, music—on a score that separates novelty from usefulness and unexpectedness, rather than treating all three as one blurry notion.
• Game analysis and coaching, where the chess case shows how newness, engine value, and departure from familiar play can be combined into a single interpretable number per move.
• Art market and curation analytics, where auction prices and image embeddings give a quantitative lens on which works and which artists stand out from prior tradition.
• Search, recommendation, and ideation tools that need to surface items that are not just new but also valuable and genuinely surprising.

Industry relevance. Any organization deploying generative AI faces the question of whether its outputs are merely different or actually creative. This paper supplies both a formula and a warning: the specific measures of newness, value, and surprise must be adapted per domain, so a single off-the-shelf creativity metric is unlikely to transfer cleanly.

Future Directions

• Complete and report the painting-domain validation. The available content does not include the painting results, so the question of how closely the formula tracks known creative painters remains open in this text.
• Resolve the label imbalance in chess evaluation. With 78% of the 639 labeled moves marked creative, the binary target is skewed; alternative sampling or labeling strategies could clarify how much of that reflects reality versus the annotation pipeline.
• Generalize beyond the two tested domains. The paper itself notes that artwork and storytelling have infinite object spaces where dictionary-based surprise is infeasible and probabilistic mismatch is required, leaving open how well the framework scales to text, music, or scientific discovery.
• Ground the domain weights. The exponents α, β, γ are learned or assumed, so how to set or learn them reliably in new domains—and whether they carry interpretable meaning—is an unresolved design question.

Target Audience

This paper suits researchers in computational creativity, AI evaluation, and human–AI co-creation who want a formal, testable definition rather than a conceptual one. It is also relevant to practitioners building creativity or novelty metrics into generative systems, to game and chess analysts interested in quantifying exceptional play, and to readers in psychology, art history, or the digital humanities who want to see Boden's three-criteria framework translated into something measurable. Readers looking for extensive benchmark tables on language or image generation may find the two-domain scope narrower than expected.

Authors’ abstract

Despite extensive research, computational creativity still lacks a conceptually grounded, practically relevant, and empirically applicable formal definition across contexts such as art, scientific research, and games. Existing approaches to creativity often focus on proposing informal conceptual definitions without providing formal computational concepts. Alternatively, they propose formal definitions without carefully examining their underlying assumptions or providing validation of these definitions by testing their performance in specific application settings. To address these problems, this paper proposes several formal postulates of creativity grounded in prior psychological work on creativity, especially as related to Boden's three core criteria of creativity: newness, value, and surprise. Furthermore, we propose a conceptual mathematical definition of creativity that aligns with these postulates. We also validate our proposed creativity formula in two applications: determining the creativity of particular chess moves, and assessing the creativity of paintings by some artists, such as Picasso, Renoir, and other painters. Experimental results demonstrate that our formula of creativity successfully captures creative patterns across both the chess and the art domains. This suggests that creativity can be effectively characterized through a unified mathematical structure based on three fundamental criteria, namely newness, value, and surprise, whose specific measurements are adapted to the particular domains.