Evaluating LLM Creativity through Long-Tail Performance

AI summary

Overview

Research area: Natural Language Processing — evaluation of large language model creativity, with connections to long-tail distribution analysis and human-preference benchmarking.

Technical level: Intermediate. The paper combines a conceptual framework (probability distributions and tail regions), a light theoretical argument, and a large empirical study; readers comfortable with LLM benchmarking terminology and basic probability will follow it comfortably, while the formal section can be skipped.

Scope: The paper proposes that heterogeneous LLM creativity benchmarks share a common target — high-quality task performance on rare (long-tail) inputs and outputs — and tests this claim across 15 benchmarks and 26 open-weight models.

What This Paper Is About

Creativity benchmarks for LLMs differ widely in their tasks (writing, problem solving, ideation) and their scoring metrics, which makes it unclear whether they are measuring the same underlying thing. The authors argue that this shared target is performance in the long tail of a reference input-output distribution, meaning a model's ability to produce good outputs for inputs and output patterns that are uncommon in a reference corpus such as pretraining data. They test whether creativity benchmarks really sit further into this tail than standard capability benchmarks, and whether tail performance matches how humans judge model creativity.

Key Contributions

1. A taxonomy of creativity evaluation design. The authors survey 82 papers and organize existing LLM creativity evaluations into a nine-dimensional taxonomy covering task and metric design choices (e.g., objective, domain, evaluation granularity), making explicit why different studies can reach contradictory conclusions.

2. A formal framing of creativity as long-tail performance. Using a factorization of a reference distribution into input probability and output-given-input probability, they define peak, input-tail, output-tail, and joint-tail regions, and connect this to the novelty-quality product formulation of creativity with a theoretical argument (via Tonelli's theorem) showing that the same quality advantage counts more when it occurs in higher-novelty regions.

3. Empirical validation of the tail hypothesis. Across 15 benchmarks and 26 open-weight models, they show creativity benchmarks occupy higher-novelty regions than standard benchmarks in the same task domains, and that higher-tail subsets of a benchmark produce model rankings that agree more closely with human creativity rankings from LM Arena.

4. A cheap diagnostic metric, q@Tail(α), and an audit of creativity-promoting methods. They introduce a metric that measures mean task quality on the most novel α fraction of an existing benchmark and apply it to test-time and training-time creativity interventions.

Main Findings

• Creativity benchmarks sit further in the tail. In all three task domains (free-form writing, mathematics reasoning, free-form QA), creativity benchmarks shift toward higher input novelty, higher output novelty given the input, or both. In writing, Arena-Hard Creative Writing sits at the high end of both axes (both approximately 0.6 to 0.7), well above standard benchmarks such as Alpaca, TinyStories, and BookMIA; LitBench and CS4 also lie further in the tail than every standard benchmark in that domain. Pooled univariate and multivariate tests confirm the creativity and standard distributions differ significantly (all p < 0.001) in the writing domain, and the pattern is stable across n ∈ {4, 5, 6} and across model families.

• Three elicitation operations map to three tail sub-regions. Curation (selecting rare or improbable inputs, as CreativeMath does with expert-filtered AMC/AIME problems) pushes benchmarks toward the input-tail. Prescription (instructing the model to be creative) pushes toward the output-tail. Constraint (adding restrictions that rule out common outputs, e.g., CS4 story requirements) pushes toward the joint-tail.

• Tail subsets rank models more like humans do. On Arena-Hard Creative Writing (N = 244, split into quintile bins), the two most-tail bins reach Kendall's τ = +0.394 and +0.333 with the LM Arena creative-writing ranking, versus +0.152, +0.091, and +0.061 for the three peak bins.

• The effect also appears on a standard benchmark. On GPQA-Diamond (N = 124), tail bins reach τ = +0.396 and +0.222, above the +0.099 to +0.175 range of peak bins — showing the tail–human alignment is not unique to creativity benchmarks.

• The trend holds at finer bin resolution. At ten-bin resolution, the association between bin-level novelty and agreement with human ranking is statistically significant on both benchmarks (Spearman ρ = 0.661–0.855, p = 0.0016–0.0376). The authors note the relationship need not be strictly monotonic at the extreme tail, since agreement peaked before the final bin in their two main examples.

• Across writing benchmarks the association is positive but weaker. Arena-Hard Creative Writing, the most-tail benchmark, reaches τ = 0.37 (p = 0.029), whereas Alpaca, near the low-novelty end, reaches τ = 0.16 and does not significantly agree (p = 0.37). Across the eight writing benchmarks, Spearman ρ = 0.26. The authors weight the within-benchmark analysis more heavily because it holds benchmark-level factors fixed.

• Most creativity-promoting methods trade quality for novelty. With q@Tail(α = 25%) averaged across 4 representative models and benchmarks, only min-p sampling improves tail quality over the vanilla model (+17.7%) — but it reduces overall novelty by 16.3% and Novelty at Peak by 44.2%. Every other tested method (Convergent-Divergent Thinking, temperature scaling at τ × 1.5 and τ × 2.0, Persona, Branch-Solve-Merge, Self-consistency, Meta-prompting, CRPO-SFT, CRPO-DPO) increases peak novelty while scoring lower quality at the tail than the vanilla model, with CRPO-DPO dropping tail quality by 79.3%.

• Tail examples are not simply hard examples. Comparing difficulty-based and novelty-based splits, the overlap is low — 0.07 on MATH (n = 1,000) and 0.23 on Cambridge CEFR (n = 300) — near or below the random baselines of 0.17 and 0.12 respectively.

Methodology in Plain English

The authors first read and catalogued 82 prior papers on LLM creativity evaluation, sorting their design choices into a taxonomy so that differences between benchmarks become visible rather than implicit.

For the empirical work, they needed a way to say how "rare" a prompt or a response is. They used the Pile pretraining corpus as the reference source of "what is common," and measured rarity as the fraction of a text's 4-grams that never appear in that corpus, queried through a tool called Infini-Gram-Mini. A higher fraction of unseen 4-grams means the text is more novel and sits further into the tail. The Pile predates all evaluated benchmarks, which limits contamination that would distort the measurement. Each prompt gets a single shared novelty score — the geometric mean of input novelty and the average output novelty across all models — so that every model is compared on exactly the same prompts.

They then took 15 benchmarks spanning three domains (10 writing, 2 math reasoning, 3 free-form QA) and 26 open-weight models from the Llama, Qwen, OLMo, Gemma, and Mistral families plus GPT-OSS-20B, Phi-4, Nanbeige4-3B-Thinking, and Nemotron-3-Nano-30B. To avoid confounding rarity comparisons with metric choice, they held the quality metric fixed within each domain: exact-match via LM-Eval-Harness for math and QA, and the Skywork-Reward-V2-Llama-3.1-8B reward model for open-ended writing.

For the human-alignment analysis, they split each benchmark's examples into five novelty bins, ranked models by their mean quality within each bin, and compared those rankings against the LM Arena creative-writing leaderboard using Kendall's τ. They also computed a simpler diagnostic, q@Tail(α), which just averages a model's quality over the α most novel fraction of a benchmark, and applied it to a set of methods designed to boost creativity.

Why This Matters

Impact on research. The paper offers a unifying explanation for why creativity benchmarks disagree with one another: they are all pushing evaluation into the tail, but into different parts of it. This gives benchmark designers a principled vocabulary (reference distribution, input-tail, output-tail, joint-tail) and a cheap diagnostic that does not require building a new benchmark. It also reframes the goal of creativity research from "make outputs rarer" to "stay high-quality when inputs and outputs are rare."

Real-world applications:

• Benchmark design and auditing: teams can check whether a proposed creativity benchmark actually reaches the tail regions relevant to its intended users, and choose a reference corpus that matches the audience (e.g., a general web corpus for consumer writing tools versus a domain literature corpus for scientific applications).
• Model selection for creative products: q@Tail(α) lets practitioners compare models on a specific existing benchmark's hardest, least-familiar inputs rather than on average performance.
• Evaluating prompting and decoding interventions: the quality-versus-novelty table gives a practical template for checking whether a "creativity booster" actually helps or merely produces stranger text.
• Post-training and data curation: the results point toward targeted improvement on rare inputs as a route to models that users perceive as more creative.

Industry relevance. Because the diagnostic reuses existing benchmarks and only requires n-gram corpus statistics, it is far cheaper than commissioning new human evaluations, which matters for teams that iterate on models frequently. The finding that prompt- and decoding-level tricks mostly raise novelty while lowering tail quality is directly actionable for product teams tempted to ship creativity settings that degrade output on unusual requests.

Future Directions

• Better reference distributions and novelty measures. The current n-gram proxy captures surface rarity but not semantic, conceptual, or culturally grounded novelty. Replacing the Pile with human-authored creative corpora, model self-samples, or domain-specific corpora — and exploring semantic notions of rarity — is an open question.

• Robustness of the quality estimator. Rankings and q@Tail may depend on the reward model used, which can carry architecture, length, style, and domain biases. Testing across multiple reward models, LLM judges, and human-calibrated quality measures is left open.

• Broadening the human reference beyond creative writing. The strongest human-alignment evidence relies on LM Arena creative-writing rankings, which reflect holistic preferences of self-selected users rather than judgments isolating creativity, and do not cover math, scientific reasoning, or design. Extending to other domains is unresolved.

• Scaling the creativity-method audit and closing the tail-quality gap. The method evaluation is limited to open models with 7-8B parameters, so methods with complex instructions such as meta-prompting may behave differently on larger or frontier closed models. The authors also flag long-tail learning during post-training and pretraining as a promising but harder route than inference-time interventions.

Target Audience

This paper is most useful to researchers and practitioners who build or select LLM creativity benchmarks, to model developers working on post-training data and decoding strategies for open-ended generation, and to evaluation scientists who need a cheap sanity check on whether a benchmark measures what it claims. It will also interest readers following the broader debate about whether creativity can be measured at all, and those working on long-tail and distribution-shift evaluation more generally. Readers wanting only practical guidance can focus on Sections 5–7; the formal treatment in Section 3.2 is optional.

Authors’ abstract

Creativity benchmarks for large language models (LLMs) vary widely in task and metric design. This raises the question of whether they capture a common notion of creativity. We propose that this shared target can be formalized as task-specific performance in the long tail of a reference input-output distribution, which describes how common input-output pairs are relative to a chosen source of text. We ground this proposition in the novelty-quality product formulation and evaluate it empirically across 15 benchmarks and 26 open-weight models, using n-gram absence from the pretraining data as a novelty proxy. Our results show that creativity benchmarks lie further in the tail regions than standard benchmarks. Moreover, on both creativity and standard benchmarks, model rankings from higher-tail subsets of each benchmark align more closely with human creativity judgments in LM-Arena. Building on these findings, we introduce q@Tail($α$), a lower-cost creativity diagnostic that measures mean task quality on the tail fraction of an existing benchmark. Applying it to creativity-promoting methods reveals substantial room to improve LLM creativity through stronger tail performance.