CreBench: Human-Aligned Creativity Evaluation from Idea to Process to Product

AI summary

Overview

Research area: Multimodal large language models (MLLMs), human-aligned evaluation, and the cognitive science of creativity. The work sits at the intersection of benchmark construction, instruction tuning, and expert human annotation.

Technical level: Intermediate. Readers need a working familiarity with MLLM architectures (vision encoder, projection module, language decoder), instruction tuning, and correlation-based evaluation, but the paper's framing and rubric design are accessible without that background.

Scope: The paper introduces a benchmark (CreBench) and an instruction-tuning dataset (CreMIT) for measuring whether MLLMs judge creativity the way human experts do, plus an expert model (CreExpert) fine-tuned on that data.

What This Paper Is About

Creativity is abstract, subjective, and multidimensional, so it is hard to check whether a multimodal model's notion of "creative" matches what humans mean. Existing vision-language benchmarks focus on objective tasks such as visual question answering, captioning, and image-text retrieval, and standard metrics like BLEU, CIDEr, and CLIPScore do not capture novelty, usefulness, or imaginative depth. The paper builds a benchmark spanning creative idea, creative process, and creative product, along with a large expert-annotated instruction dataset, and uses them to train a model that aligns more closely with human creativity judgments than advanced MLLMs including GPT-4V and Gemini-Pro-Vision.

Key Contributions

1. CreBench, a fine-grained, multidimensional creativity evaluation benchmark covering three dimensions (creative idea, creative process, creative product) organized into 12 fine-grained indicators, each scored on a five-point behaviorally anchored rubric.
2. CreMIT, a multimodal creativity instruction-tuning dataset containing 2.2K multi-source data instances, 79.2K individual human feedback entries, and 4.7M multi-typed instructions generated from expert feedback across six question formats (Reasoning, What, How, Why, Y/N, MCQ).
3. CreExpert, a multimodal creativity evaluation expert model built on LLaVA-1.5-7B via instruction fine-tuning on CreMIT, which the paper reports outperforms state-of-the-art MLLMs including GPT-4V and Gemini-Pro-Vision.
4. Open-source release of the benchmark (12 fine-grained evaluation dimensions), the CreMIT dataset, and the CreExpert model with codes and checkpoints.

Main Findings

• CreExpert ranks first overall: On the 12-dimension evaluation, CreExpert achieves 65.50% overall, versus 29.27% for GPT-4V (rank 2, the best proprietary model tested) and 27.78% for Gemini-Pro-Vision (rank 3).
• Dimension-level performance is uneven: CreExpert scores 84.14% on Creative Idea, 72.19% on Creative Process, but 40.18% on Creative Product. Creative Product is the weakest area for every model, including CreExpert.
• Large margin over the strongest open baseline: CreExpert surpasses the state-of-the-art LLaVA-1.5-7B baseline (20.57% overall) by nearly 45%, and leads GPT-4V by more than 35% in overall score.
• Fine-tuning helps most on Creative Idea: Across the four tasks (Transport, Parking, Reach, Fence), overall gains on Creative Idea range from +52.16% (Parking) to +68.91% (Fence), with Fence showing the largest overall improvement at almost 69%. Originality gains range from +54.10% to +69.79%.
• Creative Process gains concentrate in early stages: Improvements in Immersion/Preparation and Divergence exceed 50% across most tasks; in the Reach task, Immersion/Preparation improves by +51.45% and Divergence by +54.67%.
• Creative Product gains vary by task: Transport shows the most prominent overall gain (+19.62%), with Novelty at +23.74% and System Complexity at +21.03%. The Reach task shows only modest gains (+3.21% overall) including a slight drop in Manufacturability (-2.81%).
• Annotation reliability: Inter-rater reliability across all dimensions averages Fleiss's kappa = 0.71 and ICC (2,1) = 0.78, which the authors describe as substantial agreement.
• Broader dataset coverage than prior creativity datasets: In a feature comparison against OpenSketch (2021), CID (2022), Cambridge AUT (2023), AesBench (2024), APDD (2024), and IDEA (2024), CreBench (2025) is the only entry marked as covering AIGC, human feedback, instruction tuning, open world, text, process, and image modalities.

Methodology in Plain English

The researchers ran a structured study with 512 secondary students recruited through stratified cluster sampling from five schools. All participants passed a color vision test and completed a baseline creativity assessment, and each completed three creative tasks. The tasks were four open-ended, real-world problem-solving scenarios — Transport, Parking, Reach, and Fence, with "cargo river crossing" given as an example of the task type. Participants produced textual ideas, behavioral logs, and visual outputs, yielding a multimodal record of how creativity unfolded rather than only a final drawing.

Three experts in educational creativity were recruited using the Consensual Assessment Technique. They underwent two training sessions to calibrate their standards on example cases, and their annotations were monitored through agreement checks, calibration meetings, and both automated and manual quality review. Experts scored each solution on 12 indicators grouped under three dimensions, producing roughly 79.2K human feedback entries over 2.2K evaluation instances. A separate distinction in the appendix clarifies that the "criteria" define what is assessed, while the "rubrics" define how it is scored on a five-point scale.

To turn those expert ratings into training signal, the authors prompted GPT to refine expert feedback into instruction-response pairs across six question types (Reasoning, What, How, Why, Y/N, and MCQ), producing about 4.7M instruction samples.

CreExpert follows the LLaVA-1.5 architecture: a CLIP-ViT-L14 vision encoder with 336×336 input resolution encoding images into 576 visual tokens, a two-layer MLP bridging visual and linguistic representations, and a language decoder initialized from Vicuna-v1.5. The authors instantiate it on the LLaVA-1.5-7B variant. The visual encoder is kept frozen, and fine-tuning is applied only to the projection module and the language model, using LoRA within the LLaMA-Factory framework on a server with eight NVIDIA Tesla A40 48GB GPUs. The dataset is split 50/50 into fine-tuning and evaluation sets, and Pearson Correlation Coefficient is used to measure consistency between model predictions and human feedback.

Why This Matters

Impact on research. Creativity evaluation for multimodal models has lacked a shared, human-grounded standard. CreBench supplies a rubric anchored in cognitive science, design reasoning, and creativity theory, and CreMIT supplies the supervision needed to train models against it. The finding that even the strongest proprietary models score 29.27% (GPT-4V) and 27.78% (Gemini-Pro-Vision) overall shows how large the alignment gap remains. The paper also argues its work is the first benchmark to systematically evaluate multimodal creativity.

Potential real-world applications:

• Education and assessment: The data comes from secondary students working on design tasks, so the framework speaks directly to evaluating student creative work with consistent, rubric-based scoring rather than single holistic judgments.
• Design and engineering ideation tools: The Transport, Parking, Reach, and Fence tasks are real-world problem scenarios involving physical constraints and manufacturability, making the rubric relevant to evaluating early-stage design concepts.
• Creative AI assistants and content tools: The idea/process/product breakdown gives builders a way to score generated content on originality, appropriateness, novelty, and aesthetic quality rather than relying on diversity or aesthetic scores alone.
• Model development and selection: The 12-dimension breakdown shows where a system is weak (for example, Creative Product at 40.18% for CreExpert), which can direct where further training data or modeling effort is needed.

Industry relevance. Companies building or deploying multimodal models for creative work need ways to check whether their systems' judgments match human expert judgments. The release of the benchmark, the 4.7M-instruction dataset, and the model checkpoints gives practitioners a starting point for that check without building an expert annotation pipeline from scratch.

Future Directions

• Closing the Creative Product gap. CreExpert's 40.18% on Creative Product trails its 84.14% on Creative Idea by a wide margin, and the authors note that product-level improvement varies by task and is constrained by domain-specific requirements. Improving assessment of final visual artifacts, aesthetics, and manufacturability is the clearest open problem.
• Handling task dependence. The Fence task improved by almost 69% overall while Reach gained only +3.21% and even dropped in Manufacturability (-2.81%), so the conditions under which the training data transfers well are unresolved.
• Scaling and enriching the benchmark. The current benchmark rests on 2.2K instances and four tasks from 512 students; whether the 12-dimension framework generalizes to other domains and participant populations is untested.
• Broadening beyond evaluation into generation. The paper frames its goal as improving models' perception of creativity; whether the same data can improve models' ability to produce creative work, not just judge it, is left open.

Target Audience

Researchers and engineers working on multimodal large language models, human-aligned evaluation, and benchmark design; creativity and design-cognition researchers interested in computational assessment; and practitioners in education technology or creative AI tooling who need a rubric-based way to evaluate creative outputs.

Authors’ abstract

Human-defined creativity is highly abstract, posing a challenge for multimodal large language models (MLLMs) to comprehend and assess creativity that aligns with human judgments. The absence of an existing benchmark further exacerbates this dilemma. To this end, we propose CreBench, which consists of two key components: 1) an evaluation benchmark covering the multiple dimensions from creative idea to process to products; 2) CreMIT (Creativity Multimodal Instruction Tuning dataset), a multimodal creativity evaluation dataset, consisting of 2.2K diverse-sourced multimodal data, 79.2K human feedbacks and 4.7M multi-typed instructions. Specifically, to ensure MLLMs can handle diverse creativity-related queries, we prompt GPT to refine these human feedbacks to activate stronger creativity assessment capabilities. CreBench serves as a foundation for building MLLMs that understand human-aligned creativity. Based on the CreBench, we fine-tune open-source general MLLMs, resulting in CreExpert, a multimodal creativity evaluation expert model. Extensive experiments demonstrate that the proposed CreExpert models achieve significantly better alignment with human creativity evaluation compared to state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro-Vision.