CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

AI summary

Overview

• Research area: Natural Language Processing, specifically instruction tuning of large language models (LLMs) for creative text generation and reinforcement learning (RL).
• Technical level: Intermediate. Readers will benefit from familiarity with instruction tuning, decoding-time routing, LoRA fine-tuning, and RL methods such as GRPO, though the paper explains each briefly.
• Scope: The paper introduces a single-checkpoint instruction-tuning method that injects creativity-triggering tokens, a graph-based structural diversity metric, and evaluates both on narrative generation across five models (7B–32B) plus a math RL experiment.

What This Paper Is About

Post-training (alignment) makes LLMs better at following instructions and reasoning, but it tends to make their outputs more homogeneous and less creative. Existing fixes that combine a diverse base model with an aligned model require running two models at inference time, which costs extra compute and memory and requires access to base models that are not always released. The authors' goal is to teach one unified model to decide for itself when to generate creatively, recovering base-model diversity while keeping aligned-model quality.

Key Contributions

1. CreativeInstruct, an instruction-tuning method that trains a single LLM on writing-related instruction data in which base-model spans are wrapped in special [StartCreativity] and [EndCreativity] tokens. At test time the model emits these tokens itself, switching between creative and aligned behavior without a second model.
2. LLM-GED (LLM Graph Edit Distance), a structural diversity metric in which an LLM judge abstracts each story into an event graph (nodes for entities/events, edges for semantic relations and temporal ordering), canonicalizes entities, and estimates normalized graph edit distance across all story pairs.
3. Diversity and quality evaluation on narrative generation across five models ranging from 7B to 32B, comparing against Instruct, BACo (test-time routing), a Distill ablation without the creativity tags, and CrPO (for LLaMA-3.1 8B only), plus a human evaluation.
4. A demonstration that creativity helps RL, by applying GRPO to a CreativeInstruct checkpoint versus a post-trained checkpoint and evaluating on MATH and AMC.

Main Findings

• Diversity gains over the aligned model: LLaMA-3.1 8B trained with CreativeInstruct yields approximately 48% relative gains in semantic diversity and 63% gains in structural diversity over the base instruct model (as reported in the introduction).
• Beats test-time routing: Despite training on data generated by BACo, CreativeInstruct outperforms BACo at test time across model families and sizes, with average gains of 29% in semantic diversity and 28% in structural diversity, while requiring only one model at inference.
• Specific LLaMA-3.1 8B improvements: +0.149 in MiniLM cosine dissimilarity over Instruct and +0.203 over BACo; on LLM-GED, 17-point gains over BACo and Instruct.
• Qwen2.5 32B: improves over Instruct on most metrics, with a +0.082 gain in Qwen Cos-D.
• Creativity tags matter: The Distill baseline, trained on the same BACo-generated corpus without the [Start-]/[EndCreativity] tags, generally underperforms CreativeInstruct on automatic quality metrics, and CreativeInstruct achieves higher LLM-GED in every setting.
• Quality is largely preserved: On coherence, fluency, relevance (Likert 1–5), and Writing Quality Reward Model (WQRM, 1–10), CreativeInstruct remains competitive; for LLaMA-3.1 8B, Qwen2.5 32B, and Qwen3 8B it obtains the highest WQRM within its model family.
• Existing methods can still win on some metrics: On Qwen3 32B, Distill does better across most automatic diversity metrics, though CreativeInstruct remains more structurally diverse by LLM-GED. On Qwen3 8B, Instruct itself has higher Cos-D (M) (0.399 vs 0.369) than CreativeInstruct.
• Human evaluation favors creative outputs: Annotators rated CreativeInstruct generations as more creative than aligned-model generations in 70.3% of comparisons (significant under a two-sided binomial test, n=50), and preferred it on diversity in 57.4% of cases. Inter-annotator agreement was κ=0.720 for creativity, κ=0.417 for diversity, and κ=-0.167 for quality; the authors omit quality from human comparisons and rely on automatic metrics instead.
• Less entity repetition: Proper noun uniqueness was 37.1% vs 26.6% for the strongest baseline (Distill) at the prompt-group level, and 24.7% vs 16.8% corpus-wide, more than doubling Instruct's 18.1% and 12.0% (Mann-Whitney U test, p<0.001).
• Scaling helps: Diversity scores increase with more Tülu SFT samples, with LLaMA-3.1 8B performing best at 12,000 samples and scores not yet plateaued.
• Data domain matters: Training on Narrative Generation data instead of Tülu data (2,020 training samples) produced lower average scores than general-purpose Tülu data even at the same dataset size.
• LLM-GED validates well: In a test with 100 paired stories across four variation settings, LLM-GED correlated highest (0.889) with the reference diversity ranking, above Cosine Dissimilarity (all-MiniLM) at 0.852, NLI Diversity at 0.687, Cosine Dissimilarity (Qwen) at 0.698, Vendi Score (SimCSE) at 0.654, Semantic Entropy at -0.410, and all lexical metrics (highest lexical was Self-ROUGE-L at 0.678). A deterministic two-pass pipeline produced statistically equivalent scores (mean difference = -0.012, paired t-test p = 0.56).
• Creativity improves RL: Applying GRPO to a CreativeInstruct checkpoint versus the post-trained checkpoint gave roughly 4% gain on AMC and roughly 5% points on MATH. In the table, Instruct baseline was 0.374 MATH / 0.432 AMC; Instruct + RL 0.409 / 0.438; CreativeInstruct baseline 0.424 / 0.428; CreativeInstruct + RL 0.459 / 0.478. The authors note CreativeInstruct alone does not consistently improve math ability but creates a better substrate for RL, enabling better out-of-distribution generalization.

Methodology in Plain English

The authors start with a subset of the Tülu V3 SFT dataset filtered for English writing-related prompts: 4,000 unique prompts, 3 outputs each, for 12,000 training samples. For each prompt they run BACo's "prob+punc" routing variant, which generates tokens from either a base model or its aligned counterpart: punctuation and formatting tokens go to the aligned model to keep sentences coherent, high-entropy tokens go to the base model for diversity, and low-entropy tokens go to the aligned model.

They then track which model produced each token and wrap contiguous base-model spans (and spans where both models assigned the same probability, within a delta of 0.005) in [StartCreativity] and [EndCreativity] markers. Aligned models are fine-tuned on this tagged data with LoRA (rank 32, alpha 64, dropout 0.05, up to 10 epochs with early stopping patience 3, batch size 8, learning rate 2×10⁻⁴, cosine scheduler, warmup ratio 0.03). At inference the model itself decides where to insert these markers, so no second model is needed. Decoding matches BACo: temperature 1.0, top-p 0.9, up to 512 new tokens.

For the new diversity metric, an LLM judge represents each story as an abstract graph of entities, events, and temporal/semantic relations, canonically relabels entities (e.g., Character1, Location1) so that surface wording does not count as diversity, and estimates the normalized graph edit distance for all story pairs in a single prompt, producing an N×N matrix that is averaged into one score. Evaluation uses the Narrative Discourse dataset, semantic metrics (cosine dissimilarity with all-MiniLM and Qwen-3 embeddings, semantic entropy, Vendi Score with SimCSE, NLI Diversity), lexical metrics (Dist-n, EAD-n, Self-BLEU, Self-ROUGE-L, Vendi N-Gram), and quality measures (coherence, fluency, relevance, and WQRM). All LLM-based metrics use the gpt-5-mini-2025-08-07 variant.

For RL, two Qwen3 8B models — the standard post-trained one and a CreativeInstruct one — are trained with GRPO for 1,000 steps using HuggingFace TRL, with 8 rollouts and context length 2048, on a 12k training split of MATH, then evaluated on MATH and AMC, averaged over three random seeds.

Why This Matters

The paper reframes diversity and quality as a balance to be learned rather than a trade-off to be accepted, and shows that the benefits extend beyond creative writing into reasoning tasks via RL. It also offers a cheaper alternative to multi-model decoding: one checkpoint instead of two, no test-time base-model access, and better results than the routing strategy used to generate its training data. The LLM-GED metric addresses a measurement gap, since lexical and embedding metrics miss narrative-level structural repetition.

Real-world applications suggested by the paper's scope:

• Creative writing assistants and story generation tools that must produce varied rather than formulaic drafts.
• Synthetic data generation pipelines, where diverse rollouts improve coverage of downstream training corpora.
• Post-training for reasoning models, where diverse rollouts support exploration in RL and better out-of-distribution generalization.
• Deployments where base models are unavailable or where running two models at inference is too expensive in memory or latency.

For industry, the practical appeal is a single fine-tuned checkpoint that matches or exceeds a two-model inference setup, with LoRA training on general-purpose instruction data that continues to improve as the dataset scales.

Future Directions

• Scaling the training corpus further: The authors report that diversity scores had not plateaued at 12,000 samples, leaving open how far the gains extend.
• Automatic creativity-tag placement: The method currently uses a fixed routing heuristic (BACo prob+punc, with a probability delta of 0.005) to decide where tags go in training data; learning an optimal placement policy is unexplored.
• Broader RL investigation: Only GRPO on Qwen3 8B with math data is tested. Whether the diversity advantage generalizes to other RL algorithms, model sizes, and domains such as code generation is not reported.
• Improving the structural metric: LLM-GED uses an LLM to estimate edit distance; the authors compare it to a costlier deterministic pipeline (networkx branch-and-bound capped at 20 seconds per pair) and note the two are statistically equivalent, but a fully reliable automated structural metric remains an open problem.

Target Audience

Researchers and practitioners working on LLM post-training, instruction tuning, and decoding strategies will get the most from this paper, as will those studying diversity and creativity evaluation for text generation. It is also relevant to RL and reasoning researchers interested in how rollout diversity affects policy learning. The clarity of the pipeline — data generation, tagging, LoRA training, evaluation — makes it accessible to graduate students and engineers comfortable with standard fine-tuning workflows.

Authors’ abstract

While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.g., story generation) as well as those that require it implicitly, e.g., reinforcement learning (RL). We instead propose CreativeInstruct, a scalable instruction-tuning method that teaches LLMs to balance creative, base-model-like generations with the quality of post-trained models, by learning to inject special [StartCreativity] spans that bias generation toward creativity. Furthermore, we introduce a structural diversity metric based on graph edit distance, which captures narrative level variation missed by purely lexical and semantic metrics. On narrative generation, CreativeInstruct matches or exceeds the diversity of both multi-model baselines and distilled variants of their outputs, without sacrificing quality or requiring multiple models at inference time. These results are mirrored in our human evaluation, where we find that annotators rate CreativeInstruct generations as more creative than the post-trained LLMs' generations in 70.3% of cases. We also show the benefits of creative models as a substrate for RL: GRPO applied to a CreativeInstruct checkpoint improves by ~4% on AMC and ~5% points on MATH over the same training applied to the post-trained checkpoint.