On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation

AI summary

Overview

• Research area: Natural Language Processing — specifically multimodal humor generation, i.e. automatically writing funny captions for cartoon images.
• Technical level: Advanced. The paper combines a linguistic humor theory (GTVH), multi-role LLM orchestration, retrieval over a joke database, and WordNet-based semantic scoring.
• Scope (one sentence): The paper proposes and evaluates HOMER, a humor-theory-driven, three-role LLM collaboration framework that generates funny image captions by extracting conflicting scripts, expanding humor targets through "imagination trees," and conditioning a caption generator on the result.

What This Paper Is About

Large language models are weak at generating genuinely funny captions for images, because existing LLM-based methods rely on generic prompting, reasoning chains, or self-improvement — which the authors argue capture surface-level humor language rather than deep humor logic and creative imagining. The paper's goal is to build a humor generation mechanism grounded in an established humor theory (the General Theory of Verbal Humor, GTVH) so that captions are both funnier and more interpretable. The authors target the "funny caption generation" task on New Yorker cartoon contest data, where the generated caption must compete against human-written captions.

Key Contributions

1. A humor-theory-grounded generation framework (HOMER). The method operationalizes GTVH's knowledge resources — script opposition, situation, target, narrative strategy, and language — as explicit stages of an LLM pipeline, rather than relying on the model's inherent humor ability.
2. A three-role multi-role LLM architecture. The framework separates work into (1) a conflicting script extractor that produces a situation description and a set of script oppositions, (2) a retrieval-augmented hierarchical imaginator that identifies humor targets and grows "imagination trees" for them, and (3) a caption generator that produces captions conditioned on the extracted knowledge.
3. An imagination mechanism with humor-relevance pruning. Targets are expanded through multi-view LLM free-association chains ("deep" imagination) merged with retrieval from a joke database ("broad" imagination), then pruned by a humor-relevance score H that combines relevance-opposition, humor-frequency, and POS-diversity terms.
4. Extensive evaluation on two New Yorker cartoon benchmarks plus a meme dataset, with automatic metrics (pass@K) and a validated LLM humor evaluator, including ablations over every module and over the scoring terms.

Main Findings

• HOMER beats all baselines on both benchmarks. On the two New Yorker cartoon datasets with GPT-4o as the base model, HOMER achieves average improvements of 8.62% on pass@1, 6.48% on pass@3, and 5.91% on pass@5 over seven baselines (HumorousAI, LoL, Phunny, CLoT, plus CoT, few-shot, and self-consistency). The abstract summarizes this as roughly ~7% improvement on average.
• Best reported numbers with GPT-4o: on Human in AI, HOMER scores 66.41 / 83.70 / 89.18 (pass@1/@3/@5) against #Top10 human captions, 73.40 / 88.38 / 92.57 against #200–#209, and 76.32 / 90.50 / 94.19 against #1000–#1009. On Electric sheep it scores 75.53 / 89.21 / 92.10 on the High-Humor group and 79.45 / 91.48 / 93.81 on the Low-Humor group.
• Gains hold across base models. With Claude-4, HOMER reaches 64.67 / 82.67 / 87.00 on #Top10 Human in AI. On weaker open models the relative gains are much larger: Qwen-VL (7B) improvements reach +23.4 on pass@1 for #200–#209, and LLaVA-1.5 (7B) improvements reach +46.7 on pass@5 for #Top10.
• Phunny is a severe underperformer. Across base models, Phunny records very low scores (for example 16.09 / 27.47 / 32.94 on GPT-4o #Top10 and 5.92 / 9.25 / 11.11 on Qwen-VL #Top10).
• Conflict scripts are the single most important module. In the ablation table, removing conflict scripts while keeping the situation description and imagination tree (I+D+T_im) causes the largest performance drop relative to the full I+D+C+T_im model (66.41 / 83.70 / 89.18 on #Top10).
• Every module helps. Image-only generation scores 20.20 / 38.30 / 51.00 on #Top10, showing that situation description, conflict scripts, and the imagination tree are each necessary.
• Demoting the full humor-relevance score hurts. Ablating the relevance-opposition term, the frequency term, or the diversity term each produces a significant performance drop in Figure 3.
• The method is robust to its two hyperparameters. The number of retrieved jokes k and the number of humor-relevant entities δ were varied over [3, 5, 7, 9] with stable results.
• GPT-5 is the most reliable humor judge tested. Ranking accuracy on human-written caption pairs across 200 contests: GPT-5 73.5% (Human in AI) and 70.0% (Electronic sheep); GPT-4.1 68.5% / 67.0%; Humor-tuned LLaMa3 60.0% / 58.0%; Qwen-Turbo 55.5% / 54.0%; LLaMa 3 53.5% / 52.0%.
• Generalization to memes. On a public ImgFlip meme dataset, HOMER scores 83.33 pass@1 and 96.67 pass@3, versus CLoT at 76.67 / 88.33 and HumorousAI at 75.00 / 80.00.

Methodology in Plain English

The authors start from GTVH, a humor theory that says a joke works by setting up one "script" (a semantic frame of expectations) and then conflicting it with an opposing script. They turn that theory into a pipeline of three LLM roles.

1. Conflicting script extractor. One LLM pass reads the cartoon and writes a detailed situation description — location, characters, facial expressions, actions — while flagging incongruous elements (in the paper's example, oversized coffee cups in an office meeting). A second prompt then extracts the explicit script oppositions, such as "gigantic coffee cups vs. normal ones."

2. Hierarchical imaginator. From the description and conflicts, the system picks key entities as candidate humor targets, looking both locally (fine-grained details in the description) and globally (obvious objects in the image). Each target is then expanded along two directions. "Deep" imagination builds a chain of free associations, where each new item is imagined only from its predecessor (for example coffee → milk → cow), with an empirically observed average chain length of about 4. "Broad" imagination embeds the query and retrieves the top-k most similar jokes from a collected joke database, tokenizes them, and adds those tokens as leaf nodes. A humor-relevance score prunes weak leaves. That score is the sum of three terms: a relevance-opposition term built from WordNet (Wu-Palmer semantic similarity plus a Jaccard-dissimilarity measure of conceptual opposition, combined as f(x) = x·exp(-x)), a humor-frequency term (geometric mean of token frequency and normalized joke frequency across retrieved jokes), and a POS-diversity term (fraction of WordNet parts of speech the token can take, as a proxy for pun potential).

3. Caption generator. The generator randomly samples a conflict script and a humor target, walks that target's imagination tree depth-first, samples one path, and prompts the LLM with the situation description, the conflict, the chosen path, and a narrative-strategy/linguistic-style setting.

Evaluation uses pass@K, which estimates the probability that a generated caption beats a human-written caption; the authors use an unbiased estimator over K ∈ {1, 3, 5}, averaged over five trials. Set-up details: top-k = 5 retrieved jokes, δ = 5 humor-relevant entities, temperature 1 for caption generation, temperature 0 for the evaluator, a Humor-tuned LLaMa3 fine-tuned on ranked caption pairs split 8:1:1, and all experiments run on 2 NVIDIA RTX 4090 (16 GB) GPUs.

Why This Matters

• Impact on research: The paper argues that explicitly encoding a humor theory into a multi-role LLM pipeline yields captions that are both funnier and more interpretable than reasoning-chain or self-improvement approaches, and that imagination (rather than pure reasoning) is the missing ingredient in machine humor. It also contributes a validated humor-evaluator comparison and a joke-retrieval corpus.
• Real-world applications (the paper itself does not enumerate these; they are implied by the task):
  • Assisting cartoonists, caption-contest entrants, and comedy writers with candidate joke captions.
  • Automatic humorous captioning for cartoons, memes, and social-media images.
  • Creative writing and advertising tools that need controlled, varied tonal output rather than literal descriptions.
  • Accessibility and content-description tooling where a light or witty tone is desired alongside factual description.
• Industry relevance: The framework is model-agnostic — it is tested on four base LLMs (GPT-4o, Claude-4, Qwen-VL, LLaVA-1.5), and the gains are largest on the small open models, which suggests the orchestration layer can compensate for weaker base models. The multi-role decomposition also fits the broader industry pattern of composing specialized LLM agents rather than prompting one monolithic model.

Future Directions

• Broadening visual and humor domains. The paper reports only a partial generalization test on ImgFlip memes; whether HOMER transfers to other humor genres (satire, stand-up clips, non-English humor) is not reported.
• Reducing dependence on proprietary models. Caption generation uses strong closed models and evaluation relies on GPT-5; the reported performance of open models such as LLaVA-1.5 and Qwen-VL remains far below GPT-4o, so lowering the capability floor is an open problem.
• Improving humor evaluation. The best evaluator tested, GPT-5, ranks human caption pairs correctly only 73.5% and 70.0% of the time, which bounds how much confidence can be placed in automatic humor scores without human judgment.
• Scaling and refining the retrieval component. The joke database is described as reorganized from 12 open-source joke datasets in the method section and 11 one-liner joke datasets in the experiments section (the paper's provided content reports both figures); the effect of corpus size and composition, and of the pruning threshold δ, is only explored over a narrow range ([3, 5, 7, 9] for k and δ).

Target Audience

Researchers and practitioners in natural language processing and multimodal machine learning who work on humor generation, creative text generation, or LLM agent design. The paper is most useful for readers already comfortable with LLM prompting pipelines, retrieval-augmented generation, and automatic evaluation metrics, and for computational humor researchers interested in grounding generation in linguistic humor theory. Readers looking for a beginner-level tutorial on humor generation, or for extensive appendix detail, should note that the appendices (B.1–B.10, C, G) are only referenced in the provided content and not included.

Authors’ abstract

Humor is a commonly used and intricate human language in daily life. Humor generation, especially in multi-modal scenarios, is a challenging task for large language models (LLMs), which is typically as funny caption generation for images, requiring visual understanding, humor reasoning, creative imagination, and so on. Existing LLM-based approaches rely on reasoning chains or self-improvement, which suffer from limited creativity and interpretability. To address these bottlenecks, we develop a novel LLM-based humor generation mechanism based on a fundamental humor theory, GTVH. To produce funny and script-opposite captions, we introduce a humor-theory-driven multi-role LLM collaboration framework augmented with humor retrieval (HOMER). The framework consists of three LLM-based roles: (1) conflicting-script extractor that grounds humor in key script oppositions, forming the basis of caption generation; (2) retrieval-augmented hierarchical imaginator that identifies key humor targets and expands the creative space of them through diverse associations structured as imagination trees; and (3) caption generator that produces funny and diverse captions conditioned on the obtained knowledge. Extensive experiments on two New Yorker Cartoon benchmarking datasets show that HOMER outperforms state-of-the-art baselines and powerful LLM reasoning strategies on multi-modal humor captioning.