Jokes Aside: Measuring the Semantic Distance of Double Meanings

AI summary

Overview

Research area: Computational humor / natural language processing, specifically the use of embedding-based semantic metrics to model why puns and wordplay are funny.

Technical level: Intermediate — the paper assumes familiarity with word embeddings and similarity metrics, but the underlying idea (measuring how far apart two meanings of a word are) is intuitive.

Scope: The paper re-tests and extends a small set of embedding-based metrics for predicting how funny template jokes are, across three humor datasets and two embedding models.

What This Paper Is About

Earlier work on computational humor proposed that a joke works when a punchline word is strongly associated with two different setup words, is relatively rare, is ambiguous, and when those two setup words mean very different things. Those ideas were originally tested with older tools such as Google Ngrams and Word2Vec. This paper asks whether the same ideas hold up when re-implemented with modern contextual embedding vectors, and whether a newly proposed metric called symmetry adds anything. The goal is to see whether these semantic metrics can actually predict human humor ratings.

Key Contributions

1. Revisiting three existing metrics with modern embeddings. Three of the five metrics proposed by Winters et al. (2019) — obviousness, compatibility, and comparison — are re-implemented using word embeddings rather than Ngram statistics.
2. Introducing a new metric: symmetry. Defined as the closeness of the punchline word Z to both of the setup words X and Y, this metric is proposed here for the first time.
3. Multi-model, multi-dataset evaluation. Embedding vectors are collected with two models (OpenAI text-embedding-3-small and MiniLM all-MiniLM-L6-v2) across three datasets: JokeJudger, Expunations, and rJokes.
4. Dataset expansion. Expunations and rJokes are extended by adding paired sentences that express the ambiguous word at the heart of each joke in each of its two distinct meanings, giving the models explicit access to the double meaning.

Main Findings

• Embedding-based metrics predict humor poorly. Models trained on the proposed metrics performed badly at predicting humor ratings. On JokeJudger, the best model reached 57.1% accuracy, which is below the 61.5% baseline. Performance on Expunations and rJokes was even lower.
• Symmetry stands out despite the overall failure. The newly introduced symmetry metric appears consistently associated with higher-rated jokes.
• Symmetry may be necessary but not sufficient. The authors suggest it captures a required property of humor — present in funny jokes — without being enough on its own to determine whether something is funny.
• Modern embeddings do not automatically rescue older hypotheses. Re-running earlier ideas with contextual embeddings did not translate into better humor-rating prediction in this setup.

Methodology in Plain English

The jokes follow a fixed template: "I like my X like I like my Y, Z" — for example, "I like my ice like I like my dreams, crushed." Here X and Y are the two things being compared and Z is the punchline word that links them.

The researchers turn the earlier verbal hypotheses into numbers. They compute how strongly Z is associated with X and with Y, how rare or obvious Z is, how ambiguous Z is, and how far apart X and Y are in meaning — all using vector similarity from embedding models. They add the new symmetry measure, which checks how close Z sits to both X and Y at once.

They then train models on these numeric features and see how well those models predict the humor ratings people gave the jokes. Two different embedding models supply the vectors, and the experiments run over three joke datasets. For two of those datasets they also add extra sentence pairs that spell out each joke's ambiguous word in both of its meanings, so the system does not have to infer the double meaning on its own. Results are compared against a baseline.

Why This Matters

The paper is a check on a popular assumption: that if you can measure the semantic ingredients of a pun with modern embeddings, you can predict whether it lands. The answer here is largely no, at least for this template and these datasets — which is useful negative evidence for a field that leans heavily on embedding similarity.

Real-world applications this could inform:

• Humor generation and writing assistants that try to rank or filter candidate puns automatically.
• Wordplay detection and machine translation, where systems must recognize that a word is carrying two meanings at once.
• Dialogue systems and chatbots that need to decide whether a joke is likely to land before delivering it.
• Language learning and linguistic research tools that surface ambiguous expressions and their multiple senses.

Industry relevance: Teams building creative-writing tools, conversational agents, or content-ranking systems often reach for embeddings as a cheap proxy for "is this funny or clever." This paper suggests that proxy is weak on its own, and that a metric like symmetry may be a better building block than association or rarity measures — while still not being sufficient.

Future Directions

• Determine what the metrics miss. If symmetry is necessary but not sufficient, what additional signal (surprise, timing, context, cultural knowledge) would close the gap?
• Improve the evaluation setup. The abstract does not report how the expanded paired-sentence data affected results; a natural next step is to isolate the contribution of that expansion.
• Compare embedding types more systematically. With only two embedding models tested, it is open whether stronger or differently trained embeddings change the picture.
• Test beyond this joke template. Whether these findings generalize to other forms of wordplay, other languages, and non-template humor is unresolved.

Target Audience

Researchers in computational humor and computational linguistics; NLP practitioners who use embedding similarity as a proxy for semantic relatedness; and students or engineers interested in why automatic humor evaluation remains difficult. Readers looking for a working humor classifier will not find one here — the value lies in the negative result and the new symmetry metric.

Authors’ abstract

Large language models have significantly enriched the toolkit for computational humor research, particularly in the automated generation of jokes and puns. A key innovation, contextual embedding vectors, offers new opportunities to revisit and refine earlier hypotheses. Notably, Petrovic and Matthews (2013) proposed a joke generation model based on the scheme "I like my X like I like my Y, Z" (e.g. "I like my ice like I like my dreams, crushed"). They suggested that joke hilarity increases with: a) frequent association of Z with X and Y, b) rarity of Z, c) ambiguity of Z, and d) meaning distance between X and Y. Building on this, Winters et al. (2019) proposed a set of metrics, based on Google Ngrams and Word2Vector. In this work, three out of their five metrics are revisited with word embeddings: obviousness, compatibility, and comparison. Another measure, symmetry, defined as closeness of Z to both X and Y, is introduced here for the first time. Two models were used to collect the embedding vectors (OpenAI text-embedding-3-small and MiniLM all-MiniLM-L6-v2) on three datasets: JokeJudger, Expunations, and rJokes. The last two datasets, Expunations, and rJokes, were expanded by adding paired sentences that captured the ambiguous expression at the core of each joke in its two different meanings. Results revealed that models trained on the proposed metrics performed poorly in predicting humor ratings: on JokeJudger, the best model achieved 57.1% accuracy, below the 61.5% baseline, while performance on Expunations and rJokes was even lower. Nevertheless, the symmetry metric seems consistently associated with higher-rated jokes, suggesting it may capture a necessary -though not sufficient- property of humor.