Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs

AI summary

Overview

Research area: Natural Language Processing / Computational Linguistics, specifically comparative analysis of human- versus LLM-generated dialogue and computational humor. Technical level: Intermediate — the paper combines a multi-agent LLM pipeline with Systemic Functional Linguistics (SFL) annotation, so some familiarity with dialogue systems and discourse analysis helps. Scope: A comparative study of four Spanish-language improvisational comedy scenes performed by human actors versus GPT-4o agents, analyzed with automatic lexical/syntactic measures and manual SFL-based appraisal annotation.

What This Paper Is About

The authors ask what actually distinguishes "synthetic" language (text produced by a large language model) from "spontaneous" language (speech produced by humans in real-time interaction), using humorous improvisational dialogue as a deliberately hard test case. They build a human corpus from a Mexican television improv show and a matching synthetic corpus generated by multiple GPT-4o agents role-playing the same scenes, then compare the two at the level of turn structure, lexical richness, syntax, and evaluative discourse. The goal is to "spot the artificial" — to identify which features of human spontaneous interaction current LLMs fail to reproduce.

Key Contributions

1. A reproducible methodology and open multi-agent framework (code and data at github.com/ximenina/DialogicHumor) in which each LLM agent plays one character, shares conversation history, and is assigned the next speaker at random rather than following a fixed script.
2. A paired Spanish-language corpus: 4 human improvisation acts from the TV program ¿Quién dijo yo? (309 turns, 16,931 tokens) and 4 synthetic counterparts generated with GPT-4o (36, 12, 12, and 12 turns per act; 23,037 tokens in the annotated set, 155,178 tokens across repeated generations).
3. A combined analytic pipeline: automatic measures (turn length in words and characters, Moving-Average Type–Token Ratio at window sizes 25/50/100/150, POS tags and dependency relations via Stanza, named entities via GPT-5) alongside manual annotation by a trained linguist using Appraisal Theory from SFL.
4. Evidence that the gap between human and machine dialogue is located less in grammar than in interactional and evaluative behavior — turn dynamics, attitude distribution, and willingness to be negative or confrontational.

Main Findings

• Turns are far longer in synthetic dialogue: Average turn length is 9.42 words in the natural corpus versus 51 words in the synthetic corpus. Most natural turns fall between roughly 5 and 20 words, while synthetic turns cluster between 40 and 80 words, producing fewer rapid exchanges.
• Synthetic dialogue is lexically richer, which is framed as a failure: MATTR is consistently higher for synthetic text at every window size — 0.91 vs 0.80 (window 25), 0.82 vs 0.71 (50), 0.70 vs 0.62 (100), 0.62 vs 0.56 (150). The authors attribute this to human speakers' real-time reuse, repetition, and discourse routines, which LLM agents do not reproduce.
• Synthetic dialogue is more syntactically elaborated: The synthetic corpus shows higher frequencies of adjectives and adjectival modifiers (amod), determiners (det), conjunctions, and coordinating conjunctions (cc), indicating fuller noun phrases and more subordinate/coordinated clause structure. The natural corpus shows more parataxis — short, independent, fragmented clauses.
• Humans reference people and places more: Named-entity distributions are broadly similar across both corpora (person and location entities dominate in both), but the spontaneous dialogues contain a higher relative frequency of person and location entities and of proper nouns (propn), despite being much shorter.
• Attitude profiles diverge sharply: In human improvisation, affect is the most frequent attitudinal resource (55.6%), followed by appreciation (24.4%) and judgement (20%). In synthetic dialogue, appreciation dominates (51.1%), with affect at 32.8% and judgement at 16.1%. Judgement is the most stable category across both.
• LLM dialogue skews positive and polite: Human improvisers produce a balanced mix of positive and negative evaluations and are unconcerned with maintaining a socially desirable image; LLM agents lean toward supportive, compassionate, agreeable evaluations, which the authors link to alignment mechanisms and documented sycophancy.
• Rhetorical elaboration is inverted relative to expectation: Spontaneous improvisation rarely contains metaphors, similes, or elaborate comparisons, plausibly because of real-time constraints, whereas synthetic dialogue uses these devices more often.
• Engagement and graduation differ qualitatively: Human improvisation is described as inherently heteroglossic, with participants continuously expanding on and challenging each other, while synthetic dialogue constructs more contrastive and less collaborative positions. For graduation, spontaneous speech shows more intensification through repetition, while synthetic language is balanced between intensification and quantification.
• Prosody is out of reach: The paralinguistic features of spontaneous speech cannot be imitated by the models; textual substitutes such as emojis, exclamation marks, or unconventional orthography fall short.
• LLMs write dialogue rather than converse: Although scenes have coherent openings and closures, agents often had to be reminded through system prompts to stay in character and not to write other characters' lines — leading to the conclusion that current LLMs are better at writing dialogue than at engaging in spontaneous interaction.

Methodology in Plain English

The researchers picked four scenes from a Spanish-language television improv comedy show, where actors invent scenes on the spot from audience and host prompts. They transcribed the audio with an automatic speech recognition model and had a linguist correct the transcripts.

To create a machine comparison, they did not ask a single model to write a whole script. Instead, they gave each of several GPT-4o instances its own character description and set them loose in a shared conversation, with the same scenario description the human actors received. One rule told the agents to follow the "yes, and…" principle of theatrical improvisation, and the next speaker was chosen at random each turn so the exchange would not become predictable. The model temperature was set to 0.8 for creativity, and each dialogue was regenerated about 10 times in most cases; scenes were kept short because prior work reports persona drift after about 8 turns.

Only one generated dialogue per scene was manually annotated in depth, because linguistic annotation is expensive, but the repeated generations were kept and checked statistically to confirm the patterns were not a fluke of one sample.

The comparison then ran on two tracks. Automatic tools measured turn length, lexical diversity (MATTR), part-of-speech and dependency patterns, and named entities. Separately, a trained linguist manually coded the dialogues for evaluative meaning using Appraisal Theory, tagging instances of affect (emotion), judgement (evaluating behavior by social norms), and appreciation (evaluating things and situations), plus a preliminary look at engagement and graduation.

Why This Matters

The paper argues that rigorous linguistic theory is a necessary complement to NLP benchmarking: standard evaluations reward fluent, well-formed, helpful text, which is precisely what makes machine dialogue detectable as artificial in interactional settings.

Real-world applications:

• Evaluating conversational agents and AI companions for conversational naturalness, not just correctness.
• Character and dialogue design in games, interactive fiction, and virtual performance, where multi-agent turn-taking matters.
• Detecting machine-generated dialogue and authorship in forensic or content-moderation contexts.
• Informing alignment and safety work, since the tendency toward uniformly positive, agreeable evaluations is a measurable behavioral signature with practical consequences.

Industry relevance: Teams building role-play, entertainment, customer-facing, or simulation agents can use these diagnostics to test whether their systems behave like participants or like scriptwriters, and the observed drift toward politeness and positivity is directly relevant to sycophancy concerns in deployed assistants.

Future Directions

• Extending the fine-grained analysis of the engagement and graduation subsystems, which the paper explicitly leaves for future work.
• Testing whether the observed gaps shrink with instruction tuning, fine-tuning on persona-based or spontaneous-speech data, or adaptation to spoken registers.
• Scaling beyond four acts and one model (GPT-4o), since manual annotation constrained the corpus to 309 natural turns.
• Incorporating prosody and multimodal signals, which the authors identify as the aspect current text-based LLMs most clearly cannot reproduce.

Target Audience

Linguists interested in what corpus and discourse-analysis frameworks reveal about machine language; NLP researchers working on dialogue generation, persona modeling, and computational humor; and practitioners building conversational agents or character-driven interactive systems who need concrete diagnostics for conversational naturalness.

Authors’ abstract

We propose a methodology for investigating the differences between natural and synthetic language in a challenging setting: humorous improvisational dialogue. Our experimental setting compares human-produced comedy improvisations with the dialogues generated by LLMs in a multi-agent environment that we designed. The analysis combines automated linguistic measures with manual annotation grounded in Systemic Functional Linguistics (SFL). The results show that, although LLMs can create coherent and creative scenes, they often fail to reproduce key characteristics of spontaneous spoken interaction. In addition, LLM-generated interactions display a strong tendency toward politeness and positive evaluations, contrasting with the broader range of attitudes found in human humorous improvisation. Our evidence highlights both the strengths and limitations of current LLMs in modeling dialogic humor and demonstrate the value of integrating linguistic theory with NLP methods for the analysis of machine-generated language