Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection

AI summary

Overview

• Research area: Natural Language Processing, specifically sarcasm (verbal irony) detection with Large Language Models, retrieval-augmented generation, and prompt engineering.
• Technical level: Intermediate. The paper assumes familiarity with LLMs, prompting strategies, NER/POS tagging, and retrieval concepts such as BM25, but explains its pipeline steps clearly.
• Scope: The paper extends Pragmatic Metacognitive Prompting (PMP) with two context-injection strategies — web retrieval of non-parametric knowledge and elicitation of an LLM's own internal knowledge — and evaluates five prompting variants on three sarcasm datasets in English and Indonesian.

What This Paper Is About

Sarcasm detection is hard because sarcastic meaning often depends on subtle pragmatic, cultural, or local cues that a model may not have in its training data. The authors start from Pragmatic Metacognitive Prompting (PMP), a state-of-the-art prompting method for sarcasm detection, and add retrieved context for words the model likely does not understand — either from web search or from the model's own internal knowledge. The goal is to test whether supplying that context improves sarcasm classification across English and Indonesian datasets.

Key Contributions

1. Demonstrating the role of context in LLM sarcasm detection. The paper shows that both non-parametric knowledge (from web retrieval) and internal knowledge awareness act as valuable context when applied on top of the PMP method.

2. Comparing two ways to identify words the LLM does not understand. The authors introduce and contrast Token Tagging (NER plus POS tagging, restricted to entities and proper nouns) against an LLM-Based extraction in which the model itself flags words it cannot recognize or infer.

3. Implementing a simple retrieval pipeline and measuring its effect. The retrieval uses Google Search API with BM25 chunk ranking and LLM-based cleaning for the non-parametric path, and LLM-only definition generation for the internal-knowledge path. The authors report that even this simple retrieval produces a large performance margin, suggesting better retrieval methods could help further.

4. Evaluating across linguistically diverse datasets. The pipelines are tested on SemEval-2018 Task 3 and MUStARD (both English) plus the Indonesian Twitter dataset, which the authors describe as an extreme linguistic challenge due to multilingual and culture-specific slang.

Main Findings

• Non-parametric retrieval gives the largest gain on Indonesian Twitter: The abstract reports a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method.

• Self-knowledge retrieval helps on the English datasets: Internal-knowledge retrieval improves macro-F1 by 3.29% on SemEval and by 4.08% on MUStARD.

• PMP's baseline performance is uneven across datasets: Table I shows PMP with GPT-4o reaching 0.7942 accuracy and 0.7765 macro-F1 on MUStARD, and 0.8668 accuracy and 0.8318 macro-F1 on SemEval 2018. The same table shows PMP with LLaMA-3-8B at 0.5358 accuracy / 0.5469 macro-F1 on MUStARD and 0.7821 accuracy / 0.7765 macro-F1 on SemEval 2018.

• PLMs remain competitive on some datasets: In Table I, RoBERTa reaches 0.8680 accuracy / 0.8770 macro-F1 on MUStARD and 0.7500 / 0.7200 on SemEval 2018; DistilBERT reaches 0.8700 / 0.8770 on MUStARD; DC-Net-RoBERTa reaches 0.7090 / 0.6870 on SemEval 2018. On the Twitter Indonesian dataset, XLM-R-Large reaches 0.8885 accuracy / 0.7692 macro-F1, while zero-shot mT0XL reaches 0.2494 accuracy / 0.3989 macro-F1.

• Context matters most where the model lacks background: The authors attribute the gains to supplying definitions for culturally specific slang, references, or unknown terms. They explicitly avoid Google Search API retrieval on datasets made up mainly of standard English captions such as MUStARD, because the LLM already holds relevant background knowledge and web retrieval tends to add irrelevant snippets and noise.

• Keyword-extraction and retrieval pairings are matched to dataset properties: Token Tagging plus LLM-only retrieval is used for English datasets; LLM-based extraction plus Google Search API is used for the Indonesian Twitter dataset and SemEval-2018, because spaCy models cannot reliably tag Indonesian slang as entities or proper nouns.

• Reported experimental-table values are partially available: The provided content includes only the first row of the main results table (Table III), which shows Qwen-3-8B with PMP on the Indonesian Twitter dataset at 0.6078 accuracy, 0.5189 precision, 0.5219 recall, and 0.5173 macro-F1. The remainder of the table is truncated in the supplied content, so per-model and per-pipeline values beyond this row are not reported here.

Methodology in Plain English

The authors keep the two-step PMP structure intact — a first call that produces a preliminary pragmatic analysis of the input, and a second call that reflects on that analysis and issues a final YES/NO sarcasm label — and then decide how and where to insert extra context.

Context is produced in two stages. First, keywords are extracted in one of two ways:

• Token Tagging: a spaCy pipeline tags each token with its entity type and part of speech, and the system keeps only entities and proper nouns.
• LLM-Based: the model is asked directly which words it does not recognize or cannot infer from prior knowledge.

Second, those keywords are turned into usable definitions in one of two ways:

• Google Search API + LLM: keywords are used as search queries; retrieved documents are split into chunks, ranked by BM25, and the top chunks are passed to an LLM to summarize and extract key meanings. BM25 is used with the standard hyperparameters k1 = 1.2 and b = 0.75.
• LLM-only: the LLM generates a definition for each keyword purely from its internal parametric knowledge, which is faster and cheaper.

These pieces are combined into five prompting variants: the PMP baseline, PMPWL (Token Tagging plus LLM-only word info), PMPWG (LLM-based extraction plus Google Search API word info), and their few-shot versions PMPWL-FS and PMPWG-FS. Word information is placed in the user prompt during the first PMP call so the model encounters it while first reading the input; few-shot examples (k = 2, written manually per dataset theme) are placed in the system prompt during the second call, where the factor-by-factor reasoning happens.

Experiments ran on a machine with an Nvidia RTX 3090 GPU and 24 GB of VRAM, using spaCy's en_core_web_sm for tagging and three open-source 8-billion-parameter models — Qwen-3, Llama-3, and Llama-3.1 — served through Ollama with default decoding settings. Evaluation uses standard classification metrics (accuracy, precision, recall, F1) with macro averaging because the SemEval and Indonesian Twitter datasets are imbalanced.

Why This Matters

The paper argues that sarcasm detection is a "System II" task — one requiring slow, deliberate, multi-step reasoning — and remains unsolved even as "System I" tasks such as sentiment analysis and spam detection have largely been addressed. Its practical message is that an LLM's failures on sarcasm are often failures of grounding: the model does not know a slang term, a local reference, or an entity, so its downstream pragmatic reasoning degrades. Supplying that grounding selectively, only where needed, is shown to move macro-F1 substantially.

Real-world applications:

• Social media monitoring: flagging sarcastic posts so that downstream sentiment or brand-monitoring systems do not read ironic praise or ironic criticism literally.
• Review and feedback analysis: the paper's own motivating example — a music review that calls an album "incredible" because it put the listener to sleep — illustrates the risk that sentiment models misread sarcasm in product, film, or album reviews.
• Culturally and linguistically diverse moderation and analysis: the Indonesian Twitter results suggest value in handling regional slang and code-mixed content that standard entity recognizers miss.
• Conversational and multimodal systems: MUStARD is drawn from TV dialogue, pointing toward sarcasm handling in dialogue assistants and entertainment-content analysis.

Industry relevance: the work targets a cheap, plug-in improvement — prompting changes plus lightweight retrieval — rather than model retraining, which matters for teams deploying LLMs under cost and latency constraints. It also provides an open-source implementation at https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base.

Future Directions

• Optimizing retrieval of relevant contextual information. The abstract lists this explicitly as future work, motivated by the authors' observation that their simple retrieval produced a large margin, implying better retrieval could do more.
• Measuring how retrieval quality affects performance. The authors state they intend to examine this directly, since they note that retrieved snippets can introduce noise as well as signal.
• Testing higher-capacity models. The authors note that hardware constraints limited them to small-scale models, preventing evaluation of larger architectures and thus limiting their ability to study the impact of context at higher capacity.
• Open question: when to retrieve at all. The paper's design decisions — skipping Google Search API for standard-English datasets and choosing different extraction methods per dataset — raise the question of how to decide automatically when external retrieval helps and when it adds noise.

Target Audience

Researchers and practitioners working on sarcasm detection, pragmatic NLP, or retrieval-augmented prompting will get the most from this paper, particularly those interested in low-resource and non-English social media text. It is also useful for applied engineers building LLM pipelines who want a concrete comparison of external retrieval versus self-knowledge elicitation, and for readers tracking the PMP line of work and its limits. Some background in prompting terminology and basic information-retrieval concepts such as BM25 will help, though the pipeline itself is described step by step.

Authors’ abstract

Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 Task 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. These findings highlight the importance of context in enhancing LLMs performance in sarcasm detection task, particularly the involvement of culturally specific slang, references, or unknown terms to the LLMs. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. The experiment code is available at: https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base.