Sarcasm Detection on Reddit Using Classical Machine Learning and Feature Engineering

AI summary

Overview

Research area: Natural Language Processing — sarcasm detection on social media text, specifically Reddit comments.

Technical level: Beginner-Friendly. The paper uses standard, widely taught machine learning tools (TF-IDF vectorization and four classical classifiers) and includes the complete Python implementation in an appendix.

Scope: The paper evaluates whether context-free, hand-engineered text features combined with classical classifiers can distinguish sarcastic from non-sarcastic Reddit replies in a 100,000-comment subsample of SARC 2.0.

What This Paper Is About

Sarcasm is easy for humans to recognise but hard for machines, because the intended meaning of a remark often contradicts its literal wording — "Great job." can be praise or criticism depending on information the reader has but the model does not. This paper asks how far simple, transparent, older machine learning methods can get when they are given only the reply text itself, with the surrounding conversation deliberately removed. The goal is not to beat neural models, but to establish a clean, reproducible baseline showing what surface-level textual cues alone can achieve.

Key Contributions

1. A controlled context-free baseline for sarcasm detection. The author intentionally discards parent comments and all conversational metadata from SARC 2.0, keeping only the binary sarcasm label and the reply text, so that the resulting baseline isolates what wording and style alone can contribute.

2. A custom combined feature representation (TextFeatures). A single transformer joins word-level TF-IDF (unigrams and bigrams), character-level TF-IDF (3- to 5-character sequences), and five numeric stylistic measures into one sparse matrix that every classifier receives identically.

3. A like-for-like comparison of four classical classifiers. Logistic regression, a linear SVM, multinomial Naive Bayes and a random forest are each wrapped in a scikit-learn Pipeline, so feature extraction is applied identically and differences between models are not confounded by preprocessing differences.

4. Full reproducibility. The complete Python code for data loading, feature extraction, training and plot generation is included in the appendix, and a fixed random seed is used for both subsampling and the train–test split.

Main Findings

• Naive Bayes and logistic regression perform almost identically and best. Both reach an accuracy of 0.565 (Naive Bayes) and 0.564 (logistic regression), identical precision of 0.566 and 0.564, identical recall of 0.574, and identical F1 of 0.569 for the sarcastic class.

• The linear SVM is the weakest of the four. It records accuracy 0.541, precision 0.542, recall 0.534 and F1 0.538 — slightly lower than the other three models on every metric.

• The random forest shows no advantage over the linear models. It achieves accuracy 0.558, precision 0.558, recall 0.568 and F1 0.563, sitting between the SVM and the top two models but not surpassing them, despite offering a non-linear alternative.

• Discrimination above chance is limited. The ROC curve for the Naive Bayes classifier sits noticeably above the diagonal random-baseline line, with an area under the curve of about 0.59, indicating substantial overlap between the two classes in the feature space.

• Errors run in both directions. The Naive Bayes confusion matrix shows many sarcastic comments correctly recognised but many misclassified as non-sarcastic, and also some non-sarcastic replies predicted as sarcastic — a pattern the author attributes to the inherent ambiguity of the task.

• Different feature types contribute different signals. Word and character TF-IDF capture lexical tendencies such as elongated words, creative spellings and punctuation patterns, while the numeric measures quantify stylistic exaggeration. The author suggests Naive Bayes' probabilistic structure suits sparse TF-IDF data well, which may explain its top performance.

• The author is explicit that these results do not rival contextual or neural models, and that the modest scores reflect the deliberate removal of context rather than a failure of the approach.

Methodology in Plain English

The author starts from the balanced subset of the Self-Annotated Reddit Corpus (SARC 2.0), which contains replies, sarcasm labels, user and conversation metadata, and parent comments. Only the label and the reply text are kept. A subsample of 100,000 entries is drawn with a fixed random seed (the code uses random_state = 42), and replies that are empty or consist only of whitespace are removed.

The cleaned data is split 80/20 into train and test sets, stratified so the balance between sarcastic and non-sarcastic comments stays the same in both. The test set is held aside until all training is finished.

Each reply is converted into a single feature vector by a custom transformer. Three things are combined: (1) word-level TF-IDF over unigrams and bigrams, capped at 20,000 features, with English stop words removed and sublinear term frequency; (2) character-level TF-IDF over 3- to 5-character sequences, capped at 10,000 features, which picks up stylistic quirks rather than whole words; and (3) five numeric style measures — character length, word count, exclamation marks per word, question marks per word, and the ratio of uppercase letters. These are stacked into one sparse matrix.

Four classifiers are trained on that same matrix: logistic regression (maximum 500 iterations), a linear support vector machine (maximum 5,000 iterations), multinomial Naive Bayes, and a random forest of 150 trees. Because accuracy alone hides how well sarcastic comments specifically are caught, the author reports accuracy, precision, recall and F1 for the sarcastic class, plus a confusion matrix and an ROC curve for the Naive Bayes model.

Why This Matters

Impact on research. As deep learning architectures grow more complex, this paper argues there is value in documenting exactly how much a simple, interpretable pipeline can achieve under deliberately constrained conditions. It provides a reference point against which context-aware or neural sarcasm detectors can be measured, making explicit what context and richer representations actually add.

Real-world applications:

• Content moderation and community management — flagging potentially sarcastic comments so that moderation and sentiment tools do not misread tone.
• Social media and brand monitoring — interpreting public reaction to a product or announcement, where sarcastic praise can be mistaken for genuine approval.
• Customer feedback and review analysis — detecting ironic complaints or ironic compliments in short user-generated text.
• Teaching and benchmark construction — serving as a lightweight, fully reproducible exercise in feature engineering and model comparison.

Industry relevance. The approach is cheap to train, easy to inspect and diagnose, and runs on commodity hardware. For teams that need an interpretable first-pass filter or a fast baseline before investing in contextual or neural systems, this paper shows the level of performance such a pipeline can realistically deliver — roughly F1 of 0.57 for sarcasm detection — and where it will fall short.

Future Directions

• Reintroducing conversational context. The most obvious next step is to use the parent comments that this study deliberately discarded, following prior work showing that context can turn an ambiguous reply into an unmistakably sarcastic one.
• Pre-trained neural embeddings. The author suggests exploring pre-trained neural representations to test how much of sarcasm is expressed through style alone versus deeper meaning.
• Larger and more diverse datasets. Extending beyond the 100,000-comment subsample and the single Reddit corpus would test whether the observed patterns generalise.
• Multimodal signals. Related work discussed in the paper integrates images, audio or user profiles; combining such sources with text may reveal sarcastic intent that plain text cannot.

Target Audience

Researchers and students who want a transparent, reproducible baseline for sarcasm detection before moving on to context-aware or neural methods. It is also useful for practitioners building lightweight text classifiers who need to understand what hand-engineered TF-IDF and stylistic features can and cannot deliver, and for anyone teaching feature engineering and classical classifier comparison, since the full code is provided.

Authors’ abstract

Sarcasm is common in online discussions, yet difficult for machines to identify because the intended meaning often contradicts the literal wording. In this work, I study sarcasm detection using only classical machine learning methods and explicit feature engineering, without relying on neural networks or context from parent comments. Using a 100,000-comment subsample of the Self-Annotated Reddit Corpus (SARC 2.0), I combine word-level and character-level TF-IDF features with simple stylistic indicators. Four models are evaluated: logistic regression, a linear SVM, multinomial Naive Bayes, and a random forest. Naive Bayes and logistic regression perform the strongest, achieving F1-scores around 0.57 for sarcastic comments. Although the lack of conversational context limits performance, the results offer a clear and reproducible baseline for sarcasm detection using lightweight and interpretable methods.