SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations

AI summary

Overview

Research area: Natural Language Processing and computational social science, specifically context-aware text classification of social media conversations in French.

Technical level: Intermediate. The paper is readable by non-specialists, but the modeling section assumes familiarity with transformer encoders, fine-tuning, and LLM prompting strategies.

Scope: The paper introduces SPOT, an annotated French Facebook comment corpus and benchmark for detecting "stopping points" — ordinary critical interventions that pause or redirect online discussions — and compares fine-tuned encoder models against prompted instruction-tuned LLMs on a binary classification task.

What This Paper Is About

Research on online discourse has mostly studied explicit, goal-directed interventions such as counterspeech, evidence-based refutation, and social correction, while overlooking subtler everyday reactions like irony, doubt, or fragmentary criticism that nonetheless interrupt or redirect a conversation. The authors translate the sociological concept of a stopping point (Berriche, 2024) into a reproducible NLP task, arguing that whether a comment is a stopping point depends on its conversational function rather than its tone, polarity, or factual accuracy. Their goal is to build a large annotated French corpus with rich contextual metadata and to benchmark how well current models can detect these interventions.

Key Contributions

1. An operationalized annotation task. The paper converts the sociological notion of a stopping point into a binary classification task with detailed, validated annotation guidelines, including explicit decision rules for ambiguous cases such as irony, nested replies, and link-only responses.

2. The SPOT corpus. A dataset of 43,305 manually annotated French Facebook comments drawn from 1,061 posts and 253 shared URLs across 275 public French Facebook pages and groups, enriched with publication context: article (URL, title, description), post message, parent comment, domain name, and page or group name.

3. A benchmark of encoders versus LLMs. The authors compare a fine-tuned CamemBERT model across contextual variants (No Context, Context Concat, Context Embed) against instruction-tuned LLMs (Llama 3.2 3B Instruct, Mistral 7B Instruct-v0.2, Qwen2.5 7B Instruct, and GPT-4o-mini) under zero-shot, few-shot, and chain-of-thought prompting, with and without context, in both French and English prompts.

4. An error analysis of failure modes. All 360 misclassified test instances from the best model were manually categorized, providing empirical guidance on where context-aware encoders over-rely on surface lexical cues.

Main Findings

• Fine-tuned encoders beat prompted LLMs by more than 10 percentage points. CamemBERT with Context Embed reached an F1 of 78.09 ± 0.84, while the best LLM configuration (GPT-4o-mini few-shot without context) reached 62.94. LLM scores ranged from 39.23 (Qwen2.5-7B-Instruct zero-shot without context) up to 62.94, against a random baseline of 16.4.

• Contextual metadata improves encoder performance. Adding context raised encoder F1 from 74.67 ± 0.77 (No Context) to 78.09 ± 0.84 (Context Embed), the paper also describing this as an improvement from 0.75 to 0.78.

• Embedding context works better than concatenating it. Context Embed (78.09) outperformed Context Concat (76.96), which the authors attribute to concatenation diluting the comment's own linguistic features within a longer input string.

• Article text is the single most informative context element. In the ablation study, Article Concat yielded 78.03 ± 1.40, the largest gain over the 74.67 no-context baseline, followed by Post Concat (77.40 ± 1.51), Domain Concat (77.15 ± 0.65), and Page/Group Concat (76.15 ± 1.35). Parent Comment Concat (73.97 ± 1.09) performed below the no-context baseline.

• More prompting complexity did not help LLMs. Few-shot prompting without context consistently produced the highest F1 scores for both Qwen and GPT, outperforming few-shot with context and chain-of-thought with context variants.

• The annotated corpus is highly imbalanced. Stopping points constitute 4,306 of the 43,305 comments (9.9%). Of all comments, 28,457 (65.7%) are direct comments to the post and 14,848 (34.3%) are replies.

• Annotation reliability was substantial. On a 500-comment validation subset annotated by three raters, raw agreement was 0.9067 (95% CI [0.8829, 0.9306]), Fleiss' kappa was 0.8036 (95% CI [0.7494, 0.8578]), and Krippendorff's alpha was 0.8037 (95% CI [0.7496, 0.8579]), computed with 500 bootstrap samples.

• Errors cluster around implicit or contradictory cues. Of 203 false positives, 85.2% were complex cases, 77.3% involved comments with explicit critical markers, 30.0% involved reported speech, 16.3% irony, 11.8% a "reply to a stopping point" pattern, and 5.4% short or fragmentary comments. Of 157 false negatives, 79% were complex, 51.6% lacked explicit refutation markers, 17.2% were replies to stopping points, and 12.7% involved irony or humor.

• Replies are harder than top-level comments. Replies made up 45% of false negatives versus 34.3% of the overall dataset.

Methodology in Plain English

The authors started from data made available through the Social Science One consortium: the Facebook Privacy Protected Shared URLs Dataset, which covers roughly 38 million URLs publicly shared at least 100 times on Facebook between January 2017 and July 2019. They filtered for URLs reported as "fake" and shared on public French pages or groups, yielding 904 flagged URLs — a set the authors stress reflects user perception rather than verification by professional fact-checkers. Using the Minet web-mining tool, they collected 30,157 associated posts and 441,149 comments authored by 294,988 unique users.

To define what counts as a stopping point, one author conducted immersive online ethnography across roughly 50 Facebook pages and groups, reading full threads and keeping a field journal of recurring formats and borderline cases. This produced annotation guidelines that fix the unit of analysis as the individual comment or nested reply, enumerate typical targets of criticism (content credibility, source reliability, form, the poster, other users), and require a context-first procedure: open the shared URL, read the parent post, examine the immediately adjacent turns, and consult the page or group description when local norms matter.

Three annotators tested draft guidelines on ten deliberately challenging cases, reconciled disagreements, and refined the rules over two rounds. The full sample of 43,305 comments (10% of the collected corpus) was then annotated by a lead author, with a 500-comment validation subset independently coded by two additional experts to measure reliability.

For modeling, the authors fine-tuned CamemBERT in three configurations: a baseline with no context, Context Concat (context appended to the comment using the [SEP] token), and Context Embed (context encoded separately, then concatenated with the comment embedding and projected back to 768 dimensions). To stay within CamemBERT's 512-token window, each context element was truncated according to limits based on its mean length. They also evaluated Llama 3.2 3B Instruct, Mistral 7B Instruct-v0.2, Qwen2.5 7B Instruct, and GPT-4o-mini (approximately 8B parameters) under six prompt configurations — zero-shot and few-shot with and without context, plus chain-of-thought with context — testing French and English prompts and different label formulations (Yes/No, 1/0, Stop/No stop), with temperature set to 0. Fine-tuning LLMs was not attempted due to computational cost.

Why This Matters

Impact on research. The paper widens the object of study for computational social science beyond counterspeech and social correction to include ordinary, non-constructive critical interventions that standard taxonomies miss. It also provides a non-English, context-rich benchmark, addressing the English-centric and isolated-unit limitations the authors identify in prior datasets, and supplies evidence that supervised fine-tuning still outperforms prompting for fine-grained, context-dependent classification.

Real-world applications:

• Content moderation systems that surface subtle user pushback or hesitation rather than only explicit violations.
• Newsroom and fact-checking triage, where identifying threads in which users already question a claim can prioritize review.
• Platform design research on how communities self-negotiate norms and interpretive frames within comment threads.
• Computational sociology studying the spread and containment of contested content across public pages and groups.

Industry relevance. The finding that publication context — especially the article text, post message, domain name, and page or group name — measurably improves detection (from 74.67 to 78.09 F1 in the encoder experiments) is directly actionable for moderation pipelines. The gap between open-weight and proprietary LLMs under identical prompting conditions offers a practical comparison point for teams deciding between prompting and supervised adaptation for social media text.

Future Directions

1. Better context modeling. The authors propose moving beyond linear concatenation toward graph-based or hierarchical architectures that capture conversational structure, and integrating multimodal signals such as images and videos.

2. From binary to multi-label. Extending the current binary formulation to a multi-label task that automatically distinguishes different types of stopping points.

3. Separating comment types. Detecting top-level comments and nested replies in distinct models or pipelines, since replies were disproportionately represented among errors (45% of false negatives versus 34.3% of the dataset).

4. Cross-platform and cross-language expansion. Studying critical interventions on Reddit and YouTube and in other languages, while iteratively refining the annotation guide to improve label quality and model robustness.

Target Audience

Researchers and practitioners in computational social science, NLP, and content moderation who work on social media discourse, especially in non-English settings. It is particularly useful for those interested in context-aware text classification, annotation methodology and inter-rater reliability for subtle pragmatic phenomena, and rigorous benchmarking of fine-tuned encoders against instruction-tuned LLMs. Sociologists studying online criticism and platform governance will also find the operationalization of the stopping point concept relevant.

Authors’ abstract

We introduce SPOT (Stopping Points in Online Threads), the first annotated corpus translating the sociological concept of stopping point into a reproducible NLP task. Stopping points are ordinary critical interventions that pause or redirect online discussions through a range of forms (irony, subtle doubt or fragmentary arguments) that frameworks like counterspeech or social correction often overlook. We operationalize this concept as a binary classification task and provide reliable annotation guidelines. The corpus contains 43,305 manually annotated French Facebook comments linked to URLs flagged as false information by social media users, enriched with contextual metadata (article, post, parent comment, page or group, and source). We benchmark fine-tuned encoder models (CamemBERT) and instruction-tuned LLMs under various prompting strategies. Results show that fine-tuned encoders outperform prompted LLMs in F1 score by more than 10 percentage points, confirming the importance of supervised learning for emerging non-English social media tasks. Incorporating contextual metadata further improves encoder models F1 scores from 0.75 to 0.78. We release the anonymized dataset, along with the annotation guidelines and code in our code repository, to foster transparency and reproducible research.