CogniDir: Combating Cognitive Malicious Comments via Adaptive Distributional Learning for Robust Fake News Detection

AI summary

Overview

Research area: Adversarial machine learning for social media content safety, specifically comment-based fake news detection under LLM-generated attacks.

Technical level: Advanced. The paper assumes familiarity with adversarial training, information theory, and Dirichlet distributions, though the central idea can be understood without the mathematics.

Scope: The paper introduces CogniDir, a training framework that treats robustness against psychological manipulation attacks as a dynamic allocation problem over attack categories, rather than a fixed adversarial training setup.

What This Paper Is About

Fake news detectors increasingly rely on user comments as evidence, but attackers can now use large language models to inject comments designed to psychologically mislead a detector rather than simply adding noisy text. These attacks come in distinct flavors, such as distorting facts, confusing logic, or manipulating emotion, and a detector that is hardened against one flavor often remains vulnerable to another. CogniDir addresses this by detecting which attack type is currently causing the most trouble and automatically shifting more training attention toward it, producing a detector that is robust across all attack types instead of just the ones it happened to see most often.

Key Contributions

1. A cognitive-grounded attack synthesis pipeline. The authors define three mechanism-specific categories of malicious comments based on cognitive psychology (fact distortion, logical confusion, emotional manipulation) and generate mechanism-labeled examples using a two-step chain-of-thought prompt across three different LLMs, filling a gap left by public datasets that carry only news-veracity or stance labels.

2. An information-theoretic vulnerability score. The framework couples empirical accuracy with the model's probabilistic confidence into a single per-category score, expressed as a Bernoulli cross-entropy that decomposes into an entropy term plus a KL divergence term. This captures the intuition that a detector can be "accidentally right but deeply uncertain" under a given attack type.

3. Dirichlet-mean adaptive allocation. Vulnerability scores are mapped to a probability distribution over attack categories via Dirichlet concentration parameters, whose mean yields smooth, deterministic sampling proportions. This converts robustness training into a self-correcting curriculum.

4. State-of-the-art robustness results across three cross-lingual benchmarks. CogniDir improves F1 by up to 17.9% over the strongest baseline under attack, cuts average attack success rate by 45%, and narrows the inter-category robustness gap from 0.23 to 0.08.

Main Findings

• Malicious comments cripple existing detectors, including LLM-based ones. Qwen2.5-32B drops from 0.831 F1 to 0.282 on Weibo16 under attack. Traditional comment-aware detectors lose more than 18% F1 on average across attack types.

• Robustness improves substantially and consistently. CogniDir reaches F1 of 0.945, 0.947, and 0.812 on Weibo16, Weibo20, and RumourEval-19 respectively after robust training, versus the second-best baseline's post-training scores of 0.867, 0.803, and 0.709. Reported gains over the second-best method under attack range from 8.9% to 21.1%.

• Attack categories differ sharply in their effectiveness. Emotional manipulation is the strongest attack (average success rate 0.283), followed by logical confusion (0.241) and fact distortion (0.206). The authors attribute fact distortion's weaker effect to the ability of surrounding context to absorb surface-level lexical noise.

• Attack intensity matters. Increasing injected malicious comments from zero to three raises attack success rate from 0.145 to 0.287, roughly a doubling, as contradictory cues accumulate and destabilize the decision boundary.

• Adaptive training flattens the vulnerability gap. Mean attack success rate falls to 0.116 after robust training (a 45% reduction), and inter-attack variance shrinks from above 0.07 to below 0.03. Validation accuracy across epochs rises from 0.82 to 0.90 (fact distortion), 0.69 to 0.83 (logical confusion), and 0.59 to 0.82 (emotional manipulation), with the group gap collapsing from 0.23 to 0.08.

• All three components are necessary. Ablation shows removing the generation pipeline costs 21.4 macro-F1 points on average, replacing the vulnerability score with random scores costs 16.3 points, and substituting a softmax weighting for Dirichlet allocation costs 11.1 points.

• Attack direction is not biased. Direction-specific analysis on GenFEND shows meaningful success in both real-to-fake and fake-to-real flips, indicating general vulnerability rather than a single-label exploit.

Methodology in Plain English

The approach has three stages.

Stage one: manufacture the attacks you want to defend against. Because no public dataset labels comments by manipulation mechanism, the authors write a prompt that asks an LLM to first identify the key claims in a news article and decide on a misleading direction, then write a natural-sounding user comment that pushes the reader that way. Running this across three models of different sizes produces a pool of comments tagged by mechanism. Running a fixed prompt across multiple generators reduces the risk that the detector learns one model's stylistic fingerprint instead of the manipulation itself.

Stage two: train, measure where you are weak, and rebalance. Training begins with a uniform mix of the three attack categories. After each epoch, the detector is evaluated on category-specific validation sets. Two signals are recorded per category: how often it is correct, and how confident it is. These are combined into a single vulnerability score using an information-theoretic formulation, so that a category where the model is both inaccurate and overconfident scores as highly vulnerable. The scores then become parameters of a Dirichlet distribution, and the mean of that distribution is used as the sampling proportion for the next epoch. Vulnerable categories get more training exposure.

Stage three: evaluate under realistic mixtures. The final detector is tested on both category-specific sets, to diagnose per-attack failure modes, and on mixed sets that blend several attack types within the same instance, to approximate real comment streams.

The detector itself is deliberately conventional: a shared BERT encoder processes the article and each comment, a self-attention layer refines the representations, and a small classifier outputs the verdict. The novelty is entirely in how the training data is assembled and reweighted.

Why This Matters

Research impact. The paper reframes adversarial robustness in NLP from "add more adversarial examples" to "decide which adversarial examples to show next," and it grounds that decision in a measurable quantity rather than a heuristic. It also places cognitive psychology categories, rather than generic text-perturbation categories, at the center of the threat model, which is a meaningful shift for content-safety research. The finding that emotional manipulation is the hardest attack to defend against is a concrete result other researchers can build on.

Real-world applications.

• Platform trust-and-safety systems that flag misleading news posts while hostile users flood the comment thread with coordinated replies.
• Fact-checking and newsroom verification tools that need to remain reliable when a story's comment section is being deliberately weaponized.
• Adversarial red-teaming pipelines for content moderation models, using the generation procedure to probe where a deployed classifier is weakest.
• Election and public-health misinformation monitoring, where emotionally charged comment campaigns are common and detector failures have outsized consequences.

Industry relevance. The framework is a training-time technique that plugs into existing classifier architectures, so it does not require rebuilding detection infrastructure. For platforms whose moderation models are retrained periodically, the vulnerability score offers a monitoring signal: rising scores on a category indicate a shift in attacker behavior. The 45% reduction in attack success rate and the narrowing of the inter-category gap map directly onto operational goals, since production systems fail badly when robustness is uneven and attackers simply migrate to the weakest category.

Future Directions

1. Move beyond synthesized attacks. The authors acknowledge that LLM-generated mechanism-controlled comments cannot fully represent naturally occurring manipulation. Extending validation to organic comment threads with real adversarial intent is the most direct next step.

2. Test against unseen generators and prompts. The current study does not exhaustively evaluate attacks from models or prompting strategies held out from training. Out-of-distribution attack evaluation would show whether the adaptive allocation generalizes or merely overfits to the three categories it was designed around.

3. Expand the mechanism taxonomy. The framework is built on exactly three cognitive categories, chosen to match Dirichlet parameterization. Whether additional or finer-grained mechanisms (for example, appeals to authority, social proof, or manufactured consensus) yield the same benefits is untested.

4. Probe the allocation policy theoretically and operationally. The paper provides an entropy-regularized interpretation of the allocation in its appendix, but questions remain about convergence guarantees, sensitivity to the fixed offset constant, and whether the agent-style reallocation is stable under non-stationary attack distributions that change mid-training.

Target Audience

This paper is most useful to machine learning and NLP researchers working on adversarial robustness, fake news detection, or content-safety systems; to trust-and-safety and integrity engineers at social media platforms who need detectors that hold up under coordinated comment manipulation; to computational social scientists interested in how cognitive manipulation mechanisms translate into measurable model behavior; and to graduate students studying adaptive or curriculum-based training strategies, since the Dirichlet-mean allocation mechanism is transferable to other multi-source robustness problems beyond misinformation.

Authors’ abstract

The proliferation of Large Language Models (LLMs) has enabled a new class of psychologically grounded malicious comments, shifting fake news attacks from surface-level textual noise to deep cognitive and logical manipulation. This shift severely undermines existing detectors, which conventionally rely on static attack assumptions and fixed training distributions. To bridge this gap, we introduce CogniDir, an adaptive distributional learning framework that reformulates robust detection as a dynamic data mixture optimization problem for social media content safety. Grounded in cognitive psychology, we first formalize mechanism-specific cognitive adversarial paradigms to systematically expose deep-seated detector vulnerabilities. To address the vulnerability heterogeneity, CogniDir derives an information-theoretic score coupling empirical accuracy with probabilistic confidence, which is then mapped to adaptive sampling proportions through a Dirichlet-mean parameterization. This formulation enables smooth, feedback-driven reallocation of training exposure toward the most brittle attack mechanisms. Experimental results on three benchmarks demonstrate that CogniDir yields state-of-the-art robustness, improving F1 scores by up to 17.9% over competitive baselines under heterogeneous, AI-generated adversarial pressures.