Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montage

AI summary

Overview

• Research area: AI safety and adversarial NLP, specifically belief manipulation in LLM-based multi-agent systems operating over public information channels.
• Technical level: Intermediate. Readers need some familiarity with LLM agents, multi-agent systems, and the idea of belief updating as probabilistic inference, but the core argument is conceptual rather than mathematically dense.
• Scope: The paper formalizes and empirically demonstrates a class of attack in which coordinated LLM agents manipulate other agents' beliefs using only individually truthful evidence fragments, without fabricating content or using hidden communication channels.

What This Paper Is About

Existing research on malicious collusion among AI agents assumes attackers need to hide something: covert channels, backdoors, steganography, or false documents. This paper argues that none of that is necessary. If attackers strategically select, order, and distribute pieces of genuinely true information across public channels, LLM agents will assemble those fragments into false causal narratives on their own, a process the authors call "lying with truths." The goal is to formalize this threat, build a working framework that automates it, and measure how broadly vulnerable current models are.

Key Contributions

1. Formalization of the Cognitive Collusion Attack. The paper defines a new threat model in which every individual evidence fragment satisfies a Local Truth constraint (consistent with ground truth in isolation), yet the assembled sequence violates a Global Lie condition (the victim's posterior belief in a fabricated hypothesis exceeds belief in the truth). This separates local factual validity from global epistemic deception for the first time in this context.

2. The Generative Montage framework. A three-role adversarial pipeline — Writer, Editor, Director — that constructs deceptive narratives out of truthful content. The Writer synthesizes coherent drafts, the Editor permutes and sequences fragments to maximize spurious causal inference, and the Director acts as a proxy victim, running adversarial debate loops that accept, reject, or request revisions until the output is both deceptive and factually intact. Coordinated Sybil accounts then publish the fragments in the designed order.

3. The CoPHEME dataset and large-scale simulation. A benchmark built from the PHEME rumor dataset, partitioned into a truthful "Evidence Pool" (true/non-rumor tweets) and high-impact "Target Fabrications" (false/unverified rumors with large historical cascades). This decoupling allows controlled study of manipulation without introducing any false evidence.

4. Systematic vulnerability characterization. Evaluations across 14 LLM families, six real-world events, and two downstream verification strategies, quantifying attack success, confidence, and cascade effects.

Main Findings

• Pervasive vulnerability across model families. Overall attack success rates reach 74.4% for proprietary models and 70.6% for open-weights models. The attack requires no white-box access, confirming it exploits general reasoning mechanisms rather than model-specific quirks.

• Victims adopt false beliefs with high confidence. Beyond raw attack success, agents frequently report the fabricated conclusion with confidence scores above 0.8, indicating epistemic overconfidence rather than hesitation or hedging.

• Stronger reasoning increases susceptibility, not resistance. Distilled reasoning models (the DS-R1 series) are more vulnerable than their base counterparts. Explicit chain-of-thought prompting raises attack success by 3.1% (Qwen2.5-7B) and 4.7% (DS-R1-Distill-Qwen-7B) on the Charlie Hebdo event. The authors attribute this to open-weights reasoning models lacking the safety guardrails of proprietary systems, turning enhanced inference into an attack surface.

• Deception cascades downstream. When victim outputs are aggregated by Majority Vote (analogous to Twitter's Community Notes) or audited by an AI Judge, deception rates exceed 60% across model families and events. Victims actively defend their false conclusions with confident rationales, becoming "implicit colluders" who spread contamination through trusted consensus rather than through the original fragments.

• Every framework component matters, and the multi-agent structure is essential. Removing adversarial debate costs 13.5% ASR, removing the Editor's sequencing costs 7.3%, and collapsing the pipeline into a single LLM cuts ASR by 50.2% (to 26.8%). This quantifies the value of adversarial specialization and collaborative optimization over a single attacker.

• Sequence length follows an inverted-U curve. Attacks fail with too few fragments (1–5, insufficient to trigger narrative overfitting) and also with too many (16–20, causing contradictions and cognitive overload). Effectiveness peaks at 11–15 posts, defining an "optimal manipulation zone."

• Event complexity modulates vulnerability. Incidents that demand rapid causal synthesis, such as the Sydney Siege and Ottawa shooting, produce the highest deception rates, while complex political narratives such as the Putin-missing event show lower and more variable rates.

Methodology in Plain English

The authors begin from an observation about how LLMs behave: like humans, they tend to over-interpret fragmented inputs and construct causal stories that the raw evidence does not actually state. The paper calls this "narrative overfitting" and treats it as an exploitable vulnerability rather than an internal flaw to be mitigated.

To exploit it, they build a simulated social media ecosystem with three layers. The first layer is the attacker group: a Writer that pulls only true tweets from a fact-checked pool and drafts a narrative leaning toward a target false conclusion, an Editor that slices the narrative into discrete fragments and searches for the ordering that maximizes implied causal links, and a Director that simulates the victim, running debate loops until the output is deceptive yet contains no false statement. The second layer is a set of Sybil accounts that post these fragments into a public feed in the designed temporal order, mixed with ordinary traffic.

The third layer is the victim: a neutral LLM analyst whose job is to read the feed and answer user questions. Because the fragments are individually true, filtering-based guardrails do not catch them; the victim simply assembles them into a false storyline and reports it confidently. The paper then measures how often this happens, how confident the victims are, and whether downstream fact-checking mechanisms such as majority voting or an LLM judge can catch the contamination. Notably, the experiments use only local-truth evidence, so no part of the pipeline ever generates a false statement.

Why This Matters

This work challenges a foundational assumption in AI safety: that misinformation must be generated to be dangerous. Content-filtering defenses, provenance labeling, and fact-checking pipelines all operate on individual claims. Cognitive collusion operates on the relationship between claims — their ordering, timing, and juxtaposition — so none of those defenses fire. The paper also inverts a common intuition that stronger reasoning makes models safer, showing that the same capacity used for careful causal inference can be co-opted for self-deception.

Real-world applications affected:

• Social media and news platforms. Recommendation and summarization agents that synthesize feeds are the direct attack surface; bot coordination via public posts is already technically trivial.
• Financial analysis and trading agents. Market-moving narratives can be assembled from individually accurate disclosures, earnings snippets, or regulatory filings without any fraudulent statement.
• Medical and public health decision support. Assembling real study results or case reports in adversarial order could steer clinical guidance agents toward incorrect recommendations.
• Automated fact-checking and content moderation. The paper shows these systems inherit contamination from upstream victims, meaning the moderators themselves become carriers.

Industry relevance extends to any organization deploying LLM agents that ingest open-source information streams — threat intelligence analysts, compliance monitors, research assistants — where the integrity of a conclusion depends on the integrity of the sequence, not just the sources.

Future Directions

1. Extend beyond text to multimodal evidence. The current study is text-only, but images, video, and cross-modal fragments could carry equivalent semantic priming effects, potentially harder to detect because provenance tracking across modalities is immature.

2. Move from simulation to live platform conditions. Real environments include algorithmic curation, diverse user populations, and organic counter-narratives that the controlled setup excludes. Ecological validation is a prerequisite for calibrating severity.

3. Develop and evaluate concrete defenses. The paper sketches promising directions — logit-level belief trajectory monitoring to detect abrupt posterior jumps, entropy analysis to flag evidence that disproportionately collapses uncertainty toward the fabricated hypothesis, cross-model belief divergence to distinguish organic consensus from coordinated alignment, and provenance auditing to trace inferential pathways — but does not test any of them empirically.

4. Build cognitive-level benchmarks and training-time robustness. Adversarial fine-tuning on edited sequences and machine unlearning of internalized false beliefs are proposed but unexplored directions for hardening agents against evidence curation.

Target Audience

AI safety and alignment researchers studying adversarial manipulation in multi-agent systems; NLP researchers working on agentic reasoning, misinformation, and belief dynamics; platform trust-and-safety engineers and policy analysts who need to understand failure modes that content filtering cannot address; and anyone deploying LLM agents that aggregate open-source information in high-stakes domains where a coherent but wrong conclusion carries real consequences.

Authors’ abstract

As large language models (LLMs) transition to autonomous agents synthesizing real-time information, their reasoning capabilities introduce an unexpected attack surface. This paper introduces a novel threat where colluding agents steer victim beliefs using only truthful evidence fragments distributed through public channels, without relying on covert communications, backdoors, or falsified documents. By exploiting LLMs' overthinking tendency, we formalize the first cognitive collusion attack and propose Generative Montage: a Writer-Editor-Director framework that constructs deceptive narratives through adversarial debate and coordinated posting of evidence fragments, causing victims to internalize and propagate fabricated conclusions. To study this risk, we develop CoPHEME, a dataset derived from real-world rumor events, and simulate attacks across diverse LLM families. Our results show pervasive vulnerability across 14 LLM families: attack success rates reach 74.4% for proprietary models and 70.6% for open-weights models. Counterintuitively, stronger reasoning capabilities increase susceptibility, with reasoning-specialized models showing higher attack success than base models or prompts. Furthermore, these false beliefs then cascade to downstream judges, achieving over 60% deception rates, highlighting a socio-technical vulnerability in how LLM-based agents interact with dynamic information environments. Our implementation and data are available at: https://github.com/CharlesJW222/Lying_with_Truth/tree/main.