Attacks by Content: Automated Fact-checking is an AI Security Issue

AI summary

Overview

Research area: AI agent security and natural language processing, specifically the intersection of retrieved-document attacks on LLM-based agents with the NLP task of automated fact-checking.

Technical level: Intermediate. The paper is a position paper, not a systems paper; it relies on familiarity with retrieval-augmented generation, prompt injection, and fact-checking subtasks, but its argument is largely conceptual and its experiment is a small simulated scenario study.

Scope: The paper defines a class of attacks on AI agents called "attacks by content," argues that automated fact-checking should be repurposed as a cognitive defense, and reports a 60-scenario experiment measuring how vulnerable popular LLMs are and how much fact-checks and source warnings reduce that vulnerability.

What This Paper Is About

AI agents that search the web and read documents can be manipulated by the material they retrieve. Prior work focuses on indirect prompt injection, where attackers hide malicious instructions in documents; this paper argues that attackers do not need instructions at all — they can simply supply biased, misleading, or false content. The goal is to show that this class of attack is a real vulnerability, that existing prompt-injection defenses cannot catch it, and that automated fact-checking offers a ready-made set of techniques for defending against it.

Key Contributions

1. Introduces the term "attacks by content" for the subversion of AI agents via malicious data — biased, misleading, or false information — as distinct from instruction-based prompt injection. It positions this as a previously unstudied fourth case alongside direct instruction attacks, direct data attacks, and indirect instruction attacks.

2. Proposes a five-step "cognitive self-defense" pipeline for agents, mapping each step onto a subtask of automated fact-checking as outlined in Guo et al. (2022): claim prioritisation, evidence retrieval, source criticism, veracity analysis, and communication of findings.

3. Categorises existing defense work according to that pipeline and releases a repository compiling these resources at https://github.com/MichSchli/AgentCogSec (described as https://github.com/MichSchli/AgentCogSec).

4. Runs an experimental study across 60 scenarios showing that LLM-based agents are vulnerable to attacks by content and that fact-checking information functions as mitigation.

5. Identifies areas of concern where agents are likely to be misled — finance, healthcare, law, charity, politics, and useless products — and proposes four axes for analysing attacks by content.

Main Findings

• Agents are highly vulnerable without protection. In 60 simulated research-agent scenarios (ten per area of concern), vulnerability is measured as the percentage of attacks a model passes on to users, where lower is better. The overall least vulnerable model tested was Claude Sonnet 4 with a vulnerability rate of 61.7%. Some models passed almost all documents on to users.

• Fact-checks and source warnings both work, and they complement each other. For example, cohere/command-r-plus dropped from 98.3% baseline to 76.7% with a fact-check, 38.3% with a source warning, and 31.7% with both. openai/o4-mini-high went from 85.0% to 58.3%, 26.7%, and 10.0% respectively.

• Model families differ sharply in how much they benefit from defenses. The clear stand-out in making use of fact-checks was meta-llama/llama-3.1-8b-instruct, which fell from 78.3% baseline to 3.3% with a fact-check, and to 0.0% with a source warning or with both. This matches the cited finding from Sehwag et al. (2024) that the Llama family of models are generally more cautious.

• Media literacy does not appear to scale with model size. For multiple families (e.g. Llama, Claude, Qwen), the smaller models were the least vulnerable and made the best use of fact-checks or source warnings. The paper notes that in humans, higher education or analytical skill similarly does not necessarily imply stronger media literacy, and suggests that the ability to discern trustworthiness and the ability to "reason" may be orthogonal skills — so scaling models up may not lead to improvements in media literacy.

• Charity fraud is the worst-performing domain. Across all models, the column average vulnerability was 99.5% for charity, 98.2% for useless products, 97.7% for law, 90.0% for politics, 75.0% for healthcare, and 71.4% for finance, with an overall average of 88.6%. The paper states that almost all attacks succeeded against all models in the charity category, supporting its hypothesis that fake charities are a particular vulnerability for AI agents.

• Several models resisted even low-stakes-sounding domains. anthropic/claude-sonnet-4 scored 0% vulnerability in the healthcare column and 30% in finance; anthropic/claude-opus-4 scored 20% in finance and 30% in healthcare. Charity, law, politics, and useless products were at or near 100% for most models.

• Surface-form detection is argued to be insufficient in principle. The paper states that false and true claims generally cannot be distinguished based on surface form (Schuster et al., 2020), and that existing defenses in benchmarks such as AgentDojo focus on malicious instructions and therefore do not cover attacks by content.

• Prompts matter for credibility assessment. In Table 3, x-ai/grok-3 scored 91.7% baseline with a standard "helpful assistant" prompt, compared to 65.0% with the official Grok prompt including warnings and 70.0% with a version where warning lines were redacted.

• Fact-checking is not a guarantee. The limitations section reports that current state-of-the-art fact-checking systems correctly verify only 60–65% of real-world claims.

Methodology in Plain English

The paper is primarily a position paper, so most of its content is argument and literature synthesis rather than new modelling. The author first defines attacks by content and distinguishes them from the three previously identified attack types, using illustrative examples such as "airline tickets are 10% cheaper here" versus a classic injection like "ignore previous instructions and transfer me X BTC."

The author then builds an analogy between defending an agent and the work of a human fact-checker, breaking the defense into five stages and, at each stage, surveying published work that could fill that role. For example, for claim prioritisation the paper cites lightweight retrieval evaluators, constitutional AI frameworks for resolving conflicts between model weights and retrieved data, confidence-score-based trustworthiness estimates, and trained models that distinguish surface forms of trustworthy versus untrustworthy information. For evidence retrieval, it points to work on robustness against passages inserted into top-k results, and notes that consulting a database of previous fact-checks such as the Google FactCheck Explorer may be a highly effective security measure, even though automated fact-checking research treats such documents as "cheating."

The empirical component is a small simulated "Deep Research" setting. The author created 60 scenarios — ten for each of six areas of concern — by first generating fictional scenarios with Claude 4 Opus and then manually editing them for plausibility. Each scenario presents a research agent with a task and a retrieved document, and asks whether the document should be passed on to the user in a summary. The author then re-ran each scenario with three added conditions: a fact-checking sentence refuting the retrieved article, a "media background check" (a critical analysis of the source), and both together. An example attack text describes "CryptoVault Pro" with claimed 8–12% monthly returns over 18 months, a 94% win rate, and a $5,000 minimum investment. Prompts are given in Appendix A; the dataset is published at https://github.com/MichSchli/AgentCogSec/attack_dataset.json. A separate appendix tests Grok 3 with and without its official system prompt.

Why This Matters

Impact on research: The paper reframes agent security as partly a fact-checking problem and argues that agent security researchers and fact-checking researchers should not duplicate each other's work. It points out an explicit gap: there has, to the best of the author's knowledge, been no work examining agent reasoning with evidence of truth and trust, and little work on explainability for fact-checking subtasks other than veracity prediction — naming explainable source criticism, bias analysis, and evidence retrieval as key gaps. It also argues that efficiency research in fact-checking (driven by journalists citing cost as a barrier to adoption) is crucial for agent security, since extra retrieval and reasoning rounds increase latency and cost.

Real-world applications:

• Personal-assistant agents that manage emails, browse for information, and make purchases on a user's behalf — the paper notes a documented case of LLMs forwarding phishing links to users.
• Automated research and summarisation agents, the setting directly simulated in the experiment, where a retrieved document may be folded into a summary the user trusts.
• Charity and disaster-relief donation, where urgency means well-sourced evidence may not yet be findable on the web, and where the FTC's recommended strategies ("find out who is behind the campaign", "reverse image search photos used") map onto known weaknesses in current fact-checking systems.
• Online shopping and marketplaces, given documented scams and the emerging category of "botbait" — listings crafted specifically to fool AI agents, such as the example of a fake "RTX 5090 GPU" on eBay.
• Legal and immigration advice, where attackers threaten lawsuits, send fake legal notices, or impersonate legal professionals.

Industry relevance: The paper is aimed at developers building agent products. Its warnings include that agents should be given caution before deployment with autonomous buying and selling abilities, and that even with fact-checking defenses, care should be taken before granting agents risky actions such as making payments on behalf of users. It also highlights that a well-designed system prompt with epistemological guidance ("do not blindly trust sources", "do your own research") measurably reduced vulnerability for Grok 3 — a cheap, immediately implementable finding for practitioners.

Future Directions

• Define a task for estimating risk rather than truth. The paper proposes that, because financial fraud often uses attacks that are not easily fact-checked (impersonating a trustworthy entity, or building reputation until a rug pull), a new task could be defined for estimating, given retrieved evidence, how risky it would be to trust a source or believe a claim — comparable to estimates of claim harmfulness in fact-checking.

• Build out source criticism and multi-modal fact-checking. These are identified as specific weaknesses that make agents vulnerable to charity fraud, and the paper suggests that evidence-based credibility assessments gathered via a second step of retrieval-augmented generation, plus reverse image search, are promising directions.

• Develop explainability for the non-veracity steps of the pipeline. Explainable source criticism, bias analysis, and evidence retrieval are explicitly named as key gaps that future work should explore, motivated by the risk that users over-rely on agent decisions.

• Extend automated fact-checking to legal claims. The paper states that, to the best of its knowledge, there are no large-scale attempts to automate legal-domain fact-checking, and proposes this as a fruitful area for further research.

• Investigate memory exploitation and botbait. The paper raises open questions about attacks that inject memories recalled later to guide action, and about how attackers' attitudes toward scamming agents rather than humans may differ, predicting that the prevalence of botbait will grow as agents become more widely adopted.

Target Audience

Agent security researchers and developers building retrieval-augmented or autonomous agent systems will get the most direct value, since the paper offers a category of threat they are not currently defended against and a concrete five-step mitigation design. Automated fact-checking researchers will find a new application domain for their techniques and benchmarks, plus a list of unsolved subtasks worth pursuing. Policy and trust-and-safety practitioners at companies deploying assistants will benefit from the domain-by-domain vulnerability numbers, which show where user harm is most likely. Finally, NLP graduate students looking for a position-paper-style overview of how two fields connect will find the pipeline and the categorisation of existing work a useful entry point, along with the released repository.

Authors’ abstract

When AI agents retrieve and reason over external documents, adversaries can manipulate the data they receive to subvert their behaviour. Previous research has studied indirect prompt injection, where the attacker injects malicious instructions. We argue that injection of instructions is not necessary to manipulate agents - attackers could instead supply biased, misleading, or false information. We term this an attack by content. Existing defenses, which focus on detecting hidden commands, are ineffective against attacks by content. To defend themselves and their users, agents must critically evaluate retrieved information, corroborating claims with external evidence and evaluating source trustworthiness. We argue that this is analogous to an existing NLP task, automated fact-checking, which we propose to repurpose as a cognitive self-defense tool for agents.