Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

AI summary

Overview

Research area: Web measurement and AI data governance — specifically how the Robots Exclusion Protocol (robots.txt) is used to manage access by AI crawlers.

Technical level: Intermediate. The paper's concepts (robots.txt directives, HTTP user-agent headers, web crawling) are accessible to non-specialists, while the multi-vantage-point crawling methodology and longitudinal Internet Archive analysis require some familiarity with web measurement research.

Scope: A first-of-its-kind comparison of how 3,369 reputable news websites and 710 misinformation websites configure robots.txt and active blocking against 63 AI user agents, including a longitudinal analysis across six Internet Archive snapshots from September 2023 to May 2025.

What This Paper Is About

Large language models depend on web crawling for training data and up-to-date answers, and crawlers are expected to obey robots.txt files. The authors ask whether reputable news sites and misinformation sites differ in how they configure those files and in how they actively block AI crawlers. The goal is to determine whether content accessibility for AI systems is becoming asymmetric in ways that could shape what LLMs learn from.

Key Contributions

1. The first longitudinal analysis of robots.txt directives targeting AI user agents, comparing reputable news websites against misinformation websites classified via Media Bias/Fact Check (MBFC).
2. A curated list of 63 unique AI user agents, compiled from Dark Visitors, Cloudflare's Radar platform, a prior crawler-behavior study by Liu et al., and the ai.robots.txt repository.
3. A multi-vantage-point measurement of robots.txt collection using AWS instances across seven geographic regions (Germany, Sweden, the United States, Brazil, Africa, Asia, and Australia), with each site crawled three times and the most permissive valid result retained.
4. A measurement of active blocking — refusal to serve content when HTTP requests carry AI crawler User-agent strings (ClaudeBot and Anthropic-AI) — and its correlation with declared robots.txt rules.

Main Findings

• Reputable sites restrict far more often: Among websites with a robots.txt file, 60.0% of reputable news sites include a DisallowAll directive for at least one AI agent, compared to just 9.1% of misinformation sites.
• Broader agent lists: 61.3% of reputable news websites explicitly mention at least one AI agent, versus 12.3% of misinformation sites. On average, reputable news websites reference 15.5 distinct AI agents (± 12.6), while misinformation websites reference 0.77 (± 3.2).
• Partially allowed agents are rare: Partial Disallow for at least one AI agent stands at 0.7% for reputable news and 1.4% for misinformation sites. When an AI agent is defined at all, 98.4% of agents on reputable news sites are subsequently fully restricted via DisallowAll, compared with 72.7% on misinformation sites.
• Popularity alone does not explain the gap: Both reputable and misinformation websites appear in high ranks of the Tranco list, including 15 misinformation websites among the Top 10K most visited domains. Of those 15, one was inaccessible; among the remaining 14, 10 out of 14 (71.4%) impose no limitations on AI crawling access.
• Specific agents targeted: Over 50% of reputable news websites disallow GPTBot, and around 40–50% disallow CCBot, ChatGPT-User, ClaudeBot, and Google-Extended. Misinformation websites keep DisallowAll rates below 5% across all cases. Roughly 40% of reputable news websites disallow no AI agents, around 25% disallow more than 10, and the three most restrictive — arkansasonline.com, timesfreepress.com, and nwaonline.com — each disallow 54 distinct AI user agents. Over 90% of misinformation websites disallow two or fewer agents, and more than 80% disallow none.
• Both categories use active blocking: 516 reputable news websites (18.0%) actively block Anthropic-AI versus 58 misinformation websites (10.2%); 711 reputable websites (24.8%) block ClaudeBot versus 147 misinformation websites (25.6%). 486 reputable websites (16.9%) block both, compared with 56 misinformation websites (9.8%).
• Declared rules align with behavior only for reputable sites: 230 reputable news websites have DisallowAll for both agents — 47.3% of those that actively block both, and 8% of all reputable news websites. Only 1 misinformation website that actively blocked both also had a robots.txt rule disallowing both.
• The gap widened over time: The share of reputable sites disallowing at least one AI agent rose from 23% in September 2023 to 50% by January 2024, reaching nearly 60% by May 2025. Misinformation sites moved only from 4.6% to 9.2% over the same period.
• Rule changes track AI news events: 27 reputable news sites and one misinformation site changed from explicitly allowing an AI agent to DisallowAll. 26 of these moved from partial access to DisallowAll for PerplexityBot, all in 07/2024, a month after the crawler made headlines for disobeying robots.txt directives.
• Growing exclusion lists on reputable sites: By early 2025 the median number of disallowed agents on reputable sites exceeds 25 and the upper quartile reaches nearly 50. For misinformation sites the median remains at or near zero, with the entire distribution concentrated below five disallowed agents.
• Different top-blocked agents: In the September 2023 snapshot, GPTBot was already by far the most disallowed AI agent on reputable news sites (16.28% versus 2.19% on misinformation). For almost every snapshot, the most restricted AI agent on misinformation websites was Huawei's PetalBot. PerplexityBot entered the reputable top 10 in 09/2024 and its block percentage more than doubled.

Methodology in Plain English

The researchers built two lists of websites using Media Bias/Fact Check classifications: 710 misinformation websites (labeled Questionable Source or Conspiracy-Pseudoscience with Low or Very Low factual reporting) and 3,369 reputable news websites (labeled Pro-Science, Least Biased, Left-Center, or Right-Center with High or Very High factual reporting). They then fetched each domain's robots.txt file and homepage from seven cloud vantage points, crawling each site three times and keeping the most permissive valid result. A robots.txt file counted as valid only if it returned an HTTP 2XX response with parseable plain text; HTML responses were excluded, and up to three same-domain redirections were followed. They also manually compiled 63 AI user agents from four sources. For the historical view, they pulled six Internet Archive snapshots at four-month intervals between 1 September 2023 and 1 May 2025, using only archive data for consistency. To test active blocking, they sent sequential requests from a single location, varying only the User-agent value — first a custom control agent, then ClaudeBot, then Anthropic-AI — and reran the control crawl at the end to filter out websites whose behavior changed for unrelated reasons.

Why This Matters

Impact on research: The paper documents a measurable asymmetry in what AI crawlers can reach. The authors caution that they do not evaluate training datasets, but note that continued accessibility of misinformation sites may increase the likelihood their content is collected and reused as reputable sites opt out. They also flag the possibility of a self-reinforcing loop in which permissive misinformation sites — some of which may already be AI-generated themselves — are repeatedly ingested into future training corpora.

Real-world applications:

• Informing crawler operators and LLM developers about which source categories remain accessible and how that may skew training data.
• Helping policymakers assess whether voluntary mechanisms like robots.txt are sufficient for data governance, in a context where the EU AI Act went into effect on 08/2024.
• Supporting news publishers in benchmarking their own robots.txt and active blocking configurations against sector norms.
• Guiding platform and CDN operators who provide active blocking features based on declared User-agent headers.

Industry relevance: The findings matter for search engines, AI labs, CDNs, and content publishers. The authors note that active blocking cannot fully replace robots.txt because it is an all-or-nothing mechanism that can block other legitimate activities such as web search indexing by the same company — so reputable news sites benefit from combining directive-based and active blocking strategies. The paper also cites a 2025 NewsGuard report finding that 28% of responses from 11 leading LLMs contained false information when prompted with known misinformation claims, and a 2023 analysis of Google's C4 dataset that identified content tied to white supremacy, propaganda, and conspiracy theories.

Future Directions

• Determine whether the accessibility gap actually translates into different proportions of misinformation in AI training corpora, since the paper measures only declared policies and blocking behavior, not training data composition.
• Investigate why misinformation sites rarely adopt AI restrictions — whether it stems from limited awareness, limited technical capacity, or unwillingness — a question the authors raise but do not resolve.
• Track whether the trend of reputable sites expanding exclusions continues, and whether the median number of disallowed agents keeps rising beyond 25 as observed by early 2025.
• Study the self-reinforcing loop hypothesis, where AI-generated misinformation on permissive sites is redistributed, re-crawled, and re-emitted in new AI-generated content.
• Examine the interaction between robots.txt directives and active blocking as AI crawler compliance and non-compliance patterns evolve.

Target Audience

This paper is most useful to AI researchers and practitioners working on training data provenance and data ethics; web measurement and security researchers studying crawler behavior and access control; policymakers and regulators evaluating voluntary versus enforceable content controls; and news media organizations, CDN providers, and crawler operators who set or respond to robots.txt policy.

Authors’ abstract

Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable news websites and misinformation sites differ in how they configure these files, particularly in relation to AI crawlers. Analyzing a curated dataset, we find a stark contrast: 60.0% of reputable sites disallow at least one AI crawler, compared to just 9.1% of misinformation sites in their robots.txt files. Reputable sites forbid an average of 15.5 AI user agents, while misinformation sites prohibit fewer than one. We then measure active blocking behavior, where websites refuse to return content when HTTP requests include AI crawler user agents, and reveal that both categories of websites utilize it. Notably, the behavior of reputable news websites in this regard aligns more closely with their declared robots.txt directive than that of misinformation websites. Finally, our longitudinal analysis reveals that this gap has widened over time, with AI-blocking by reputable sites rising from 23% in September 2023 to nearly 60% by May 2025. Our findings highlight a growing asymmetry in content accessibility, as reflected in robots.txt directives, that may shape the training data available to LLMs, raising essential questions for web transparency, data ethics, and the future of AI training practices.