The Pulse
OpenAI Backs a Shared Robots.txt Signal for AI Training
OpenAI says it will honor a shared robots.txt signal that lets websites state AI-training preferences for multiple providers. The company plans to implement the voluntary mechanism in the coming months, while keeping its existing crawler co
AI.info Team · Written with AI · fact-checked ·
OpenAI said on October 30, 2026, that it will honor a shared signal in robots.txt, giving website operators a way to state AI-training preferences for multiple participating providers through one set of rules. The company says it aims to implement the signal in the coming months and will publish implementation details later. For now, the announcement is a commitment to adopt the mechanism, not a report that it is already active.
“This voluntary effort makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.”
OpenAI
A shared rule instead of provider-by-provider entries
Website operators have typically had to identify individual crawlers and maintain separate robots.txt rules for each provider. The proposed signal separates a preference about how content may be used from rules that govern a particular crawler. That distinction could reduce repeated configuration, while leaving sites free to keep their existing crawler-specific instructions.
The Internet-Draft linked from OpenAI’s announcement describes a cross-provider policy token, __usage_ai-training. A site could use it in a robots.txt group to express a preference against training across all paths, then allow an exception for a specific directory, such as a press kit. The draft uses the existing robots.txt group and path rules; it adds no directive names or new syntax.
The signal does not control crawling by itself
The draft draws a line between content-use preferences and traffic management. A training preference is not a crawler directive, though a provider may use it to inform whether it retrieves content. Website owners that want to restrict crawling can still set rules for a provider’s own crawler token.
OpenAI says its existing robots.txt controls will continue to apply. Its crawler documentation, for example, says disallowing GPTBot indicates that a site’s content should not be used to train OpenAI’s generative AI foundation models. The company also describes OAI-SearchBot as a separate crawler used to surface websites in ChatGPT search, so the existing controls let operators distinguish training from search visibility.
An IETF draft, not an adopted standard
The linked proposal, “Machine-Readable Training Preference Signal,” is an individual Internet-Draft authored by Kevin Kelley, whose document lists Anthropic as his affiliation. The IETF Datatracker says the draft is not endorsed by the IETF and has no formal standing in its standards process. A separate working-group draft defines related preference categories, including AI training, AI use and search, but also says expressing a preference does not ensure recipients will follow it.
That limitation makes provider participation and implementation details consequential. OpenAI is encouraging other AI providers to honor the signal, but its announcement does not establish that every provider will adopt it or explain how the company will handle every site-specific preference. Its next stated step is to share implementation details after work on its own support progresses; until then, website operators still have the crawler-specific controls OpenAI already documents.