OpenAI said on Friday it will honor a common signal in robots.txt that lets websites express their preferences about use of their content for foundational generative AI training, in a post on its site. Google separately announced plans to support a shared AI training control in robots.txt, in a Search Central blog post. Neither company says it is honoring the signal yet: OpenAI aims to implement it in the coming months, and Google aims to support it in 2027. Both said the effort draws on the AI Preferences working group at the Internet Engineering Task Force (IETF).
OpenAI called the effort voluntary. It said website operators previously had to identify individual crawlers and maintain separate rules for each provider. “We aim to implement the signal in the coming months, and will share another update with details about our implementation when we do so,” OpenAI said. It said the signal will not replace its existing robots.txt controls, which “will continue to apply and be respected.” OpenAI said it encourages other AI providers to honor the signal.
Google said website owners will be able to use a shared “policy token” in robots.txt to set broad preferences across participating AI providers. When a site uses the token to opt out of training and Google-Extended is not explicitly specified, Google said it “plans to interpret that as if the publisher had disallowed Google-Extended.” Google said other AI providers “can handle it similarly.”
Google said existing company-specific controls will keep working alongside the new signal. It said websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. “We aim to support this new control along with other AI providers in 2027,” Google said.
Google described its proposal as a draft that brings in the AI training definition developed at the IETF while discussions continue on the rest of the AIPREF vocabulary. It said the proposal is consistent with the EU Code of Practice for General Purpose AI model providers, which recognizes robots.txt as a primary method for publishers to express machine-readable preferences. Google said it will keep working with stakeholders to standardize the control through the IETF group.
Neither post names other companies adopting the signal. Search Engine Land reported that Microsoft announced separate AI training and AI grounding preferences for Bing, Copilot and Microsoft 365. It quoted Microsoft’s announcement as saying the company is working with other industry participants on a shared AI training preference in robots.txt, and listed February 1, 2027 as the date Microsoft begins honoring its new preferences as it crawls pages.
Related coverage of publisher content and AI: the UK CMA’s investigation into Meta’s commercial terms, and Meta’s delay of those terms after publishers raised scraping concerns.
Sources
- OpenAI: Making it easier for websites to express AI training preferences (October 30, 2026)
- Google Search Central Blog: Supporting a shared AI training control for websites (John Mueller, October 30, 2026)
- Search Engine Land: Microsoft Bing gives publishers more control over how AI uses their content (Barry Schwartz, October 30, 2026; search excerpt, page behind a bot check)
- AI Tech Daily: UK competition regulator opens investigation into Meta’s commercial terms for publishers and other businesses
- AI Tech Daily: Meta delays new business terms to Nov. 20 amid publisher pushback over AI scraping