3 min readNew DelhiNov 1, 2026 11:28 AM ISTThe New York Times publisher AG Sulzberger slammed AI companies (Photo/Canva)AI giants OpenAI, Google, Microsoft, and Anthropic are looking to adopt a new standard that would let publishers collectively signal that their content must not be used for AI training purposes.The opt-out standard known as ‘policy token’ is a shared signal proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). Publishers will be able to add the policy token to their website’s robots.txt file, which has long served as a protocol for which web crawlers can and cannot scrape a site, whether for search indexing, archiving, or AI training.However, the policy token is slightly different as it lets publishers indicate their AI training preferences across multiple AI companies, instead of doing it separately for each AI company.The move comes against the backdrop of a long-running tussle between publishers and tech platforms that has only intensified in the generative AI era. Publishers have increasingly challenged the use of their content to train AI systems without permission or compensation.The dispute has been compounded by a surge in AI crawlers scraping publishers’ sites to collect data for model training and other AI applications. While publishers can use existing standards such as robots.txt to signal their preferences, these controls could have limitations and must be configured for different AI providers. A common standard may reduce that burden.Also Read | Meta deletes provision after experts flag ‘broad’ scraping permissions for AI trainingThere have been several attempts to combat unauthorised scraping in the past. Some web infrastructure companies like Cloudflare block AI crawlers by default for their customers. Other organisations have come up with standards such as Really Simple Licensing (RSL), which gives publishers the ability to charge crawlers that visit their sites.The proposed policy token is different in that it has received support from most major AI model developers. OpenAI, Google, Microsoft, and Anthropic have said that they will restrict their AI crawlers from accessing a site if they encounter the shared token, according to a report by The Verge.Story continues below this adAnthropic said it has already adopted the signal. OpenAI has said it will implement it in the “coming months.” Google and Microsoft plan to support it next year. A potential holdout is Meta. While the Facebook and Instagram-parent has signaled support for the new setting, the company has not said when it will honour it.“We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry,” Meta communications head Andy Stone was quoted as saying.Previously, Microsoft announced granular controls to give publishers the ability to opt out of AI training or AI grounding – the process of focusing an AI model on a verified source of information to reduce hallucinations. Websites will still appear in Bing’s search results regardless of their preferences, the Windows-maker had said.Google has also implemented similar controls that allow websites to control whether their content appears in AI search features or is used for AI training, without impacting visibility in search results.© IE Online Media Services Pvt LtdAdvertisementLoading Recommendations...AdvertisementAdvertisement