OpenAI, Google, Microsoft and Anthropic back shared robots.txt opt-out for AI training

OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that lets publishers opt out of AI scraping. The control, a shared "policy token," allows publishers to apply their AI training preferences across multiple AI companies instead of setting them one by one. The Verge reported the news, citing earlier coverage by Search Engine Roundtable.

The signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will sit in a website's robots.txt file, which has long indicated which crawlers may scrape a site, whether for search indexing, archiving or AI training. Until now, sites have had to state their AI training preferences separately to each company.

All four companies say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Microsoft will go further, offering publishers the choice to opt out of AI training or AI grounding, the process of pointing a model to a verified source of information to reduce hallucinations. Microsoft notes that sites will still appear in Bing's search results regardless of their preferences.

The change addresses a practical burden for publishers, who face a continuing influx of AI crawlers feeding content into AI models. A single signal that several of the largest AI developers respect reduces the work of maintaining separate rules for each crawler.

It also sits alongside other approaches. The Really Simple Licensing (RSL) standard gives publishers a way to charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default. The shared token is a preference signal rather than a payment mechanism, and, like robots.txt generally, it depends on crawler operators choosing to comply. Microsoft's split between training and grounding hints that the definition of what counts as an opt-out may not be uniform across companies.

Timing varies. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the "coming months." Until then, publishers will likely still need company-specific directives for some crawlers.

Also worth watching is how the proposal moves through the IETF process and whether other AI companies beyond the four named sign on.