Four AI companies consent to joint opt-out against AI scraping in robots.txt

Four AI companies consent to joint opt-out against AI scraping in robots.txt

OpenAI, Google, Microsoft and Anthropic say they will adopt a common standard that lets publishers say no to AI scraping across all four companies at once – via a single line in robots.txt.

Four of the biggest players in artificial intelligence have agreed to a shared mechanism that lets web publishers opt out of AI scraping across the companies – instead of configuring preferences separately at each one. The report came on October 30, 2026, from The Verge (Emma Roth), citing earlier coverage in Search Engine Roundtable.

For publishers and websites, which for years have had to contend with a growing crowd of AI crawlers, this could simplify work that until now has required separate settings for each company. But the standard is a signal, not enforcement – and that is exactly what the main questions are about.

How the "policy token" works

The new signal is called a "policy token," and its location is a deliberate choice: it is to live in the website's robots.txt file. That file has long been the standard way to tell web crawlers what they may and may not fetch – whether for search indexing, archiving, or AI training.

The proposal was put forward by Kevin Kelley, technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF) – the international standards body that develops many of the internet's technical protocols.

The benefit for publishers is practical: with a shared token, a single preference can apply to several AI companies at once, instead of the publisher having to set up opt-outs individually with each crawler operator. In practice, that means a publisher can say "don't use my content for AI training" once, and have the signal interpreted by all companies that support the standard.

Different timelines across the four companies

Uptake of the signal is spread out in time, and the figures are the companies' own statements – not confirmed dates:

• Anthropic has already implemented support for the signal.
• OpenAI says it will implement it "in the coming months."
• Google and Microsoft plan support "next year."

That the proposal comes from Anthropic, and that the company is also the first to implement it, is worth noting: it is unusual for competitors in this industry to agree on a single shared mechanism for publisher control.

Microsoft's granular variant

Microsoft stands out with more fine-grained controls. The company will give publishers the ability to say no to two things separately: AI training and AI grounding. Grounding is the process of pointing an AI model toward verified information sources to reduce hallucinations – something distinct from the lengthy training of the model itself.

At the same time, Microsoft is adding a guarantee that could make the choice easier for publishers: websites will still appear in Bing's search results regardless of what preferences the publisher has set. This spares publishers the dilemma that has long accompanied robots.txt-based blocking – the risk of disappearing from search when refusing AI use.

The context: an industry in friction with publishers

The announcement does not come in a vacuum. Publishers have for some time been fighting a steady stream of AI crawlers that harvest content from their websites and feed it into AI models. Several initiatives have already attempted to give publishers power back:

• Really Simple Licensing (RSL) is a standard that lets publishers charge crawlers that visit their sites – a licensing and compensation model, not an opt-out.
• Cloudflare has introduced default blocking of AI crawlers, so that AI traffic is shut out unless the publisher actively allows it.

Against this backdrop, the new token signal is a third approach: voluntary opt-out via an existing, well-established technical channel. It does not resolve the question of payment the way RSL attempts to, and it does not block traffic the way Cloudflare does – but it lowers the barrier for publishers who want to signal preferences across the industry.

Open questions

The most important unanswered question is what happens if a crawler ignores the signal. The sources say nothing about technical enforcement, remedies, or sanctions against companies that fail to honor preferences. A signal in robots.txt is historically a courtesy convention, not a technical barrier – and whether the four companies will actually respect the token in practice remains to be seen.

Technical documentation also remains to be seen: the exact wording of the IETF draft, any blog posts from the companies, and reactions from the publishing industry are not yet public in the coverage. And the timelines – "next year" at Google and Microsoft, "in the coming months" at OpenAI – are claims from the companies themselves, without set dates.

There is also a principled question: whether a shared opt-out standard is enough at all for publishers who want compensation, not just refusal. There, RSL offers a different model – payment for crawling – and the two initiatives may turn out to complement or compete with each other in the time ahead.

For now, the safest thing one can say – and also the most concrete – is that four companies that usually compete hard have agreed on one shared way to let publishers say no. Whether the no is actually respected is the next chapter.