AI training opt-out: one robots.txt rule for four AI companies
Google, Microsoft, OpenAI and Anthropic plan to honor one robots.txt rule that blocks AI training. This article gives the rule, the dates and the checks to do.
On 30 October 2026, Google, Microsoft, OpenAI and Anthropic said that they will support one shared robots.txt rule for AI training. With this rule, a website owner can say "do not train AI models on my content" to all participating companies in one step. Today, you must write a separate rule for each company.
If you have a website, this affects you. The rule is new, it is not active everywhere yet, and it does not do everything that its name suggests.
The rule is one group in robots.txt
The proposal is the IETF draft "Machine-Readable Training Preference Signal", published on 30 October 2026. Its author is Kevin Kelley, a standards lead at Anthropic. It uses a "policy token" in place of a crawler name. The example in the draft:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Disallow means that you do not want the content of that path used for AI training. Allow means that you agree to it. The normal robots.txt path rules apply, so you can block the full site and allow one folder.
The draft is an individual Internet-Draft. It is "not endorsed by the IETF" and has "no formal standing" yet. It uses the definition of AI training from the IETF AI Preferences (AIPREF) working group: using content to change the parameters of a generative AI model.
Each company starts at a different time
- Anthropic: Kelley wrote on the IETF mailing list that Anthropic "has already started implementing support" and that it will "take us some time to add support across our services."
- Google: the Search Central blog says Google aims to support the rule in 2027. If your robots.txt blocks the token and does not name Google-Extended, Google will treat it as a block of Google-Extended. The blog says that sites that allow Google-Extended can appear in Gemini Apps answers through grounding.
- Microsoft: the Bing blog says Microsoft will honor the new preferences from 1 February 2027, when it recrawls a page.
- OpenAI: according to The Verge, OpenAI says it will add support in the "coming months."
- Meta: The Verge reports that Andy Stone, Meta's head of communications, supports a common signal. He did not say that Meta will honor it.
The rule covers training, not search or AI answers
The draft covers AI training only. It does not cover search, grounding or AI agents. Grounding means that an AI tool reads your page at question time to write its answer.
- It is not a crawling block. A crawler can still fetch your pages. To stop a crawler, you still need a group with its own name, for example GPTBot.
- A wildcard group (
User-Agent: *) does not match the token. Your current wildcard rules do not express a training choice. - Microsoft says that training and grounding choices do not change your eligibility for normal Bing results. Google says in its crawler documentation that Google-Extended does not affect inclusion or ranking in Google Search. The same page says that Google-Extended controls training and also grounding in Gemini Apps.
- Microsoft adds a separate grounding control, but only per page, with a meta tag or HTTP header such as
<meta name="Bingbot" content="ai-grounding:disallow">. If you block grounding, your business can show less in Copilot and other AI answers. - The draft applies the rule that was in your robots.txt when the content was collected. So a new rule does not apply to content that a company collected before it.
For Bing there is one more date. Microsoft will stop reading the old NoArchive and NoCache tags as a training or grounding choice after 1 March 2027.
What is not known
- OpenAI gave no exact date. Meta did not say if it will honor the rule.
- The draft can change before it becomes a standard, or it can fail.
- No company said how it will show that it honors the rule.
This rule is about training. Agents that browse your site for a customer are a different case, which we described in our article on websites that block AI agents.
Checks for your website
- Open yourdomain.com/robots.txt and read it. Make a note of the AI crawler names it already blocks or allows.
- Decide on training first. If you do not want your content in AI training, add the
__usage_ai-traininggroup withDisallow: /. - Keep the crawler-specific groups that you already have. The new group does not replace them.
- Know the Google trade-off. When you block the new group, Google also stops grounding Gemini Apps answers on your site. To stay in those answers, you must allow Google-Extended, and that also allows Google training.
- Do not block search crawlers such as Googlebot or Bingbot. The training rule does not need it.
- If you use
NoArchiveorNoCacheto keep Bing from using your pages in AI answers, set the new preferences before 1 March 2027. - If your website builder or CDN writes robots.txt for you, check that it lets you add a custom group.