The New robots.txt Rule to Opt Out of AI Training (Not Live Yet)

Today, keeping your site out of AI training means writing a separate robots.txt rule for every AI company, and sometimes for every crawler each company runs. On October 30, 2026, Google, Microsoft and OpenAI each said they'll honor one shared rule on top of their own controls. It's two lines long. None of them honor it yet, so for now keep your per-company rules and add the new one.

What happened

The rule comes from an IETF Internet-Draft called "Machine-Readable Training Preference Signal", dated October 30, 2026 and written by Kevin Kelley of Anthropic. It's an individual draft built on the definitions of the IETF's AI Preferences working group, so it isn't a final standard yet.

The same day:

• Google announced plans to support "a shared AI training control in robots.txt" and said it aims to support it "in 2027". Its crawler docs say early 2027.
• Microsoft said it will honor the new preferences starting February 1, 2027.
• OpenAI said it "will honor a common signal in robots.txt" and aims to implement it "in the coming months".

SiliconANGLE reported that Anthropic also announced plans to support it, and that Meta supports the idea but hasn't committed. I couldn't find an Anthropic announcement of its own, so treat those two as reported.

The rule

The draft uses a special user agent name, a "policy token", that crawlers don't need to send:

User-Agent: __usage_ai-training
Disallow: /

That says "don't use any of my pages for AI training." You can carve out folders the normal robots.txt way. The draft's own example keeps a press kit open:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

"AI training" here means using your content to change the weights of a generative AI model. Training models used only for non-generative tasks, like classification or ranking, isn't covered.

The catch

• Your old wildcard rule doesn't count. The draft says a User-agent: * group must not be read as matching the token. If there's no User-Agent: __usage_ai-training line in your file, you haven't expressed a training preference under this signal.
• It's about use, not access. The draft says the token "is not a directive for crawling behavior." Your crawler rules keep working the way they did, and Microsoft says the same about its settings.
• It's voluntary. OpenAI calls it a "voluntary effort", and existing company controls keep working alongside it.
• Google ties it to Google-Extended. If you use the token to opt out and don't mention Google-Extended, Google plans to treat that as a Google-Extended block. Google-Extended also controls grounding in Gemini Apps, and Google says sites that specifically allow Google-Extended are eligible to appear in Gemini Apps answers through grounding. So opting out of training this way can also take you out of Gemini Apps grounding. Allowing Google-Extended by name keeps you eligible for those answers, but it also allows Gemini training, since one token controls both.
• Bing has a deadline. Microsoft is replacing its old NoArchive and NoCache tags for training and grounding. If your site uses them, Microsoft says to do this before March 1, 2027: publish your training preference (in robots.txt or per page), publish page-level grounding tags, and record a temporary grounding preference in Bing Webmaster Tools (it opens December 15, 2026). After March 1, those old tags stop counting as a training or grounding preference.

Bing's grounding tags

Microsoft added a second, separate setting called AI Grounding, which covers using your page to help answer a question in Copilot, Bing's AI answers or Microsoft 365. It's set per page with a meta tag or a header:

<meta name="Bingbot" content="ai-grounding:disallow">
X-Robots-Tag: Bingbot: ai-grounding:disallow

There's also ai-grounding:disallowextended, which allows Microsoft's own products but not the grounding services Microsoft provides to third parties. Both settings default to Allow. Microsoft warns that blocking grounding reduces your visibility in AI answers, and that neither choice guarantees traffic or payment.

Try it

1. Open yoursite.com/robots.txt and look for any existing AI crawler rules (GPTBot, ClaudeBot, Google-Extended).
2. Decide what you want: no training, no grounding, both, or neither.
3. If you want no training, add the two-line __usage_ai-training group. Keep your existing per-company rules, since the new token isn't honored yet.
4. If you're on Bing's NoArchive or NoCache tags, before March 1, 2027: publish your training preference (robots.txt or per page), publish page-level grounding tags, and record a temporary grounding preference in Bing Webmaster Tools (opens December 15, 2026).

Here's how you get paid

My opinion: most local businesses have no idea which AI bots can read their site, and they won't learn a draft spec to find out. That's a small, clear job you can sell.

Sell an AI crawler setup:

• Training rule. Add the shared token, plus the per-company rules that work today. Explain the Google-Extended trade-off for Gemini Apps grounding.
• Bing grounding tags. Set them page by page the way the owner wants, with the visibility trade-off explained in plain words.
• Quarterly check. Re-read the file as new bots and rules appear, and confirm Google, Microsoft and OpenAI's support when it actually ships.

Be honest with clients about what it does: it's a preference that the big providers say they'll respect, and Microsoft describes its settings as covering future training.