MeanCEO: Tech Startups and Startup Ideas

robots.txt AI Training Opt-Out: Google, Microsoft, OpenAI and Anthropic Back One Shared Line. What Founders Should Add Now, and What It Won't Stop

TL;DR: What is the new robots.txt AI training opt-out, and should founders use it?

On 30 October 2026, Anthropic's Kevin Kelley published an IETF draft that adds one shared robots.txt token, __usage_ai-training. A group like "User-Agent: __usage_ai-training" plus "Disallow: /" tells AI companies you don't want your content used to train generative models. Anthropic says it has started implementing it, Microsoft will honour it from 1 February 2027, Google aims for 2027 and OpenAI for "the coming months". It doesn't block crawling or Google Search, and it can't pull back content already collected. If you sell content, add it now. Just check what it means for Gemini first.
Until last week, saying "don't train on my site" meant keeping a list of AI crawlers, one rule per company, and hoping you didn't miss the new one. Most founders I know never got round to it. Their robots.txt is whatever their website builder put there.
On 30 October, four of the biggest AI companies agreed on one shared line instead. Here's what the new token does, who honours it and when, the one catch for Google's Gemini, and what this means if you run a small content or SaaS business in Europe.

What happened

On Friday 30 October 2026, Kevin Kelley of Anthropic published an Internet-Draft called "Machine-Readable Training Preference Signal". It defines a single policy token, __usage_ai-training, that you put in the User-Agent line of a normal robots.txt group. The draft's own example looks like this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That says: no AI training on any page, except pages under /press-kit/. There are no new commands and no new syntax. Parsers that don't know the token simply skip the group, like any other unknown bot. "AI training" here has a narrow meaning, taken from the IETF working group's vocabulary: "Using an asset to modify the learned parameters of a generative AI model."
Status: Proposed. This is an individual draft, not an IETF standard yet, and the IETF itself says drafts should only be cited as "work in progress". What makes it news is who signed up on the same day:
  • Anthropic. In his post to the IETF mailing list, Kelley wrote: "Anthropic has already started implementing support for this draft." He added that it "complements our existing crawler controls rather than replacing them" and that it will take time to add support across Anthropic's services.
  • Google. In a Search Central blog post, John Mueller wrote that when a site opts out with the token and doesn't name Google-Extended, Google plans to treat it as if the site had disallowed Google-Extended. "We aim to support this new control along with other AI providers in 2027."
  • Microsoft. Bing says it will honour the shared training preference from 1 February 2027, as it crawls, and is adding its own page-level controls for training and grounding.
  • OpenAI. OpenAI says it will honour the signal: "We aim to implement the signal in the coming months." Its existing robots.txt controls stay.
CompanyWhat it saidWhenIts own controls
AnthropicHas started implementing supportRolling out, no date givenKept; the token complements them
GoogleReads a token opt-out as a Google-Extended disallow, unless you name Google-Extended"In 2027"Google-Extended keeps working
Microsoft (Bing)Honours the token in robots.txt, plus page-level training and grounding tagsFrom 1 February 2027NoArchive and NoCache stop counting on 1 March 2027
OpenAIWill honour the shared signal"In the coming months"GPTBot and other controls stay
Sources: company announcements and Kevin Kelley's IETF mailing-list post, all dated 30 October 2026, read 2 November 2026.

What the token does, and what it won't do

Read the draft closely and three limits jump out. First, "This policy token is not a directive for crawling behavior." Bots can still fetch your pages; you're only stating how the content may be used. If you want to stop crawling, you still need rules for each crawler's own name.
Second, a catch-all rule doesn't count. The draft says groups for User-Agent: * "MUST NOT be interpreted as matching the policy token." So a site that blocks everything with a wildcard has still expressed no training preference. Kelley put it bluntly: a robots.txt file without the token, "as is true of every one published today, expresses no preference at all."
Third, it doesn't reach back in time. The preferences that count are the ones in your robots.txt when the page was fetched. Anything already collected stays collected. The draft also says "Private agreements MAY override preferences", so a licensing deal beats your robots.txt.

The Gemini catch: training and grounding are tied at Google

This is the part most coverage skips. At Google, the token maps to Google-Extended. According to Google's crawler documentation, Google-Extended controls two things: training future Gemini models and grounding, which means using your content to answer prompts in Gemini Apps and Grounding with Google Search on Vertex AI. The same page says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
So if you add the token and say nothing about Google-Extended, you may also drop out of Gemini Apps answers. Google's post says it directly: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." If Gemini visibility matters to you, name Google-Extended in its own group. Naming it overrides the shared token, but at Google it also allows Gemini training. You can't split the two there today.
Microsoft goes the other way and splits them. Bing's training preference can sit in robots.txt or on a single page, while grounding is page-level only: "There is no site-wide robots.txt grounding preference." Bing also warns that blocking grounding "will significantly reduce your content's visibility in AI-generated experiences". OpenAI keeps search and training apart with two bots: its crawler page says OAI-SearchBot is for ChatGPT search and GPTBot for training, and each setting is independent.
If you've followed our coverage, you know why this matters. AI search still sends a small share of traffic, as our roundup of this autumn's AI search traffic studies showed, but platforms are getting stricter about who gets the clicks. Reddit going login-only is the latest example. Being cited in AI answers is worth protecting even while you say no to training.

What to put in your robots.txt

Here's a practical starting point for a bootstrapped content or SaaS site: no AI training anywhere except your press page, search and AI answer bots still welcome, and Gemini grounding kept on purpose.
# Shared AI training opt-out (IETF draft, 30 Oct 2026)
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

# Optional: stay eligible for Gemini Apps grounding
# (at Google this also allows Gemini training)
User-Agent: Google-Extended
Allow: /
Leave the second group out if you'd rather opt out of Gemini training too and accept the grounding trade-off. Microsoft also lets you set the training preference on a single page, with a meta tag or an X-Robots-Tag header:
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
Your goalWhat to setWho reads it
No AI training anywhere, still in searchToken group with Disallow: /Anthropic now; Microsoft from 1 Feb 2027; Google and OpenAI later
No AI training except some foldersToken group with Disallow: / plus Allow lines for those foldersSame as above
Stay in Gemini Apps answersA separate Google-Extended group with Allow: / (also allows Gemini training)Google
No training on one pageBingbot meta tag or X-Robots-Tag with ai-training:disallowMicrosoft, from 1 Feb 2027
No Bing grounding on a pageBingbot tag with ai-grounding:disallow or disallowextendedMicrosoft, from 1 Feb 2027
Stop a bot from crawling at allA Disallow group for that crawler's own name, such as GPTBotEach crawler, as today
Sources: IETF draft, Google, Microsoft Bing and OpenAI documentation, read 2 November 2026.
If your robots.txt has never been touched, start with our guide to technical SEO for startup websites. It covers where the file lives and how to test it, so one typo doesn't block Googlebot.

The Europe angle: a machine-readable "no" finally has a common format

For European site owners this token is more than convenience. Under Article 4(3) of the EU's Copyright in the Digital Single Market Directive (Adopted, EU law since 2019), commercial text and data mining is allowed unless the rightholder has reserved it "in an appropriate manner, such as machine-readable means in the case of content made publicly available online." Until now, everyone argued about which machine-readable means counted. A single token that the big AI labs have publicly promised to read is a much clearer reservation than a list of bot names.
Google also says the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the AI Act, which it says recognises robots.txt as a primary way for publishers to express machine-readable preferences. None of the four announcements says anything else about Europe, and none promises payment. Microsoft is explicit: "Neither choice guarantees inclusion, traffic, or payment." Meta wasn't part of the 30 October announcements; if its new commercial terms on AI crawling worried you, note that the shared token doesn't cover Meta unless it signs up. This is general information, not legal advice.

What this means for you

  1. Decide what is actually yours to protect. Paid courses, templates, original research and gated guides are your moat. Your pricing page and blog posts that sell your product probably aren't. Opt out the first group, keep the second open.
  2. Add the token now, even though most providers aren't live yet. It costs two lines and Anthropic is already implementing it. When Microsoft, Google and OpenAI switch on, your preference is already there. It only counts for fetches after you add it, so earlier is better.
  3. Make a deliberate Gemini decision. If customers find you through Gemini, add an explicit Google-Extended allow and accept the training trade-off. If not, leave it out. Either way, write the reason down.
  4. Don't touch your search bots. The token is for training, not crawling. Never "tidy up" by blocking Googlebot, Bingbot or OAI-SearchBot. Our guide to SEO for bootstrapped startups explains why organic search still pays the bills for most of us.
  5. Using NoArchive or NoCache? Move before 1 March 2027. After that date Bing stops reading them as training or grounding preferences. Publish the new tags and, if you want to keep grounding restricted, record the temporary setting in Bing Webmaster Tools from 15 December.
  6. Follow the draft. It's version -00 and expires on 3 May 2027 unless updated. Wording can change. We track updates in our AI SEO news for November 2026 and SEO news for November 2026.
My take: this is the first time the AI labs have agreed on anything that makes life easier for small publishers. It's not enforcement, and it won't pay you a cent. But a two-line "no" that four companies have promised in public to read is a lot better than the bot-of-the-month game we've all been losing. Add it, and keep your search visibility switched on.
Are you getting customers from AI search yet?

More and more traffic is coming from Perplexity, ChatGPT, Grok and other AI tools.

👉 Rank on ChatGPT

FAQ: the robots.txt AI training opt-out

What is __usage_ai-training?

It's a shared policy token for robots.txt, proposed in an IETF draft published on 30 October 2026 by Kevin Kelley of Anthropic. You use it in a User-Agent line, followed by normal Disallow and Allow rules, to say whether your content may be used to train generative AI models.

Which AI companies will respect it?

Anthropic says it has started implementing it. Microsoft will honour it from 1 February 2027. Google aims to support it in 2027, and OpenAI says it aims to implement it in the coming months. Other providers haven't announced support yet.

Will the token hurt my Google rankings?

Google says Google-Extended, which is how it will read the token, does not affect inclusion in Google Search and isn't a ranking signal. It can remove you from grounding in Gemini Apps unless you explicitly allow Google-Extended.

Does User-agent: * with Disallow: / already opt me out of AI training?

Not under this proposal. The draft says wildcard groups must not be read as matching the token, so a catch-all rule expresses no training preference. A wildcard Disallow would also block search engines from crawling your site.

Can I get my content removed from models already trained on it?

No. The token only covers content fetched while it's in your robots.txt. It can't undo past collection, and the draft says private agreements can override it.

Is this legally binding in the EU?

The token itself is a voluntary preference, not law. But the EU's Copyright in the Digital Single Market Directive lets rightholders reserve text and data mining through machine-readable means for public online content, and a widely recognised token makes that reservation clearer. Ask a lawyer about your own case.

About the Author

Violetta Bonenkamp, also known as MeanCEO, is an experienced startup founder with an impressive educational background including an MBA and four other higher education degrees. She has over 20 years of work experience across multiple countries, including 5 years as a solopreneur and serial entrepreneur. Throughout her startup experience she has applied for multiple startup grants at the EU level, in the Netherlands and Malta, and her startups received quite a few of those. She’s been living, studying and working in many countries around the globe and her extensive multicultural experience has influenced her immensely.
2026-11-02 04:20 News and Trends