__usage_ai-training
A cross-vendor policy token, not a crawler. One Disallow group expresses an AI-training opt-out to OpenAI, Google, Microsoft, and Anthropic at once.
Reference reviewed by Sona
| Operator | OpenAI, Google, Microsoft & Anthropic |
|---|---|
| Powers | Generative-AI training use at four vendors, controlled by one robots.txt group |
| Purpose | Training opt-out token |
| User-agent token | __usage_ai-training |
| Respects robots.txt | Yes |
__usage_ai-training is the first robots.txt token in this directory that belongs to no single operator. It comes from an IETF Internet-Draft (draft-kelley-aipref-training-preference-signal, authored at Anthropic and published 30 October 2026), and on the day it appeared, OpenAI, Google, Microsoft, and Anthropic each stated they would honor it. The idea: instead of maintaining a separate opt-out group per vendor, one group under this token expresses a machine-readable preference about generative-AI training to every crawler operator that implements the draft.
Mechanically it adds no new syntax. You write an ordinary robots.txt group - User-agent: __usage_ai-training, then Allow and Disallow rules - and the paths you disallow carry a preference against AI-training use, while allowed paths carry a preference in favor. But it has one matching rule that ordinary tokens do not: a wildcard group never applies to it. A site whose entire robots.txt is User-agent: * with Disallow: / has expressed no training preference at all under the draft; only a group naming the token counts. A file with no such group is silent, not opted out.
Equally important is what the token does not do. It expresses a usage preference, not an access rule - crawlers still fetch your pages, your search indexing and AI citations are untouched, and the signal has no effect on retrieval or user-triggered fetching. It also does not replace the per-vendor controls: Google says Google-Extended remains in force, OpenAI scopes its commitment to foundational generative-AI training, and every operator outside the four signatories is unaffected. Treat it as a fifth layer, not a consolidation.
The draft is an individual submission, not an adopted IETF standard, and vendor implementation is phased - announced commitments, not yet a verified behavior you can observe in logs. The pragmatic posture while that settles: add the token group to state your preference on the record, and keep your existing GPTBot, Google-Extended, Applebot-Extended, and CCBot rules exactly as they are.
How __usage_ai-training behaves
- Sends zero requests - it is a preference signal read from robots.txt, not a crawler.
- Must be named explicitly: wildcard (*) groups never apply to it, and a robots.txt without the token expresses no preference.
- Governs generative-AI training use only - crawling, search indexing, and user-triggered fetches continue regardless.
- Honored (per announced commitments) by OpenAI, Google, Microsoft, and Anthropic; every other operator ignores it.
Allow __usage_ai-training
An explicit Allow states a preference in favor of training use to all four major labs at once - the clearest way to volunteer your content for future models.
User-agent: __usage_ai-training Allow: /
Block __usage_ai-training
One group opts you out of generative-AI training at OpenAI, Google, Microsoft, and Anthropic simultaneously - without costing you crawling, search visibility, or AI citations.
User-agent: __usage_ai-training Disallow: /
How to verify __usage_ai-training
No published verification method
There is no traffic to verify - the token never sends a request, so any log line claiming a __usage_ai-training user-agent is fabricated by definition. Whether the four vendors honor the preference cannot be confirmed from your logs at all; like Google-Extended, the only test is output-side, by checking whether future models reproduce content only your site publishes. Note the draft is weeks old, so even output-side evidence will lag.
Check an IP against this botCommonly confused with __usage_ai-training
The single-vendor version of the same idea. Google-Extended opts out of Gemini training only; this token addresses four vendors at once - and Google states it does not replace Google-Extended.
Also a no-traffic usage token, but Apple is not among the four signatories - an __usage_ai-training group says nothing to Applebot-Extended.
__usage_ai-training FAQs
Does this replace Google-Extended, Applebot-Extended, or my GPTBot rules?
No. Google states Google-Extended remains in force, Apple is not a signatory at all, and per-crawler access rules do a different job (blocking fetches rather than signaling usage preference). Keep existing groups and add this one alongside them.
My robots.txt is User-agent: * with Disallow: / - am I opted out of AI training?
Not under this draft. Wildcard groups deliberately do not match policy tokens, so an opt-out requires a group that names __usage_ai-training explicitly. Absence of the token means no preference expressed.
Will using it hurt my search rankings or AI citations?
No. The token expresses a training-use preference only; it does not block any crawler from fetching pages, so search indexing, AI answers, and citations are unaffected.
Who actually honors it?
OpenAI, Google, Microsoft, and Anthropic announced commitments on 30 October 2026, with implementation phased over the following months - OpenAI scopes its support to foundational generative-AI training. The draft is voluntary and not an adopted IETF standard, and no other operator is bound by it.
Can __usage_ai-training read your page right now?
Test any URL and see exactly what AI crawlers receive.