An individual Internet-Draft posted to the Internet Engineering Task Force’s Datatracker on October 30 proposes a shared way for website owners to signal whether their content can be used to train generative AI models, rather than maintaining separate robots.txt rules for each company. The document introduces a common policy token called __usage_ai-training and was authored by Kevin Kelley of Anthropic. The IETF says the draft is not endorsed by the organization and has no formal standing in its standards process. The proposal is designed to work with the existing robots.txt format.
Under the proposal, a website operator can add the token to a robots.txt user-agent group and use familiar Allow and Disallow rules to express training preferences for particular paths. The draft does not introduce new robots.txt directives or change how existing crawler rules work. It also makes clear that the token is not a crawling instruction: it communicates a preference about how retrieved content may be used, and software that does not recognize the token will continue to ignore that group.
OpenAI said on October 30 that it will honor the shared signal and plans to implement it in the coming months. The company said the common approach should reduce the need for publishers to maintain separate training opt-outs for every AI provider. Google also announced support plans, but its Search Central update says it aims to add the control in 2027. If a site uses the new token to opt out and does not explicitly specify Google-Extended, Google plans to treat that preference as a disallow for Google-Extended.
The shared token is intended to sit alongside existing provider-specific controls, not immediately replace them. Its practical effect will depend on which AI companies adopt and honor it. The proposal does not create a technical barrier to scraping, guarantee that every crawler will comply, or stop a service that ignores the token from fetching pages. It gives publishers another way to express their preferences as companies develop broader controls for AI crawlers, but it is not yet a finalized industry standard.
If this article helped you, please consider supporting our work. Every small contribution keeps Abijita.com independent and running.
