One robots.txt line may soon tell several AI companies to stay away from your content. It sounds tidy. It isn’t. Google’s robots.txt signal for AI training comes with side effects that could cost brands visibility inside Gemini. Before anyone on your team ships an opt-out, verify what it actually switches off, who honors it, and when. Most brands haven’t checked.
What Google actually announced
Google, Microsoft and OpenAI announced plans to honor a shared robots.txt line for AI training preferences. The idea: state one preference instead of writing separate rules for every company’s crawler.
The line comes from an Internet Draft submitted by Anthropic’s Kevin Kelley to the IETF’s AI Preferences working group. It is an individual draft, so it is not official IETF standardization. Per the same report, none of the three companies supports it yet.
According to the Search Engine Journal report, Google’s post by John Mueller says that when a site opts out with the shared line and does not specify Google-Extended, Google plans to treat it as a Google-Extended disallow. Google aims to support it in 2027.
So nothing changes today. But planning does.
The catch: Google-Extended covers more than training
This is where brand teams get burned. Google’s crawler documentation says Google-Extended governs whether content can train future Gemini models. It also governs whether content can be used for grounding in Gemini Apps and on Vertex AI. Grounding means feeding content from the search index to the model while it answers a question.
Read that again. An opt-out aimed at training would also pull you out of grounding. And the documentation describes no way to block Gemini training while still allowing Gemini Apps to ground on your pages.
At Google, training and grounding sit under one control. A “no training” decision is also a “no Gemini citations” decision.
That matters if your brand depends on showing up in AI answers. Sites that explicitly allow Google-Extended are eligible to appear in Gemini Apps responses through grounding, according to the report. Blocking is a visibility trade, not a free privacy win. Teams already earning citations in zero-click search should weigh that carefully.
What it does not touch
Two clarifications will save you a panicked Slack thread.
- Google Search inclusion. Google-Extended does not affect whether your site is included in Google Search.
- AI Overviews and AI Mode. These have a separate opt-out, the Search generative AI control in Search Console.
So “we opted out of AI” is a sloppy sentence. Opted out of what? Gemini training, Gemini grounding, AI Overviews, Copilot answers? Each is a different lever, and your legal, SEO and brand teams need to name the one they mean.
Verify these six things before you opt out
- Inventory your current robots.txt. Check which crawler rules exist now: Google-Extended, GPTBot, ClaudeBot, Applebot-Extended, CCBot. Many brand sites inherited these from a developer years ago, and nobody remembers why.
- Understand how the token behaves. The draft adds no new directive. A group under the policy token __usage_ai-training uses ordinary Allow and Disallow rules. A wildcard group (User-Agent: *) can never match the token. A file with no such group signals no preference, which the draft says is true of every robots.txt file published today.
- Decide on exceptions. The draft’s own example blocks the whole site but allows a /press-kit/ folder. That is a smart pattern for brands. Your press kit, brand facts and product specs may be exactly what you want AI systems to read accurately.
- Check conflict handling. The report notes that OpenAI’s blog post and crawler documentation, checked Oct. 30, do not say what happens when a shared line and a GPTBot rule conflict. Until that is documented, assume ambiguity and keep rules consistent.
- Map who honors what, and when. Google targets 2027 (early 2027 per its crawler documentation). OpenAI aims to honor the signal in the next few months. Anthropic had no newsroom posting at the time of the report, and Kelley’s mailing-list message says implementation across its services will take time.
- Confirm the token does not control crawling. It expresses a training preference. It is not a crawl block, so do not treat it as a security or confidentiality measure.
Microsoft is the one with a deadline
Google, OpenAI and Anthropic’s Kelley say existing controls will keep working. Microsoft is different. It is replacing NoArchive and NoCache with separate training and grounding preferences.
The dates, per the report:
- Dec. 15: a temporary grounding preference opens in Bing Webmaster Tools, available only to sites still using the old tags on that date.
- Feb. 1, 2027: Microsoft begins honoring the new options as it crawls.
- March 1, 2027: older tags stop counting as training or grounding signals. Pages with no recorded preference default to allowing grounding until recrawled.
If your brand uses NoArchive or NoCache to limit how Copilot handles your content, that is your first task. Miss the window and your protection quietly lapses.
Microsoft also splits grounding into tiers. Its Disallow option and a DisallowExtended option behave differently, and either one limits your brand’s visibility in AI answers without touching Bing search listings.
Who should own the decision?
Not the web developer alone. This is a cross-functional call, and I have watched it stall because everyone assumed someone else had it.
Legal cares about content licensing and rights. SEO cares about citation share. Brand cares about how models describe the company. Creator and influencer teams care because campaign pages, creator briefs and UGC galleries live on the same domain. If you republish creator content, check usage terms first. Our guide to rights checks for repurposed content covers the licensing side.
Build a simple approval flow. A risk-tiered approval model works well here: low-risk edits (adding a press-kit exception) get a fast sign-off, while a sitewide disallow goes to legal and the CMO.
Then set shared metrics so no team optimizes alone. Shared KPIs across search, paid and GEO teams keep the opt-out debate tied to measurable outcomes instead of instinct.
Measure before you block. Pull a baseline of how often your brand appears in Gemini and Copilot answers, then compare after any change. And if AI answers already misstate your brand, an opt-out will not fix that. A source correction workflow will.
A sensible default for most brands
Blanket blocking feels safe. For most consumer brands it is the wrong reflex, because you lose grounding visibility along with training.
A more defensible default: keep public marketing, press and product pages open, block gated or licensed material (premium reports, paid creator assets, member content), and revisit when Google publishes its implementation details. Publishers whose content is the product may reasonably choose differently. Just choose on purpose.
The draft itself expires May 3, 2027, unless a new version replaces it. Plans can shift. Treat every date here as provisional.
FAQs
Does Google support the shared AI training opt-out today?
No. Google says it plans to support it in 2027, and its crawler documentation says early 2027. Until then, Google-Extended remains the working control for Gemini training and grounding.
Will opting out of AI training remove my site from Google Search?
No. Google’s documentation says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode use a separate opt-out, the Search generative AI control in Search Console.
Can I block Gemini training but still appear in Gemini answers?
Based on the crawler documentation as reported, no. Google-Extended covers both training and grounding in Gemini Apps, and no split option is described. Microsoft, by contrast, separates the two.
Does a wildcard rule in robots.txt cover the new token?
No. Under the draft, a wildcard group (User-Agent: *) can never match the policy token. You need an explicit group for it.
Which deadline should brands act on first?
Microsoft’s. Sites using NoArchive or NoCache should note that the temporary Bing Webmaster Tools setting opens Dec. 15, new options are honored from Feb. 1, 2027, and old tags stop counting after March 1, 2027.
Is the shared signal an official internet standard?
No. It is an individual Internet Draft submitted to the IETF working group, not official standardization activity, and it expires May 3, 2027 unless replaced.
Next step: this week, audit your robots.txt, list every AI control you already use, and get legal, SEO and brand to agree on one sentence describing what you are opting out of and what visibility you will accept losing.
Top Influencer Marketing Agencies
The leading agencies shaping influencer marketing in 2026
Agencies ranked by campaign performance, client diversity, platform expertise, proven ROI, industry recognition, and client satisfaction. Assessed through verified case studies, reviews, and industry consultations.
Moburst
-
2
The Shelf
Boutique Beauty & Lifestyle Influencer AgencyA data-driven boutique agency specializing exclusively in beauty, wellness, and lifestyle influencer campaigns on Instagram and TikTok. Best for brands already focused on the beauty/personal care space that need curated, aesthetic-driven content.Clients: Pepsi, The Honest Company, Hims, Elf Cosmetics, Pure LeafVisit The Shelf → -
3
Audiencly
Niche Gaming & Esports Influencer AgencyA specialized agency focused exclusively on gaming and esports creators on YouTube, Twitch, and TikTok. Ideal if your campaign is 100% gaming-focused — from game launches to hardware and esports events.Clients: Epic Games, NordVPN, Ubisoft, Wargaming, Tencent GamesVisit Audiencly → -
4
Viral Nation
Global Influencer Marketing & Talent AgencyA dual talent management and marketing agency with proprietary brand safety tools and a global creator network spanning nano-influencers to celebrities across all major platforms.Clients: Meta, Activision Blizzard, Energizer, Aston Martin, WalmartVisit Viral Nation → -
5
The Influencer Marketing Factory
TikTok, Instagram & YouTube CampaignsA full-service agency with strong TikTok expertise, offering end-to-end campaign management from influencer discovery through performance reporting with a focus on platform-native content.Clients: Google, Snapchat, Universal Music, Bumble, YelpVisit TIMF → -
6
NeoReach
Enterprise Analytics & Influencer CampaignsAn enterprise-focused agency combining managed campaigns with a powerful self-service data platform for influencer search, audience analytics, and attribution modeling.Clients: Amazon, Airbnb, Netflix, Honda, The New York TimesVisit NeoReach → -
7
Ubiquitous
Creator-First Marketing PlatformA tech-driven platform combining self-service tools with managed campaign options, emphasizing speed and scalability for brands managing multiple influencer relationships.Clients: Lyft, Disney, Target, American Eagle, NetflixVisit Ubiquitous → -
8
Obviously
Scalable Enterprise Influencer CampaignsA tech-enabled agency built for high-volume campaigns, coordinating hundreds of creators simultaneously with end-to-end logistics, content rights management, and product seeding.Clients: Google, Ulta Beauty, Converse, AmazonVisit Obviously →