Shared robots.txt AI Training Opt-Out: What Kolkata Websites Should Do Now

On 30 October 2026, Google and Microsoft announced support for a single, shared robots.txt rule that tells AI companies not to train models on your website. The rule covers training only: it does not remove you from Google Search, Bing or AI answers, so most businesses can block training and still be cited by AI assistants. For brands that rely on AEO services in Kolkata to win AI visibility, the job now is to make two separate decisions, one for training and one for AI answers, and to stop blocking AI crawlers by accident.

What exactly was announced on 30 October 2026?

Two announcements landed on the same day, and they fit together.

Google published Google’s shared AI training control announcement on its Search Central blog. It says Google plans to support a shared “policy token” in robots.txt, based on work from the AI Preferences (AIPREF) working group at the Internet Engineering Task Force (IETF). The key points:

• Website owners will be able to express an AI training preference across participating AI providers in one step, instead of listing every company’s crawler by name.
• If a site opts out of training with the shared token and does not mention Google-Extended, Google plans to treat it as if Google-Extended had been disallowed.
• Company-specific controls such as Google-Extended keep working alongside the shared signal.
• Google says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
• Google aims to support the new control in 2027. It is still a draft proposal, not a finished standard.

Microsoft went further. In Bing’s training and grounding preferences post, it adopted the same shared training rule and added a separate “AI Grounding” preference. Grounding is when an AI system reads your page to help write an answer, usually with a citation and a link back.

Training vs grounding: why the difference matters

For two years, many site owners treated “AI” as one switch. They blocked GPTBot, Google-Extended or other AI user agents and assumed the only effect was protecting their content. In practice, some of those rules also reduced their chances of being quoted in AI answers. The new controls split the decision in two.

Preference	What it controls	Where you set it	Default
AI Training (shared rule)	Whether your content can be used to train future generative AI models	robots.txt for a site or path; Microsoft also allows a page-level override	Allow
AI Grounding (Microsoft)	Whether your content can help inform AI-generated answers in Bing, Copilot and Microsoft 365	Page level only	Allow
Google-Extended (Google)	Google’s existing control for AI training; allowing it keeps you eligible for grounding in Gemini Apps	robots.txt	Allow
Normal search crawling	Whether you appear in classic Google and Bing results	robots.txt rules for Googlebot and Bingbot	Allow

Microsoft’s grounding preference has three options: Allow, DisallowExtended (your content can inform Microsoft’s own AI experiences such as Copilot and Microsoft 365, but not the grounding services Microsoft sells to third parties) and Disallow. Microsoft states plainly that choosing DisallowExtended or Disallow for grounding will reduce brand visibility in AI experiences, and that neither choice guarantees inclusion, traffic or payment. Your eligibility for ordinary Bing search results does not change either way.

What does the new robots.txt rule look like?

Bing’s implementation guide shows the shared training rule written as a special user agent called a policy token. A whole-site training opt-out looks like this:

User-Agent: __usage_ai-training
Disallow: /

Ordinary path rules still apply, so you could, for example, opt out a members-only resources folder while leaving your press kit open. Two cautions before you copy it:

• The syntax comes from an IETF Internet-Draft that is still being finalised, so details may change.
• Support is staggered. Microsoft starts honouring the new preferences from 1 February 2027, and Google aims to support the shared control in 2027. Keep your existing company-specific rules until each provider confirms support.

Key dates for Indian website owners

1. 30 October 2026: Google and Microsoft announce support for the shared AI training preference.
2. 1 February 2027: Microsoft begins honouring the new AI Training and AI Grounding preferences as it crawls.
3. 1 March 2027: Microsoft stops reading the old NoArchive and NoCache tags as training or grounding preferences. If your pages use them, publish the new preferences before this date.
4. 30 September 2027: the latest date until which Microsoft’s temporary transition grounding preference, recorded in Bing Webmaster Tools, applies to a page (or until the page is recrawled, whichever is earlier).
5. During 2027: Google’s target window for supporting the shared control.

How AEO services in Kolkata should respond: a step-by-step plan

None of this needs panic. It needs a short audit and a written decision. This is the process we recommend to clients who want to protect their content without disappearing from AI answers.

Step 1: Audit your current robots.txt and meta tags

Open yoursite.com/robots.txt and list every AI-related rule: GPTBot, ClaudeBot, Google-Extended, CCBot and others. Note who added each rule and why. Then check page templates for NoArchive and NoCache meta tags, which some security and caching plugins add without anyone noticing. If you are on WordPress or a custom CMS, your developer or our web design and development team can pull this list in under an hour.

Step 2: Decide training and grounding separately

Ask two questions with whoever owns the website and the brand:

• Training: Are we comfortable with AI companies using our content to train future models? Publishers, course creators and research-heavy firms often say no. Many service businesses do not mind.
• Grounding: Do we want AI assistants to read and cite our pages when someone asks for a recommendation? For almost every Kolkata business that sells locally, the answer should be yes.

Step 3: Protect the pages that win AI citations

Service pages, pricing explainers, FAQs, location pages and case-study pages are the pages AI assistants quote. Keep them open for grounding and for Google-Extended. If you choose to block training, use the shared token rather than blocking AI user agents outright, so you do not lose citations by accident. Our earlier piece on the Google AI Mode links update explains why those citation links now matter for traffic.

Step 4: Clean up old NoArchive and NoCache tags

If your site uses these tags, decide whether they still reflect what you want. After 1 March 2027 Microsoft will not treat them as AI preferences, so replace them with the new page-level preferences if you genuinely want to limit training or grounding.

Step 5: Measure AI visibility, not just rankings

Once your rules are settled, track whether you are actually being cited. Search Console and Bing Webmaster Tools both report more AI-related data than they did a year ago; our guide to Search Console AI insights walks through the reports worth checking each month.

What this means for businesses in Kolkata and West Bengal

Most businesses in Kolkata are not large publishers worried about licensing deals. They are clinics in Salt Lake, manufacturers in Howrah, coaching institutes in Ballygunge, D2C brands selling across India and B2B service firms in Sector V. For them, the bigger risk is invisibility: a buyer asks ChatGPT, Gemini or Copilot for the best option in Kolkata and the business is not mentioned at all.

Some practical examples of how the decision might look:

• A local service business (a dental clinic, an interior design studio, a CA firm) usually gains nothing by blocking AI. Leaving training and grounding at the default keeps every door open.
• A content-led business (an edtech company, a market-research firm, a regional news or lifestyle portal) may want to block training on premium articles while keeping service and landing pages open for grounding.
• An ecommerce or D2C brand should keep product, category and FAQ pages open for grounding, because AI shopping answers depend on them, and decide training separately for blogs or lookbooks.

The same principle applies across India: write down a policy, apply it deliberately, and review it when Google and OpenAI confirm their support dates. If you are still building your overall AI search approach, our comparison of GEO and traditional SEO strategy is a useful starting point.

Common mistakes to avoid

• Blocking every AI crawler “to be safe”. This can quietly remove you from AI answers your customers already use.
• Assuming the new rule is live everywhere. It is a draft, and each provider has its own timeline.
• Removing company-specific rules too early. Keep Google-Extended and similar rules until the shared token is confirmed as supported.
• Letting plugins decide. Security, SEO and caching plugins can add robots rules or meta tags. Check what they output.
• Treating it as a one-time task. Revisit the policy every quarter while the standard is finalised.

Where AEO and GEO fit in

Robots.txt rules only decide whether AI systems may use your content. Whether they actually choose to cite you depends on the content itself: clear answers near the top of the page, structured data, consistent business details across the web, and pages that address the long, conversational questions people now type into AI tools. That is the work our AEO and GEO services focus on, alongside the technical hygiene covered above.

Next steps

Check your robots.txt this week, write down your training and grounding decision, and put 1 February and 1 March 2027 in the calendar. If you would like a second pair of eyes on your rules or your AI visibility, book a free 30-minute strategy call with the Kreate Edgee team and we will walk through it with you.