If you want to keep AI assistants from training on your content but still want to show up in search and get cited in AI answers, you need to control the crawlers separately — because training bots and search bots are not the same. Blocking Google-Extended in your robots.txt stops your content from being used to train Google's Gemini models, but it does not remove you from Google Search or Google's AI Overviews. That single distinction is the whole game, and most site owners get it wrong by blocking the wrong bot and quietly disappearing from answers they wanted to win.

Training crawlers vs indexing crawlers: know the difference

Every major AI company now runs at least two distinct user-agents with different jobs. One gathers data to train the model; the other fetches pages in real time to answer a live question and cite a source. Blocking one does not block the other, and treating them as one bucket is how sites lose citations.

The practical takeaway: if you block GPTBot and Google-Extended but leave the search-facing agents open, you opt out of training while staying eligible to be cited. If you sledgehammer everything with a blanket Disallow, you also cut yourself out of the answer boxes that send referral traffic.

How to opt out of AI training without hurting visibility

The controls live in robots.txt, and they are per-user-agent. A minimal opt-out of training that preserves search and citation eligibility looks like this in plain terms: allow Googlebot and the search-facing agents everywhere, and disallow only the training agents. For example, disallow Google-Extended, GPTBot, ClaudeBot, and CCBot for all paths, while leaving Googlebot, OAI-SearchBot, and ChatGPT-User untouched.

A few rules keep this from backfiring. Robots.txt is case-insensitive on the directive but the user-agent token must match exactly — Google-Extended is not GoogleExtended. Each user-agent block is independent, so a bot obeys the most specific group that names it and ignores your generic User-agent: * rules if a named block exists for it. And robots.txt only governs crawling; it is a request, not a firewall, so a well-behaved bot honors it while a bad actor does not. For crawlers that ignore the rules, you verify identity and block at the edge instead of trusting the user-agent string — see our guide on verifying whether a crawler is really GPTBot or a fake.

The strategic question: should you block training at all?

Opting out of training feels protective, but it comes with a trade-off worth thinking through. Being included in training data is part of how a model comes to "know" your brand exists, which shapes whether it mentions you unprompted later. Blocking training does not remove you from live-retrieval citations, but it can reduce the baseline familiarity a model has with your name. For most publishers the sensible split is: allow live-retrieval and citation agents so you win referral traffic and brand mentions, and decide on training crawlers based on whether your content is a commercial asset you license or protect.

If your business model is selling access to content, blocking training bots and negotiating licensing makes sense. If you are a service business that wants to be recommended, blocking training can be counterproductive. The newer Content Signals extension to robots.txt lets you express intent more granularly — allowing search and AI-input use while disallowing training — instead of the blunt all-or-nothing block.

Verify what each bot actually sees

After you edit robots.txt, do not assume it worked. Fetch it as each bot would and confirm the right groups apply, then check your server logs to see which agents are actually hitting you and whether they are respecting the rules. The full roster of agents to account for is bigger than most people expect — our complete list of AI crawlers in 2026 breaks down all of them and what each one is for. Remember that being crawlable is necessary but not sufficient: whether you then get quoted depends on content quality and structure, which is a separate battle covered in how ChatGPT chooses which sites to cite.

The one mistake to avoid above all: a stray Disallow: / under a search agent, or a blanket block that catches OAI-SearchBot and Googlebot along with the training bots. That is how sites vanish from AI answers overnight without realizing why. To catch that kind of accidental self-block — plus the dozens of other crawl and rendering issues that quietly keep you out of AI answers — run the free CheckMy.site scanner and see exactly what the bots can and cannot reach on your site.