Every major AI company now runs its own fleet of web crawlers, and they visit your website far more often than most owners realize. Some collect training data, some build search indexes for AI answers, and some fetch pages live while a user is chatting with an assistant. Blocking the wrong one can quietly remove your site from AI generated answers. Allowing the wrong one can feed your content into training datasets you never agreed to. This guide lists all 18 AI related user agents worth knowing in 2026, grouped by what they actually do.

Three Very Different Kinds of AI Bots

Before the list, one distinction matters more than anything else. AI bots fall into three categories, and the consequences of blocking each category are completely different.

Training crawlers collect content to train future AI models. Blocking them keeps your content out of training data but has no effect on whether AI assistants can cite you today.

Search index crawlers build the retrieval indexes that AI answer engines use to find and cite sources. Blocking them removes your site from AI generated answers and the referral traffic those citations bring.

User triggered fetchers load a specific page live, at the moment a real person asks an assistant about it. Blocking them means the assistant cannot read your page even when a user explicitly requests it.

Training Crawlers

GPTBot (OpenAI)

The best known AI crawler on the web. GPTBot gathers publicly available content that may be used to train OpenAI models. It respects robots.txt and OpenAI publishes its IP ranges for verification. Blocking GPTBot does not remove your site from ChatGPT search results, because those are handled by a different bot listed below.

ClaudeBot (Anthropic)

The primary crawler operated by Anthropic, the company behind the Claude assistant. It respects robots.txt directives and identifies itself clearly. An older token, anthropic-ai, still appears in some robots.txt files but current crawling is done under the ClaudeBot name.

Google-Extended (Google)

This one causes the most confusion of all. Google-Extended is not a separate crawler you will ever see in server logs. It is a control token you can reference in robots.txt to tell Google not to use your content for training Gemini models. Regular Googlebot still crawls your site for Search either way. Critically, blocking Google-Extended does not remove you from AI Overviews in Google Search, which are powered by the normal search index.

Applebot-Extended (Apple)

The same pattern as Google. The regular Applebot powers Siri and Spotlight suggestions, while Applebot-Extended is a robots.txt token that opts your content out of Apple foundation model training. Blocking the extended token does not affect how your site appears in Siri or Safari.

Meta-ExternalAgent (Meta)

The crawler Meta uses to collect public web content for training its Llama family of models and improving its AI products. It declares itself in robots.txt compatible form.

Amazonbot (Amazon)

Collects web content that improves Alexa answers and other Amazon services, including AI features. It respects standard robots.txt rules.

CCBot (Common Crawl)

Common Crawl is a nonprofit that has archived the public web since 2008 and publishes free datasets. Many AI models, especially open source ones, were trained on Common Crawl data. Blocking CCBot removes your future content from those public datasets, which indirectly affects many smaller AI systems at once.

Bytespider (ByteDance)

The crawler from the company behind TikTok, used for content recommendation and AI training. Bytespider has a documented history of aggressive crawling, and multiple independent analyses have reported that it does not consistently honor robots.txt. If you need to block it reliably, use firewall level rules rather than robots.txt alone.

Search Index Crawlers for AI Answers

OAI-SearchBot (OpenAI)

Probably the most commercially important bot on this list. OAI-SearchBot builds the index behind ChatGPT search features. When ChatGPT cites a source with a link, that source was discovered through this index. Blocking OAI-SearchBot removes your site from ChatGPT citations, which for many publishers has become a measurable traffic channel. Note that this is completely separate from GPTBot: you can block training while staying visible in ChatGPT answers, or the reverse.

PerplexityBot (Perplexity)

Builds the search index for Perplexity, the AI answer engine that cites every source it uses. Perplexity sends real referral traffic, so most content sites benefit from allowing this bot.

DuckAssistBot (DuckDuckGo)

Fetches and processes content for DuckAssist, the AI answer feature inside DuckDuckGo search. A smaller channel, but one with privacy conscious users.

User Triggered Fetchers

ChatGPT-User (OpenAI)

When a person pastes your URL into ChatGPT or asks it to read a specific page, the fetch arrives under this user agent. It acts on behalf of a human in real time.

Claude-User (Anthropic)

The equivalent fetcher for Claude. It appears when a user asks Claude to open or analyze a specific page during a conversation.

Perplexity-User (Perplexity)

Fetches pages live when a Perplexity user asks about a specific site or clicks deeper into a cited source. Perplexity has stated this agent may fetch a page even where robots.txt restricts general crawling, because it acts on direct user instruction, similar to a browser.

Meta-ExternalFetcher (Meta)

Retrieves individual links on demand for Meta AI assistant features, for example when a link is shared in a conversation and the assistant needs to summarize it.

MistralAI-User (Mistral)

The live fetcher for Le Chat, the assistant from French AI company Mistral. It retrieves pages when users reference them in conversation.

How to Allow or Block Each Bot

All well behaved bots on this list read robots.txt. The syntax is the same for every one of them:

# Allow AI answer engines to cite you
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

A popular middle path in 2026 is exactly this combination: allow the search and user triggered bots so assistants can cite you and send traffic, while disallowing pure training crawlers. Our robots.txt guide covers the full syntax, and our article on why AI bots cannot see your website explains the most common misconfigurations.

Two Warnings From the Real World

robots.txt is voluntary. It is a convention, not a firewall. Reputable companies follow it, but nothing forces compliance. If a bot ignores the rules, only server level or CDN level blocking actually stops it.

Your CDN may be blocking bots without telling you. Several large CDN providers introduced default AI bot blocking during 2025. Thousands of site owners who never touched robots.txt are returning 403 errors to GPTBot and ClaudeBot at the network edge and do not know it. The only reliable way to detect this is to request your own pages using an AI bot user agent and compare the response to a normal visit.

Check All 18 in One Scan

The CheckMy.site scanner tests your robots.txt against every user agent in this article, detects CDN managed robots.txt substitution, and performs a live fetch test that requests your homepage as GPTBot and ClaudeBot to reveal firewall level blocking that robots.txt analysis alone can never see. The full report covers 158 checks across AI visibility, SEO, performance and security, and takes about ten seconds to run.