GPTBot, ClaudeBot, PerplexityBot: Every Major AI Crawler and How to Control It

If you have looked at your server logs in the last year, you have seen new names: GPTBot, ClaudeBot, PerplexityBot, meta-externalagent. These are AI crawlers, and they are not all doing the same job. Some collect text to train the next model. Some fetch a page live because a user just asked a question about it. Blocking the wrong one can quietly remove you from the AI answers that are starting to send real traffic.

This is a reference for which bot is which, what each one is for, and the exact robots.txt lines to allow the crawlers that cite you while opting out of the ones that train on you. One thing to be honest about up front, because the rest of the article depends on it: robots.txt is a request, not a fence. It works on voluntary compliance. The major AI companies documented below do honor it, but a directive in robots.txt does not physically stop anything the way an authentication wall does.

Training crawlers vs retrieval fetchers

The single distinction that makes this whole topic tractable is the difference between a training crawler and a retrieval fetcher.

A training crawler collects content in bulk to build the dataset a future model learns from. GPTBot and ClaudeBot are the headline examples. Blocking these is how you say "do not use my content to train your models." Blocking them does not remove you from that AI's live answers, because live answers usually come from a different bot.

A retrieval fetcher hits your page in real time because a user asked something and the assistant is looking up an answer right now. ChatGPT-User and Claude-User fire when a person clicks a link or the assistant browses on their behalf. A search-index bot like OAI-SearchBot builds the index that ChatGPT search reads from. Block these and you can disappear from AI answers even though your content was never used for training.

So the useful mental model is: training crawlers are the ones you might want to opt out of on principle, retrieval and search bots are the ones you almost certainly want to keep, because they are the path to citations. If you want the deeper strategy on earning those citations, we cover it in how to get cited by ChatGPT, Perplexity, and Google AI Overviews.

The major AI crawler user-agents

Here are the user-agents worth knowing, grouped by operator, with what each one does as of mid-2026. User-agent tokens change over time, so treat the token column as "what to match in robots.txt" and confirm against each vendor's published docs when you are locking down a policy.

| User-agent | Operator | What it does | Training? | | --- | --- | --- | --- | | GPTBot | OpenAI | Bulk crawl for model training | Yes | | OAI-SearchBot | OpenAI | Builds the search index behind ChatGPT search | No | | ChatGPT-User | OpenAI | Live fetch when a user action triggers browsing | No | | ClaudeBot | Anthropic | Bulk crawl for model training | Yes | | Claude-User | Anthropic | Live fetch for a user's Claude request | No | | PerplexityBot | Perplexity | Crawls to index pages for Perplexity answers | No | | Perplexity-User | Perplexity | Live fetch when a user's question needs your page | No | | Google-Extended | Google | Opt-out token for Gemini training and grounding | Yes | | Applebot-Extended | Apple | Opt-out token for Apple Intelligence training | Yes | | CCBot | Common Crawl | Open crawl dataset many models train on | Yes | | Bytespider | ByteDance | Bulk crawl associated with training | Yes | | meta-externalagent | Meta | Crawl for Meta AI training and products | Yes |

A few notes that the table cannot hold. Google-Extended and Applebot-Extended are not separate crawlers with their own traffic. They are control tokens. Googlebot still crawls your site for search as it always did, and Google-Extended only governs whether that content feeds Gemini and AI features. Applebot-Extended works the same way over the normal Applebot. Blocking Google-Extended does not affect your Google Search ranking, and this trips people up constantly.

CCBot belongs to Common Crawl, a nonprofit that publishes a giant open web crawl. It is worth calling out separately because many models train on the Common Crawl dataset rather than crawling you directly, so blocking CCBot has broader reach than its single line suggests.

Robots.txt that opts out of training but stays in AI search

Here is the practical goal for most publishers: do not let the models train on your content, but do stay reachable by the retrieval and search bots that can cite you. That is a selective policy, not a blanket block.

# Opt out of AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Keep AI search and live retrieval allowed (no Disallow lines needed;
# they are allowed by default, shown here only for clarity)
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

The comment about the Allow lines matters. In robots.txt, anything you do not disallow is allowed. You do not need to list OAI-SearchBot or ChatGPT-User to permit them; they are permitted by default. The Allow lines above are documentation for the next person who edits the file, not a functional requirement. What makes this policy selective is that GPTBot is disallowed while OAI-SearchBot is not, even though both are OpenAI.

If instead you want to block everything AI, put every training and retrieval user-agent in its own group with Disallow: /. And if you want the opposite, a site that welcomes all of it, do nothing; the default is open.

Build and sanity-check the file with our robots.txt generator so you do not typo a user-agent token or accidentally disallow your whole site with a stray slash. The classic trap of robots.txt versus a noindex tag (they do different jobs and one does not remove pages from Google) is worth understanding before you rely on either; we walk through it in the robots.txt guide for SEO.

The llms.txt file is a different lever

Separate from robots.txt, there is an emerging convention called llms.txt: a Markdown file at your root that gives AI systems a curated map of your most important content in a clean, easy-to-parse form. It is not an access-control file and it does not block anything. Think of it as a table of contents written for language models rather than a gate.

Adoption is still early and no major AI vendor has committed to reading it as a ranking input, so set expectations accordingly. It costs little to publish and it does no harm, which is roughly the honest case for it today. You can generate one with our llms.txt generator, and we go deeper on what it is and is not in the llms.txt AI SEO guide.

Why "voluntary compliance" is the whole story

It is worth stating plainly because a lot of coverage glosses over it. robots.txt has no enforcement mechanism. It is a text file that well-behaved crawlers choose to read and obey. The named companies in the table above have published documentation saying they respect it, and in practice they do. But a crawler that ignores robots.txt faces no technical barrier from the file itself.

If your requirement is "this content must not be accessed by machines," robots.txt is the wrong tool. Put the content behind authentication, rate-limit or block by user-agent and IP at the server or CDN layer, or use a WAF rule. robots.txt is the right tool for the common, reasonable goal of telling compliant crawlers your preferences. Just do not confuse a politely honored request with a lock.

Frequently Asked Questions

Does blocking GPTBot remove me from ChatGPT answers?

No. GPTBot is the training crawler. ChatGPT's live answers and search results come from OAI-SearchBot and ChatGPT-User, which are separate user-agents. If you block GPTBot but leave those allowed, you opt out of training while staying eligible to be cited in ChatGPT. Blocking all three is what removes you entirely.

Will blocking Google-Extended hurt my Google Search ranking?

No. Google-Extended only controls whether your content is used for Gemini and Google's generative AI features. Regular Googlebot, which handles Search indexing and ranking, is a different crawler and is unaffected. You can opt out of AI use without touching your organic search visibility.

Is robots.txt legally binding on AI companies?

Robots.txt is a voluntary standard, not a law. It works because reputable crawlers choose to honor it, and the major AI operators document that they do. It is not an access control mechanism and provides no technical enforcement. For content that genuinely must not be accessed, use authentication or server-level blocking instead.

What is the difference between ClaudeBot and Claude-User?

ClaudeBot is Anthropic's bulk training crawler; block it to opt out of training. Claude-User is the live fetcher that retrieves a page when a person's Claude request needs it. Keeping Claude-User allowed lets your content appear in Claude's answers to users even if you have blocked training.

Do I need both robots.txt and llms.txt?

They do different jobs. Robots.txt controls crawler access and is honored by compliant bots. Llms.txt is an optional, emerging convention that offers AI systems a curated content map; it controls nothing and adoption is still limited. Publishing llms.txt is low-cost and optional; a correct robots.txt is the file that actually governs access.