Every AI company runs more than one crawler, and they do different jobs. Here is what each one does, what blocking it actually changes, and a robots.txt you can copy.
The short answer. If you want to be recommended by AI assistants, allow their search crawlers: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity, and Googlebot and Bingbot for the search engines assistants draw on. Whether you also allow the training crawlers, such as GPTBot and ClaudeBot, is a separate choice about future models. It doesn’t decide whether you can be cited today.
Read the operators’ own documentation and the crawlers fall into three groups.
These index pages so an assistant can find them when it searches during a conversation. OpenAI says sites that opt out of OAI-SearchBot won’t appear in ChatGPT search answers. Anthropic says disabling Claude-SearchBot may reduce your visibility in Claude’s search results. Perplexity says PerplexityBot surfaces and links sites in its answers and isn’t used for training. Meta’s meta-webindexer does the same job for Meta AI.
Google and Microsoft don’t run separate crawlers for this. Google says AI Overviews and AI Mode use the same controls as Search, so Googlebot is the one that matters there.
ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User open a page because someone in a conversation asked about it or pasted the link. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, since a person requested the page. Anthropic says its crawlers, Claude-User included, honour robots.txt.
GPTBot, ClaudeBot, meta-externalagent and Common Crawl’s CCBot collect pages that may be used to train models. Two entries here aren’t crawlers at all. Google-Extended controls whether Gemini may use pages Googlebot already fetched, for training and for grounding, and Google says it has no effect on Search. Applebot-Extended controls whether Apple may train on what Applebot fetched.
| If you block | What changes |
|---|---|
OAI-SearchBot | Your pages can’t appear in ChatGPT search answers |
Claude-SearchBot | Claude’s search can’t index your pages |
PerplexityBot | Perplexity can’t surface and link your pages |
Googlebot | You leave Google Search, including AI Overviews and AI Mode |
GPTBot, ClaudeBot | Your content is excluded from future training. Live search answers aren’t affected |
Google-Extended | Gemini can’t train on or ground answers in your pages. Google Search isn’t affected |
CCBot | New pages stay out of Common Crawl, which many models are trained on |
From each operator’s documentation, read October 24, 2026: OpenAI, Anthropic, Perplexity, Google, Meta, Apple, Common Crawl.
This allows every search and answer crawler, keeps your private paths private, and leaves the training decision explicit. Delete the last group if you’re happy for your public pages to be used in training.
# Everyone: crawl the public site, not the app
User-agent: *
Disallow: /app/
Disallow: /admin/
# AI search and answer crawlers: same rules, named so nobody
# "tidies" them into a block later
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
Disallow: /app/
Disallow: /admin/
# Training: opt out (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlOne rule trips people up: a crawler follows only the group that names it. If you give OAI-SearchBot its own group, it ignores everything under User-agent: *, so the private paths have to be repeated, as above.
User-agent: * / Disallow: / blocks every compliant crawler, search ones included.The AI crawler checker reads your robots.txt the way the RFC says a crawler must and shows the verdict for 18 crawlers, with the line that decided each one.
Letting the search crawlers in makes you eligible to be cited. It doesn’t make an assistant pick you when a buyer asks for a shortlist. That depends on what the model already knows about your category, which sources it trusts, and whether your pages answer the question that was asked. The next step is to find out what ChatGPT, Claude and Gemini say today. See the guide to getting mentioned for what moves it.
Usually not the search ones. OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether those assistants can find and cite your pages when they search. Blocking training crawlers such as GPTBot and ClaudeBot is a separate decision about whether your content is used to train future models, and it doesn’t remove you from live answers.
No. Google says Google-Extended doesn’t affect Google Search, and AI Overviews and AI Mode are part of Search. They follow Googlebot’s robots.txt rules and the noindex and nosnippet controls.
It depends on the operator. Meta says its crawlers may cache robots.txt for up to 24 hours, Amazon says about 24 hours, and DuckDuckGo says 72 hours for DuckAssistBot. Plan for a few days.
For crawlers that follow it, yes. It’s a request, not a lock: some user-triggered fetchers say they may not follow it, and anything that ignores the standard isn’t stopped by it. Firewall and CDN rules are the tool for enforcement.