See whether your robots.txt lets ChatGPT, Claude, Perplexity, Gemini and other AI crawlers read a page. You get a verdict for each crawler and the exact line that decided it.
It fetches robots.txt from the host you enter and reads it the way RFC 9309 says a crawler must. A crawler follows the group that names it, and only falls back to User-agent: * when nothing names it. Within that group the longest matching rule wins. A missing file means every page is allowed. A file that errors means every page is blocked.
It does not check firewalls, CDN bot settings or noindex tags. For the page-level signals, use the structured data checker.
Most AI companies run more than one crawler, and they do different jobs. Blocking one does not block the others.
For a B2B site, the usual question is not “should we block AI?” but “are we accidentally blocking the bots that decide whether we’re cited?” A blanket User-agent: * / Disallow: / left over from a staging site does exactly that. Our guide to AI crawlers and robots.txt walks through the choices.
| Token | Operator | What it does |
|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search. Docs |
ChatGPT-User | OpenAI | Fetches for a ChatGPT user; robots.txt may not apply |
GPTBot | OpenAI | Model training |
Claude-SearchBot | Anthropic | Claude’s search results. Docs |
Claude-User | Anthropic | Fetches for a Claude user |
ClaudeBot | Anthropic | Model training |
PerplexityBot | Perplexity | Perplexity’s answers; not training. Docs |
Perplexity-User | Perplexity | Fetches for a user; generally ignores robots.txt |
Googlebot | Search, including AI Overviews and AI Mode. Docs | |
Google-Extended | Token: Gemini training and grounding; not Search. Docs | |
Bingbot | Microsoft | Bing search. Docs |
meta-webindexer | Meta | Meta AI search results. Docs |
meta-externalagent | Meta | Model training and indexing |
Applebot-Extended | Apple | Token: Apple model training. Docs |
Amzn-SearchBot | Amazon | Search in Amazon products such as Alexa. Docs |
DuckAssistBot | DuckDuckGo | DuckDuckGo’s AI-assisted answers. Docs |
MistralAI-User | Mistral | Fetches for a Mistral user. Docs |
CCBot | Common Crawl | Open web archive many models train on. Docs |
Purposes are from each operator’s own documentation, read on October 24, 2026. Operators add and rename crawlers, so check their pages before you change a production robots.txt.
Not from ChatGPT search. OpenAI documents GPTBot as the crawler for model training and OAI-SearchBot as the one that decides whether a site can appear in ChatGPT search answers. Blocking GPTBot and allowing OAI-SearchBot is a common setup: out of training, still eligible to be cited.
No. Google says Google-Extended governs whether Gemini may train on or ground answers in your content, and that it does not affect Google Search. AI Overviews and AI Mode are part of Search and follow the same controls: Googlebot in robots.txt, plus noindex and nosnippet on the page.
The operators of the search and training crawlers listed here say theirs do. The fetchers that act for a user are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for that page.
Being crawlable is only the first step. Whether an assistant recommends you depends on what it has read about your category, which sites it trusts and how often you appear next to the questions buyers ask. Run a free check to see where you stand.
Usually because robots.txt did not load. RFC 9309 says a crawler that gets a server error, or no answer, must assume the whole site is off limits until it can read the file. Check that /robots.txt returns a 200 with plain text.
No. It reads robots.txt, which is what crawlers are asked to follow. A firewall, CDN or bot-protection rule can block a crawler regardless of robots.txt. If the checker sees a 403 for robots.txt itself, it says so, because that often means bots are being turned away before they read anything.
A crawler that can read your pages is necessary, not sufficient. The question your team gets asked is whether ChatGPT, Claude and Gemini recommend you when a buyer asks for a shortlist, and who they name instead. That’s what Mentioned measures: the same buyer questions every week, the full answers, and a ranked list of what to fix.