When a strange user agent shows up in the server logs, people search its name. We pulled that demand for 16 AI crawlers — and found the bots that decide whether you get cited are the ones nobody looks up.
| User agent | Operator · role | US / month | Worldwide / month |
|---|---|---|---|
GPTBot | OpenAI · training | 350 | 1,900 |
Bytespider | ByteDance · crawler | 200 | 1,600 |
OAI-SearchBot | OpenAI · search | 200 | 1,100 |
CCBot | Common Crawl · training corpus | 150 | 1,000 |
Meta-ExternalAgent | Meta · training | 150 | 1,000 |
Amazonbot | Amazon | 150 | 800 |
PerplexityBot | Perplexity · search | 80 | 500 |
Google-Extended | Google · Gemini training/grounding token | 80 | 450 |
Claude-SearchBot | Anthropic · search | 50 | 250 |
ChatGPT-User | OpenAI · user-triggered fetch | 40 | 150 |
DuckAssistBot | DuckDuckGo · answers | 30 | 150 |
Applebot-Extended | Apple · training token | 30 | 100 |
anthropic-ai | Anthropic · legacy token | 10 | 80 |
Claude-User | Anthropic · user-triggered fetch | 10 | 20 |
Perplexity-User | Perplexity · user-triggered fetch | 10 | 20 |
“claudebot” shows 4,700 US and 14,000 worldwide searches a month, which would make it the most-searched crawler by far. We left it out of the table because Ahrefs files it under the parent topic “claude” — a large share of those searches are almost certainly people looking for the Claude chatbot, not Anthropic’s crawler. Treat that number as an upper bound, not a finding.
Blocking a training crawler like GPTBot keeps your pages out of future model training; it does not remove you from ChatGPT’s live search answers. The crawlers that do control live citation — OAI-SearchBot, Claude-SearchBot, PerplexityBot — get a fraction of the attention. The risk is a site owner who reads about GPTBot, writes a broad block, and switches off the search crawlers along with it. Our guide to AI crawlers and robots.txt lists what each operator says blocking its bot changes, with links to their documentation, and our free AI crawler checker shows which of them your robots.txt currently allows.
Searches about blocking are small: “block ai crawlers” draws about 70 US searches a month and “block gptbot” about 40. Most people searching a bot’s name seem to want to know what it is before they decide.
Search interest is one thing; what sites actually block is another. Our crawl of 4,762 robots.txt files found the same pattern in practice: training crawlers are blocked about twice as often as AI search crawlers.
Search volumes are Ahrefs Keywords Explorer estimates of average monthly searches, pulled on October 25, 2026 through our SiteWide connector. They are modelled estimates, not Google’s own counts, and they are rounded by Ahrefs. KD is Ahrefs’ keyword difficulty (0–100). CPC is Ahrefs’ estimate of what advertisers pay per click in the US. Role labels summarise what each operator documents; see the linked guide for sources. Search interest measures curiosity, not crawl volume or traffic — a bot can crawl heavily and be searched rarely.
You are welcome to quote and chart these figures with a link back. Suggested citation: Mentioned, “Which AI Crawlers Do Site Owners Look Up? Search Data for 16 Bots,” October 25, 2026, https://mentioned.at/learn/ai-crawler-search-demand. Data and text are published under CC BY 4.0.