Claude answers from what it learned in training and, more and more, from a live web search. Each path has its own crawler and its own rules. Here is how both work, from Anthropic’s own documentation, and what you can change.
From training. Claude learned from web text gathered before its training cutoff. If your brand was widely and consistently described then, Claude may name you with no search at all. That knowledge changes only when a new model is trained, so it moves slowly.
From a live search. Claude can search the web while it answers. Anthropic’s help center says Claude “searches the web when it helps”, that a user can also ask it to, and that “every response includes citations” (Claude Help Center). For “best tool for” and “alternatives to” questions, that search is where a newer or smaller brand gets its chance.
Anthropic documents three user agents, each with its own job (Claude Help Center). Block one and you switch off only that job:
| User agent | What it does | If you block it | Top sites blocking it |
|---|---|---|---|
ClaudeBot | Collects web content that could be used for training | Your future pages stay out of training data | 13.4% |
Claude-SearchBot | Crawls to improve the quality of Claude’s search results | Anthropic says it may reduce your visibility in search results | 5.4% |
Claude-User | Fetches a page when a user’s question needs it | Claude can’t open your page for that user | 5.6% |
Share of 4,762 robots.txt files from the Tranco top 10,000 that disallow each crawler from the homepage, read October 28, 2026. Method in our crawler study.
Most sites have got the split right: of the 640 top sites that block ClaudeBot, 385 (60%) still let Claude-SearchBot in. The ones to worry about are the accidental blocks. 108 of the 256 sites that block Claude-SearchBot never name it; a User-agent: * rule written for other bots does it. Anthropic also honours the non-standard Crawl-delay line if you need to slow its crawlers rather than stop them.
Anthropic doesn’t say in its help pages which search engine Claude uses. Its subprocessor list does. On October 28, 2026, Anthropic’s Trust Center listed three companies under “Web Search”:
The list doesn’t say how the three divide the work, so don’t read more into it than this: your pages need to be in a web index Claude can reach, and Brave’s is the one you can check yourself. Search your brand and your category at search.brave.com. Brave says its crawler doesn’t announce its own user agent, and that if Googlebot can’t crawl a page, Brave won’t either (Brave Search Help). So a page you keep away from Googlebot is kept away from Brave too. The image search inside Claude is separate: Anthropic’s help center says it’s powered by Bing.
* group.claude.com/crawling/bots.json.Ask Claude the questions your buyers ask and read the answers, including which sites it cites. Mentioned asks Claude, ChatGPT and Gemini five buyer questions about your category, shows the sentence where you’re named or who was named instead, and tracks the same questions weekly once you save them. For the other assistants, see our guides to ChatGPT, Perplexity, Gemini and Copilot.
No. ClaudeBot collects training data. Claude’s search answers depend on Claude-SearchBot and Claude-User, which Anthropic documents separately. Block ClaudeBot and leave the other two allowed if you want to stay out of training but remain searchable.
Anthropic’s subprocessor list names three web search providers for Claude: Brave Search, TurboPuffer and Parallel Web Systems. It doesn’t say how they split the work. Image search in Claude is powered by Bing, according to Anthropic’s help center.
When it searches the web, yes. Anthropic says every web search response includes citations, with links and relevant quotes, so you can see which pages an answer drew on.
Ask Claude the questions a buyer would ask about your category, with web search on, and read whether you’re named. Because answers vary, ask several questions and repeat them over time. Mentioned does this automatically on Claude, ChatGPT and Gemini.