Which Sites Block AI Crawlers? We Read 4,762 robots.txt Files

Key findings

• Training crawlers are blocked about twice as often as AI search crawlers. 14.8% of robots.txt files block GPTBot and 13.4% block ClaudeBot, against 6.0% for OAI-SearchBot and 5.4% for Claude-SearchBot.
• Most sites that opt out of training stay in AI search. Of the 705 sites blocking GPTBot, 421 (60%) still let OAI-SearchBot in. Of the 640 blocking ClaudeBot, 385 (60%) still allow Claude-SearchBot.
• 4.5% of sites block all three AI search crawlers (OAI-SearchBot, Claude-SearchBot and PerplexityBot), which shuts out the crawlers ChatGPT search, Claude and Perplexity use to find pages to cite. 9.3% block at least one.
• Many search blocks are accidental-looking. 100 of the 284 sites that block OAI-SearchBot (35%) never name it: a catch-all User-agent: * rule does it. 56 sites block every crawler by default and allowlist Googlebot.
• 79.7% block none of the 19 AI crawlers we checked, and 113 block every one of them.
• The biggest sites block more. In the top 1,000, 16.6% block GPTBot and 8.2% block OAI-SearchBot, against 14.6% and 5.7% further down the list.

Training opt-outs rarely touch search

Every major AI company now runs separate crawlers for separate jobs, and each says so in its documentation. OpenAI says “each setting is independent of the others”: GPTBot gathers training data, OAI-SearchBot decides what can appear in ChatGPT search. Anthropic splits the same way between ClaudeBot and Claude-SearchBot (Claude Help Center), and Google-Extended controls Gemini training and grounding without affecting Google Search (Google Search Central).

Site owners have taken the hint. Among sites that block a company’s training crawler, most keep its search crawler:

The reverse almost never happens: 0 sites block OAI-SearchBot while allowing GPTBot, and 1 does the same for Anthropic.

One opt-out does reach answers. Google-Extended covers grounding in Gemini as well as training, so the 579 sites (12.2%) that block it have asked Google not to use their pages to ground Gemini’s answers, while staying in Google Search.

The catch-all problem

A rule for User-agent: * applies to every crawler the file doesn’t name. A site that writes Disallow: / under * and then allows Googlebot and Bingbot by name has, perhaps without meaning to, blocked every AI search crawler too. We found 126 sites with a blanket block; 56 of them allowlist Googlebot, and 38 of those leave OAI-SearchBot out. Only 18 blanket-blocking sites name OAI-SearchBot as an exception.

That is how 100 of the 284 OAI-SearchBot blocks and 108 of the 256 Claude-SearchBot blocks happen without the bot ever being named. If you meant to let AI search in, an allowlist written before those crawlers existed is the first place to look.

Who blocks AI search

Among the 200 highest-ranked sites we could read, these block at least one of OAI-SearchBot, Claude-SearchBot and PerplexityBot from their homepage:

Social networks and large publishers dominate the list. Most are sites with licensing businesses or content deals to protect, not brands that want buyers to find them.

The full table

Share of the 4,762 sites with a readable robots.txt that disallow each crawler from their homepage, overall and split by Tranco rank.

What to do with your own robots.txt

1. Decide training and search separately. Blocking GPTBot or ClaudeBot doesn’t remove you from ChatGPT or Claude answers; blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot does.
2. Read your User-agent: * group. If it disallows /, every AI crawler you haven’t named is blocked.
3. Check the live file, not the one in your repo. 199 sites in our sample answer /robots.txt with an HTML page, which crawlers read as no rules at all.
4. Test it. The AI crawler checker runs the same parser we used here against your site, and AI crawlers and robots.txt has a sample file that lets answer crawlers in and keeps training crawlers out.

Method and limits

We took the top 10,000 domains of the Tranco list GQ3LK (a research ranking that combines Chrome UX Report, Cloudflare Radar, Cisco Umbrella, Majestic and Farsight data over September 28 to October 27, 2026). On October 28, 2026 our crawler requested each domain’s homepage over HTTPS, following redirects. Many Tranco entries are CDN, API or tracking domains with no website: 2,155 gave no HTTP answer at all and 1,415 answered 403. We kept the 5,480 domains whose homepage returned an HTML page with status 200, counted once per final hostname after redirects. Of those, 4,762 served a robots.txt we could parse; 405 had none (a 4xx answer, which means no rules), 199 returned an HTML page and 114 failed or answered 5xx.

Each robots.txt was read with the RFC 9309 parser behind our AI crawler checker: a crawler obeys the groups that name its product token, falls back to * only when none does, and the longest matching rule wins. “Blocked” means the file disallows that crawler from the homepage path /, which is what a full opt-out looks like. A site that blocks only some sections counts as allowed. Shares are of sites with a readable robots.txt; as shares of all 5,480 websites they are slightly lower. The crawler list and roles come from each operator’s own documentation, linked from the checker; Bytespider, Amazonbot and the retired anthropic-ai token are included because many files still name them.

Our crawler identified itself as Mentioned-Research/1.0. Sites that refuse unknown bots at the firewall are missing from the sample, and those sites are probably stricter with AI crawlers too, so treat every blocking rate here as a floor rather than an estimate of the whole web. robots.txt states a preference; it doesn’t show whether a crawler obeys it, and a few sites may serve a different file to the crawlers they name. This is a snapshot of one day.

Cite this

You are welcome to quote and chart these figures with a link back. Suggested citation: Mentioned, “Which Sites Block AI Crawlers? We Read 4,762 robots.txt Files,” October 28, 2026, https://mentioned.at/learn/ai-crawler-blocking-study. Data and text are published under CC BY 4.0.

Questions

Run a free check