robots.txt Mistakes and New Directives in 4,762 Top Sites

Key findings

• 3.7% of robots.txt files contain a line crawlers ignore under RFC 9309: an unknown directive, a path without a leading slash, a line with no colon, or a rule before any User-agent.
• Content-Signal, Cloudflare’s new AI-use directive, is already in 4.7% of files (224). 161 of them say ai-train=no; 223 say search=yes.
• 12.2% still set Crawl-delay, which Google says it doesn’t support. Anthropic says it honours it.
• 24 files still use Noindex, a line Google’s parser ignores.
• 70.9% declare a sitemap, the one non-rule line every major search engine reads.
• 199 sites answer /robots.txt with an HTML page and 6 have HTML pasted into the file itself.

The lines that do nothing

RFC 9309, the robots.txt standard, defines user-agent, allow and disallow. Google adds sitemap and says that “other fields such as crawl-delay aren’t supported” and that rules other than those are ignored by its parser (Google Search Central). Our parser follows the RFC and also accepts the widely used crawl-delay, host and clean-param. Leaving Content-Signal aside, here is what it skipped:

The unknown directives are a record of ideas that never became standards. Besides Content-Signal, the most common were Noindex (24 files), AI-related inventions such as LLMs, LLM-policy and Agentmap (10, 8 and 8 files), and the old Request-rate and Visit-time. A line pointing at your llms.txt from robots.txt is harmless, but no crawler we know of reads it there.

Lines with no colon were usually something else entirely: in 6 files, HTML or CSS from a web page had ended up inside robots.txt.

Content-Signal: a new line for AI use

Cloudflare’s Content Signals add one line to robots.txt saying what a site’s content may be used for: search (building a search index and returning links and short excerpts), ai-input (feeding content into a model at answer time, as in retrieval or grounding) and ai-train (training or fine-tuning models). Cloudflare is clear that these are preferences, which crawlers may ignore. In our sample:

The pattern matches what sites do with crawler rules: yes to search, no to training, and on AI answers, most either say yes or say nothing. Only 10 of 224 sites opt out of ai-input, the signal that covers being quoted in AI answers. Our parser treats the line as unknown, like any crawler that hasn’t adopted it; we left it out of the error counts above.

Clean up your own file

1. Delete Noindex and anything else Google lists as unsupported. To keep a page out of results, use a noindex meta tag on the page, and let crawlers fetch it so they can see the tag.
2. Start every path with /, and put every rule under a User-agent line.
3. Fetch the live file and confirm it’s plain text, not your homepage.
4. Check what each AI crawler is allowed to do with the AI crawler checker, which lists every line it ignored and why. For the crawlers themselves, see which sites block AI crawlers.

Method and limits

We took the top 10,000 domains of the Tranco list GQ3LK (a research ranking that combines Chrome UX Report, Cloudflare Radar, Cisco Umbrella, Majestic and Farsight data over September 28 to October 27, 2026). On October 28, 2026 our crawler requested each domain’s homepage over HTTPS, following redirects. Many Tranco entries are CDN, API or tracking domains with no website: 2,155 gave no HTTP answer at all and 1,415 answered 403. We kept the 5,480 domains whose homepage returned an HTML page with status 200, counted once per final hostname after redirects. 4,762 of them served a robots.txt we could parse; their median size was 600 bytes.

Files were parsed with the RFC 9309 parser behind our AI crawler checker, which reports every line a crawler would skip. Directive counts are the share of files that contain each field at least once, comments removed. Other parsers are more forgiving in places (Google’s accepts some misspellings, as ours does), so a flagged line is one a strict reader ignores, not proof that every crawler does. Our crawler identified itself as Mentioned-Research/1.0. Sites that refuse unknown bots at the firewall are missing from the sample, and those sites are probably stricter with AI crawlers too, so treat every blocking rate here as a floor rather than an estimate of the whole web.

Cite this

You are welcome to quote and chart these figures with a link back. Suggested citation: Mentioned, “robots.txt Mistakes and New Directives in 4,762 Top Sites,” October 28, 2026, https://mentioned.at/learn/robots-txt-errors. Data and text are published under CC BY 4.0.

Questions

Run a free check