AI Crawler Census checks whether a site's robots.txt blocks the crawlers that feed ChatGPT, Perplexity, and Google AI Overviews, and shows what a page looks like to those bots versus a normal browser -- paste a URL and it tells you which AI crawlers are allowed in, which are blocked, and why.
It's backed by an original census of the top 5,000 sites (Tranco list), collected 2026-09-07. Of the 4,999 sites checked, 2,771 have a robots.txt file at all. Among those:
- 296 sites (10.7%) block AI-search crawlers like GPTBot and OAI-SearchBot while still allowing Googlebot -- often without meaning to, since most site owners write rules for Google and forget the newer AI agents exist.
-
- 238 sites block both GPTBot (OpenAI's training crawler) and OAI-SearchBot (the one that actually indexes pages for ChatGPT search) outright.
-
- Individual crawler block rates vary a lot: CCBot (Common Crawl) is blocked by 20.7% of sites with a robots.txt, GPTBot by 19.3%, ClaudeBot by 18.6%, but the search-specific bots are blocked far less -- OAI-SearchBot only 8.7%, Perplexity's crawler 13.0%.
The distinction matters because a training-crawler block and a search-crawler block have very different consequences: blocking GPTBot keeps your content out of OpenAI's model training, but blocking OAI-SearchBot or PerplexityBot means your pages simply cannot be cited in ChatGPT or Perplexity answers, even indirectly, no matter how good the content is.
The property also ships a free llms.txt generator/validator and a robots.txt linter, both usable without an account. No signup, no paid tier -- the whole thing runs client-side against the site you enter.
