CrawlCheck is a free browser-based AI crawler audit and robots.txt generator that checks your site's robots.txt file against a database of 60+ AI crawlers and builds a replacement policy file, plus JSON-LD structured data for FAQ, HowTo and Article markup. It runs at crawlcheck.dev with no signup.
What is CrawlCheck?
CrawlCheck is a web tool that takes a site's existing robots.txt as input and reports, crawler by crawler, which AI agents that file allows and which it blocks, drawn from a maintained database of more than 60 user agents. Its second module writes a new robots.txt policy from presets or per-crawler toggles and updates the output live for copying to https://yourdomain.com/robots.txt. A third module generates FAQPage, HowTo and Article JSON-LD wrapped in a script tag for pasting into a page's head. All parsing, auditing and generation run locally in the browser — nothing is uploaded, logged or tracked.
What makes CrawlCheck stand out?
- Four-group crawler database — crawlers are sorted into AI search and indexers (10 preset entries such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, Amazonbot, YouBot, DuckDuckBot, Bingbot, PetalBot), assistant and live fetch (11, including ChatGPT-User, Claude-User, Claude-Web, Perplexity-User, Gemini-Deep-Research, Amzn-User, MistralAI-User, cohere-ai, DuckAssistBot, Copilot/Microsoft), agents and developer tools (9, including ChatGPT Agent, Claude-Code, Cursor, Devin, Google-Agent, Google-Gemini-CLI, Windsurf, viberank) and training data crawlers (31, including GPTBot, ClaudeBot, Google-Extended, Bytespider, TikTokSpider, CCBot, meta-externalagent, Bedrockbot, DeepSeekBot, MistralAI-Training, AI2Bot, LAIONDownloader, ImagesiftBot, Omgilibot).
- Live robots.txt generator — start from a preset or tune crawler by crawler; the file header records that it was generated with CrawlCheck, and the default policy uses a wildcard allow, producing a 61 allowed / 0 blocked state with copy and download buttons.
- Risk flags on aggressive crawlers — names like Bytespider, DeepSeekBot, LAIONDownloader and ISSCyberRiskCrawler carry a warning marker in the generator.
- Structured data builder — FAQPage, HowTo and Article forms output JSON-LD with fields for questions and answers, how-to title, description and steps, and headline, description, author, date published, image URL and publisher.
- Client-side processing — the FAQ states explicitly that no data is sent anywhere; parsing, auditing and generation happen in the browser.
Who should use CrawlCheck?
- SEO and GEO specialists: audit a client's robots.txt against the full crawler list before a launch or migration and hand back a policy that keeps citation-driving search bots allowed.
- Publishers and site owners: find out whether a Disallow line written years ago is hiding the site from ChatGPT, Perplexity and Claude at once.
- Web developers: produce a drop-in robots.txt file and add FAQ or HowTo JSON-LD to template heads.
- Agency teams tracking AI visibility: check which training crawlers are feeding on a client's content and decide what to do at the CDN level.
What can you do with CrawlCheck?
- Audit an existing file: paste your robots.txt and read a per-crawler allowed/blocked breakdown across the four crawler groups, with the cost of each block noted.
- Generate a policy: choose a preset or toggle individual crawlers, then copy or download the resulting robots.txt for your domain root.
- Emit JSON-LD: fill a FAQ, HowTo or Article form and copy the generated script block into a page's head.
- Plan enforcement: use the warning flags to identify crawlers that ignore robots.txt and block them at the CDN or WAF instead.
How does CrawlCheck work?
All three tools sit on one page and run in the browser. Paste a robots.txt into the auditor for an allow/block report; adjust presets or per-crawler toggles in the generator until the live preview matches your intent, then copy it to your robots.txt path; or fill a schema form to produce JSON-LD markup. No account or upload step is involved.
Pricing
CrawlCheck is free and requires no signup; the page states it runs entirely in your browser. No paid tiers or usage limits are listed.
FAQ
Is CrawlCheck free?
Yes. CrawlCheck is free, requires no signup, and its parsing, auditing and generation all run locally in your browser. The page lists no paid plans or account tiers.
Does blocking GPTBot hurt my Google rankings?
No. According to the site's own FAQ, GPTBot only trains OpenAI models, while Google Search relies on Googlebot, a separate crawler. Blocking GPTBot has no effect on Google rankings, and blocking Google-Extended does not affect Search either.
Some AI crawlers ignore robots.txt — what can I do?
robots.txt is a request, not a wall. CrawlCheck's generated file notes that crawlers ignoring it, such as Bytespider, must be enforced at the CDN or WAF level, for example with Cloudflare user-agent rules.
What is GEO?
GEO, or Generative Engine Optimization, is the practice of making content easy for AI answer engines such as ChatGPT, Perplexity, Claude and Gemini to find, trust and cite. CrawlCheck frames technical accessibility — a robots.txt policy that admits the right AI crawlers — as its foundation.
Is my robots.txt uploaded to a server?
No. The tool states that all parsing, auditing and generation happen locally in the browser, and nothing is uploaded, logged or tracked.
