Free robots.txt tester
See which of 45 AI and search crawlers your robots.txt allows on any page - GPTBot, ClaudeBot, PerplexityBot, Googlebot and more - with the exact rule behind each verdict and a copy-paste fix.
How it works
1. Fetch: We read robots.txt from your site's root - or use the rules you paste.
2. Match: Each crawler's token is matched to its User-agent group, and the longest matching Allow or Disallow rule for your page's path decides the verdict (RFC 9309).
3. Fix: For every blocked crawler you get the deciding rule, plus a snippet that allows them if the block wasn't intentional.
Robots.txt Tester FAQs
What does a robots.txt tester do?
It reads your robots.txt and works out, for each crawler, whether it may fetch a given URL. This one checks 45 AI and search crawlers at once - GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot and more - and shows the exact rule that decides each verdict, so you can see why a bot is blocked rather than just that it is.
How are robots.txt rules matched?
Following RFC 9309, the standard Google and most crawlers use: a group naming the crawler specifically beats the catch-all User-agent: * group; within the group, the rule with the longest matching path wins; and when an Allow and a Disallow rule are equally long, Allow wins. * matches any characters and $ anchors the end of the path. If no rule matches, the crawler is allowed.
Can I test a robots.txt before I publish it?
Yes. Choose "paste a robots.txt", paste your draft rules and optionally enter the URL or path you want to test. Nothing is fetched from your site, so you can try changes safely before they go live.
Does allowing a bot in robots.txt mean it can reach my site?
No - robots.txt is only one layer. A firewall, CDN bot protection or rate limit can still block a crawler that robots.txt allows. The free AI Crawl Checker tests every crawler live against your page to catch those blocks too.
Do all AI crawlers respect robots.txt?
The major training and search crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot) document that they follow it. Some user-triggered fetchers - such as Google-Agent and Google-NotebookLM, which fetch a page because a person asked them to - are documented to ignore it, and the tester flags those.
What happens if my site has no robots.txt?
Crawlers treat a missing robots.txt (HTTP 404) as permission to fetch everything. A robots.txt that returns a server error (5xx) is worse: Google treats it as "crawl nothing" until it can read the file again.
Can AI crawlers actually reach you?
This tool checks one thing. The free AI Crawl Checker tests 45 AI crawlers live against your page - robots.txt, firewalls, rate limits, noindex, sitemaps and more - and gives you the fix for each one that fails.
Want to see which AI bots actually visit, continuously? Try Sona Agent Analytics or book a demo.