Free robots.txt tester
See which of 47 AI and search crawlers your robots.txt allows on any page - GPTBot, ClaudeBot, PerplexityBot, Googlebot and more - with the exact rule behind each verdict and a copy-paste fix.
Test one page against 47 crawlers in seconds.
A blocked bot goes unnoticed, and nobody tells you.
- A Disallow rule that blocks AI crawlers by accident
- A User-agent: * rule that overrides what you meant
- A robots.txt that returns a server error
- Fetchers that ignore robots.txt
Sona shows you which rule blocks which bot.
- 47 AI and search crawlers tested at once
- The exact rule behind each verdict
- A copy-paste fix for every blocked crawler
- Draft rules tested before you publish
What is a robots.txt file?
A robots.txt file sits at the root of your site and tells crawlers which paths they may fetch. Each crawler looks for the group of rules that names it, then follows the rules in that group.
This is step one, not the finish line. A page also needs clear structure and tags AI engines can read. This free tester looks at your robots.txt rules. The free AI Crawl Checker tests whether AI crawlers can actually reach your site.
Check your robots.txt in three steps
Three steps from page URL to a clear verdict.
Fetch
We read robots.txt from your site's root. Or you can paste your own rules to test them before you publish.
Match
Each crawler's token is matched to its User-agent group. The longest matching Allow or Disallow rule for your page's path decides the verdict (RFC 9309).
Fix
For every blocked crawler you get the deciding rule. You also get a snippet that allows it if the block wasn't intentional.
What this checks
It reads your robots.txt and works out, for each crawler, whether it may fetch a given URL.
The crawlers
Who gets tested.
- 47 AI and search crawlers at once
- GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot and more
The rules
How a verdict is decided (RFC 9309).
- A group naming the crawler beats the User-agent: * group
- The rule with the longest matching path wins
- Allow wins when two rules are equally long
The result
What you get back.
- The exact rule behind each verdict
- A copy-paste fix for every blocked crawler
- Crawlers that ignore robots.txt are flagged
This one checks 47 AI and search crawlers at once - GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot and more - and shows the exact rule that decides each verdict, so you can see why a bot is blocked rather than just that it is.
See the difference a clear robots.txt makes
The same robots.txt, before and after the fix.
- Disallow: / blocks GPTBot and ClaudeBot
- Allow: /blog/ lets GPTBot and ClaudeBot through
Why robots.txt matters
A small text file decides which bots can read your site.
Allowed is not always reachable
A firewall, CDN bot protection or rate limit can still block a crawler that robots.txt allows.
Not every bot obeys it
Some user-triggered fetchers are documented to ignore robots.txt, and the tester flags those.
A server error is worse than none
A robots.txt that returns a server error (5xx) is worse: Google treats it as "crawl nothing" until it can read the file again.
This check is a snapshot. Sona is the system.
Robots.txt decides which crawlers may fetch your pages. Sona Agent Analytics shows which AI crawlers read your pages and how many visitors they send back. AI Search Insights shows whether you are cited in AI answers.
AI Search Insights
"Am I being cited in AI answers, and how do I compare to competitors?"
- Track the prompts your buyers actually ask across ChatGPT, Gemini, Google AI Overviews, Perplexity, Claude, DeepSeek, Grok & more
- See your rankings and share of answer against competitors
- Find which sources answer engines cite, and which of your pages get surfaced
Agent Analytics
"Are AI crawlers reading my site, and is AI search sending me traffic?"
- See AI crawler activity by bot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended
- Measure referral traffic clicking through from AI answers
- Page-level view of what gets crawled and what drives AI referrals
Robots.txt Tester FAQs
Quick answers about robots.txt and this tester.