To allow GPTBot, add a User-agent: GPTBot
block with Allow: / to your robots.txt and
make sure no broader rule blocks it. Then verify the bot can actually
reach your pages with a live crawl probe, a rule that
looks right can still be undone by a conflicting wildcard or a
firewall.
Crawler access is the first requirement for AI visibility: an engine that cannot fetch your pages cannot cite them.
GPTBot is OpenAI's web crawler. It gathers content that helps ChatGPT discover, understand and cite websites. If you want to appear in ChatGPT's answers, GPTBot needs to be allowed. Blocking it, often by accident, makes you invisible to one of the largest AI audiences on the web.
The robots.txt rules to add
robots.txt lives at the root of your domain
(https://yoursite.com/robots.txt). To explicitly welcome
the major AI crawlers, add:
User-agent: GPTBot
Allow: /User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Here's what each one controls:
- GPTBot, OpenAI's crawler for training and improving ChatGPT.
- OAI-SearchBot, OpenAI's crawler for ChatGPT search results and citations.
- ClaudeBot, Anthropic's crawler for Claude.
- PerplexityBot, Perplexity's answer-engine crawler.
- Google-Extended, Google's control for Gemini and AI Overviews.
Watch out for the rule that silently blocks everything
The most common reason AI crawlers get locked out isn't an explicit block, it's a broad catch-all. If your file contains something like:
User-agent: *
Disallow: /
…then every bot without its own named block is blocked. Crawlers match
the most specific user-agent block, so adding the named
Allow rules above protects the AI bots even if a wildcard
disallow exists. Double-check there's no stray Disallow: /
inside a GPTBot block left over from a past "block AI" decision.
Allowing AI crawlers ≠ giving up control
You can still protect private paths. Keep AI bots out of admin, checkout or staging areas while allowing the content you want cited:
User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /cart/
The GPTBot user agent string
Two different things get called "the user agent". robots.txt matches
on the short token only, which for OpenAI's training crawler
is exactly GPTBot. Your server logs, on the other hand,
record the full user agent header, which looks like this:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbotOpenAI runs three separate agents, and they do different jobs:
- GPTBot, crawls content used to train and improve the models.
- OAI-SearchBot, crawls content for ChatGPT search. This is the one that produces live links and citations in answers, so if your goal is getting cited, it matters more than GPTBot.
- ChatGPT-User, fetches a specific page on demand when a user asks ChatGPT to open a URL mid-conversation.
They are independent rules. Allowing GPTBot does not
allow OAI-SearchBot. This trips up a lot of sites that
think they have opted in to ChatGPT and have only opted in to
training. The version numbers also increment over time, so if you
write log filters or firewall rules, match on the token and not the
full string.
How to detect GPTBot in your server logs
The fastest check is a grep against your access log:
grep -i "gptbot" access.log | tail -20All OpenAI agents at once
grep -iE "gptbot|oai-searchbot|chatgpt-user" access.log | wc -l
One important caveat: a user agent header is just a string the client sends, and anyone can set it to anything. Scrapers routinely impersonate GPTBot to look legitimate. If the distinction matters to you, verify the request actually came from OpenAI by checking the source IP against OpenAI's published crawler IP ranges, or by running a reverse DNS lookup on the IP and confirming it resolves to an OpenAI-owned hostname. User agent alone proves nothing.
If you find no GPTBot hits at all, that is a signal in itself. It usually means one of three things: the crawler is blocked in robots.txt, it is being stopped at the CDN or firewall before it reaches your application, or your site simply has not been discovered yet because nothing links to it.
Verify it actually works
A correct-looking rule can still fail in practice, a CDN, WAF or bot-management firewall may block the crawler's IP before robots.txt even matters. So don't assume; verify. Sona's free AI visibility checker runs a live GPTBot probe that confirms whether OpenAI can actually fetch your pages right now, alongside 34 other AI-readiness checks.
Allowing crawlers is step one. To understand the full picture of what makes AI engines cite you, read how to check your website's AI visibility and how to get cited by ChatGPT and Perplexity.
