Block the training crawlers, allow the search crawlers, and don't expect robots.txt to stop the bots that act on behalf of a person. GPTBot, ClaudeBot and Google-Extended feed model training. OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether your pages show up in AI answers. The policy is easy. The file is where it goes wrong.
Which AI crawlers should you block in robots.txt?
Block the crawlers that collect content for model training and allow the ones that power AI search. OpenAI documents this split itself: GPTBot is for training, OAI-SearchBot is for search, and each setting is independent. A site can appear in ChatGPT search while opting out of training. Anthropic and Perplexity draw the same line.
Here is the list that matters, from each vendor's own documentation:
- Training: GPTBot (OpenAI), ClaudeBot (Anthropic) and the Google-Extended token, which controls whether Google may use your content for Gemini.
- Search: OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot. Perplexity states that PerplexityBot is not used to crawl content for AI foundation models.
- Fetching for a person: ChatGPT-User, Claude-User and Perplexity-User. They load a page because someone asked a question about it.
The next two sections cover where these rules break in practice.
The mistake that undoes your rules: the wildcard group
A bot that has its own group in robots.txt ignores the wildcard group. That surprises people. The standard, RFC 9309, says crawlers obey the User-agent: * group only "if no matching group exists." Once you add a group that names GPTBot, GPTBot reads that group and nothing else.
Say your file has a wildcard group that keeps every crawler out of /checkout/ and /account/. Then you add a GPTBot group that blocks /blog/. GPTBot is now free to fetch /checkout/ and /account/, because the wildcard rules never reach it. The fix is dull: repeat the shared rules inside every bot group you add. Then open the live file and read each group as if you were that bot.
What can't robots.txt control?
Robots.txt can't control bots that fetch a page because a person asked. OpenAI says of ChatGPT-User that "robots.txt rules may not apply," since users start those actions. Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason. Anthropic lists Claude-User as a separate bot, so check its page before assuming either way.
The standard itself is blunt about the limits. RFC 9309 says its rules "are not a form of access authorization," and Google's documentation says robots.txt "is not a mechanism for keeping a web page out of Google." If a page must stay private, it needs a login. A Disallow line also advertises the path to anyone who reads the file.
Does blocking GPTBot or Google-Extended hurt your search visibility?
Blocking GPTBot does not remove you from ChatGPT search, because that is OAI-SearchBot's job. Blocking Google-Extended does not affect Google Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." What it controls is Gemini Apps, the Vertex AI API for Gemini, and grounding in those products.
Blocking the search bots is the move with a cost. OpenAI says sites that opt out of OAI-SearchBot don't appear in ChatGPT search answers, though they can still show up as navigational links. Anthropic says that disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." If your goal is to be named when someone asks an assistant for a supplier, those are the bots you want to let in.
One thing no vendor page states: whether allowing training changes how often your brand is mentioned in answers. Treat advice that promises it as a guess. Blocking training is a choice about your content, not a visibility tactic.
How to set it up and check it
Write one group per bot, with the same path rules repeated in each. Put Disallow: / in the groups for GPTBot, ClaudeBot and Google-Extended if you want out of training. Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Do this for every subdomain, since Anthropic notes the file must be repeated on each one.
Then verify instead of trusting the file. Give it about a day: OpenAI says it can take roughly 24 hours after a robots.txt update for its systems to adjust, and RFC 9309 says crawlers should not use a cached copy for more than 24 hours. Check your server logs for the user agent names above. Don't block by IP address. Anthropic warns that IP blocking can stop its bots from reading robots.txt at all, so they can't see your opt-out. It publishes its ranges at claude.com/crawling/bots.json if you need to confirm a request is genuine.
Getting cited in AI answers takes more than a clean robots.txt: the content has to be worth quoting. That's the work behind the organic marketing and AI-answer visibility side of what onlyspell does.
If you want a second pair of eyes on your robots.txt and how your site shows up in AI answers, talk to us about your project.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search results come from OAI-SearchBot, which you control separately. OpenAI says each setting is independent, so a site can allow OAI-SearchBot and block GPTBot.
Does blocking Google-Extended affect my Google rankings?
No. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It only controls whether your content is used for Gemini Apps, the Vertex AI API for Gemini and grounding in those products.
Can robots.txt stop AI companies from reading my pages?
Only the bots that choose to obey it. RFC 9309 says the rules are not access authorization. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Private content needs a login, not a Disallow line.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central: Google's common crawlers
- Google Search Central: Introduction to robots.txt
- Perplexity: Perplexity crawlers
- IETF: RFC 9309, Robots Exclusion Protocol
If you still need additional help in developing and designing your website, our magic is here to help. Contact us today, and our wizards will help you immediately with your digital marketing needs!
