In brief
Most AI companies now run separate crawlers for model training, for search, and for fetching a page when a user asks. You can block training crawlers such as GPTBot and still allow search crawlers such as OAI-SearchBot and PerplexityBot. If you block the search crawlers, your pages are unlikely to be cited in those engines' answers.
Key points
- OpenAI, Anthropic and Perplexity each document separate bots for training, search and user requests.
- Blocking a training bot like GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot.
- Google-Extended and Applebot-Extended are control tokens, not separate crawlers, and do not affect Google or Apple search.
- Bots that fetch a page because a user asked may not follow robots.txt.
- robots.txt is a request, not a lock, so check your server logs to see what really visits.
- Check CDN and firewall rules too, because a bot blocked there never reaches robots.txt.
The steps at a glance
- 1
Read your current robots.txt
Open yourdomain.com/robots.txt and note any rules that name AI bots or block everything with a wildcard.
- 2
Decide your policy per bot type
Choose separately for search bots, user request fetchers and training bots. Most businesses that want AI visibility allow the first two.
- 3
Write clear rules
Add a user-agent group per bot you want to treat differently, with Allow or Disallow lines, and keep private paths blocked for all.
- 4
Check CDN and firewall settings
Make sure bot protection at your host or CDN is not blocking the search crawlers you allowed in robots.txt.
- 5
Confirm in server logs
After a few weeks, check access logs to see which AI bots visit, how often, and which pages they fetch.
What are AI crawlers?
An AI crawler is a bot run by an AI company that fetches web pages. Some collect pages to train models. Some build a search index so the AI can cite pages in answers. Others fetch a single page when a user asks a question that needs it.
The split matters because each type has a different effect on your business. Blocking a training bot changes whether your text helps build future models. Blocking a search bot changes whether you can be cited in answers today. Mixing the two up is the most common mistake we see in robots.txt files.
Which AI user agents visit websites, and what is each one for?
Each vendor documents its own bots. The table below is taken from those pages. Check them again before you make changes, as vendors add bots from time to time.
| User agent | Company | What it is for | Follows robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI models | Yes |
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search answers | Yes |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user's request needs it | May not apply, as the user starts it |
| ClaudeBot | Anthropic | Collects web content used for Anthropic's models | Yes |
| Claude-SearchBot | Anthropic | Crawls to improve Claude's search results | Yes |
| Claude-User | Anthropic | Fetches pages when a Claude user asks a question | Yes |
| PerplexityBot | Perplexity | Finds and links pages in Perplexity answers, not used for model training | Yes |
| Perplexity-User | Perplexity | Visits a page to answer a user's question | Generally ignores it |
| Google-Extended | Control token for Gemini training and grounding, no crawler of its own | Yes, as a token | |
| Applebot-Extended | Apple | Control token for Apple model training, does not crawl | Yes, as a token |
| Meta-ExternalAgent | Meta | Collects content for AI model training and product indexing | Yes |
| CCBot | Common Crawl | Builds the open Common Crawl dataset, widely used in AI research | Yes |
Sources are OpenAI's crawler overview, Anthropic's crawler help page, Perplexity's bot docs and Google's common crawlers list. Apple's Applebot page, Meta's crawler docs and Common Crawl's CCBot page cover the rest.
Does blocking GPTBot stop ChatGPT from citing you?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot to find pages it can cite.
OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. They can still appear as navigational links. So a site that blocks OAI-SearchBot has, in effect, stepped out of ChatGPT search. A site that blocks only GPTBot has not.
The same pattern holds at Anthropic and Perplexity. Each has a search bot that is separate from any training collection. If AI visibility matters to you, the search bots are the ones to keep open. The guide on how to rank in ChatGPT covers what happens after the crawler has your page.
How do Google AI Overviews and AI Mode fit in?
Google's AI features use the normal Googlebot crawl. There is no separate Google AI search bot to allow.
Google-Extended is different. It is a robots.txt token that controls whether content Google has crawled can be used to train future Gemini models. It also covers grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search.
To limit what appears in AI Overviews, Google points to snippet controls such as nosnippet and max-snippet. Those also limit your normal search snippets, so use them with care. The AI Overviews guide covers what earns a citation there.
What should your robots.txt look like?
There is no single right answer. It depends on how you feel about training use. Here is an example for a business that wants to be cited in AI search but does not want its content used for training.
# Search and user request bots, allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /account/
Disallow: /checkout/
# Training bots and control tokens, blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /account/
Disallow: /checkout/
Sitemap: https://www.example.co.uk/sitemap.xml
A few points about this file. Several user agent lines can share one group of rules. A bot follows the most specific group that names it, so the named groups override the wildcard group. That is why the private paths are repeated in the first group.
If you are happy for your content to be used in training, simply leave the training bots out. They will then follow the wildcard group like everyone else.
Should you block AI training bots?
It is a business choice, and there are fair points on both sides. No one can tell you for certain how training data affects later answers.
The case for allowing them is simple. Models learn what they know about brands partly from training data. Engines that answer from model knowledge alone, such as Gemini or Claude asked without web search, can only describe you from what they learnt. If your own pages were never in that data, the model relies on what others wrote about you.
The case for blocking is also real. Publishers who sell their writing, or firms with paid research, may not want it used to build products they get nothing from. Some organisations also have legal or policy reasons to opt out.
A sensible middle path for most service businesses is to allow the search bots, then decide on training bots with whoever owns the content. Write the decision down, so the next person to edit robots.txt knows why the rules are there.
How often should you review robots.txt?
A quick review every quarter is enough for most sites. Do it sooner after a site move, a new CMS or a change of hosting.
Vendors also change their bots. New user agents appear and old ones get new jobs. When you review, open each vendor's crawler page and compare it with your rules. It takes ten minutes, and it catches the case where a new search bot is being blocked by an old wildcard rule.
Why might AI bots still visit pages you blocked?
robots.txt is a request, not a lock. Well run crawlers follow it. Google's own robots.txt guide notes that other crawlers might not.
There are three common reasons you may still see visits.
- User request fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.
- Fake user agents. Some scrapers pretend to be well known bots. Common Crawl, for example, warns about crawlers falsely claiming to be CCBot and publishes IP ranges to check against.
- Caching. Changes to robots.txt are not picked up instantly. Give it a day or so before judging.
If you need a page kept private, use a login or password. robots.txt is public and anyone can read the paths you list in it.
What else can block AI crawlers by accident?
Hosting and CDN settings are the usual cause. Many services now offer bot protection or AI bot blocking as a setting. If that is switched on, the bot is turned away before it ever reads robots.txt.
It is worth checking three places.
- Your CDN or firewall dashboard for bot rules or AI crawler settings.
- Your security plugin, if you run WordPress or a similar CMS.
- Your server logs, to see whether allowed bots get normal responses or error codes.
A business can write a perfect robots.txt and still be invisible to ChatGPT search because of one checkbox in a CDN. It is a short check that is easy to skip.
How do you check which AI bots actually visit?
Server access logs are the most reliable record. Each request shows the user agent, the page, the time and the response code.
Filter the logs by the bot names in the table above. Look at how often each one visits, which pages it fetches, and whether it gets errors. A search bot that visits often but only ever sees your cookie page has a problem worth fixing. A training bot you blocked that still appears may be a fake.
This also helps with prioritising. If the search bots rarely reach your service pages, improve your internal links and sitemap before worrying about content. Pages that are never fetched cannot be cited, a point covered in how AI engines choose sources.
How Axiom GEO helps
AI crawler analytics loads your server access logs and shows which AI bots crawl your site, how often and which pages. It covers GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot, Amazonbot and more. AI citation tracking then shows which of your pages the engines actually cite in answers. Putting the two side by side shows whether a page is not being crawled or is crawled but not chosen.
Sources
These are the external pages this guide relies on.