Skip to content
GEO by AXIOMAI
Glossary

What is an AI crawler?

Also known as AI bot, LLM crawler, AI user agent

Definition

An AI crawler is a bot run by an AI company that visits web pages to collect content. It may train models, power live AI search answers, or fetch a page a user asked about.

Key points

  • AI crawlers are bots run by AI companies to collect or fetch web content.
  • Training crawlers and search crawlers do different jobs and can be allowed or blocked separately.
  • Blocking an AI search crawler keeps your pages out of that engine's cited answers.

What does it mean in practice?

Look at the server logs for a garden centre's website and you may see visits from bots such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Each one belongs to an AI company and has its own job.

OpenAI is a good example. It documents several crawlers. GPTBot collects content that may be used to train models. OAI-SearchBot finds pages to show in ChatGPT search. ChatGPT-User visits a page when a user asks ChatGPT to. Each can be allowed or blocked on its own in robots.txt.

If an AI search crawler cannot reach your pages, those pages cannot be cited in its answers. Blocking OAI-SearchBot, for example, keeps a site out of ChatGPT search summaries. Blocking a training crawler is a different choice, with different trade-offs.

It helps to check your robots.txt, firewall and CDN settings, since some security settings can block AI bots without you realising.

AI crawler analytics in Axiom GEO reads your server access logs and shows which AI bots crawl the site, how often and which pages. Our guide to AI crawlers and robots.txt explains what to allow and block.

These terms connect to AI crawlers.

  • llms.txt, a proposed guide file for AI tools.
  • Grounding, the live look-up that search crawlers support.
  • Citation, what a crawled page can become in an answer.
  • AI referral traffic, the visits that can follow a citation.

Common questions

Should I block AI crawlers?

It depends on which ones and why. Blocking an AI search crawler, such as OAI-SearchBot, keeps your pages out of that engine's answers. Blocking a training crawler, such as GPTBot, is a separate choice about whether your content may be used to train models. One option is to allow search crawlers and decide on training crawlers case by case.

How do I see which AI crawlers visit my site?

Your server access logs record every visit with a user agent name. Looking for names such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot shows which AI bots visit, how often and which pages. Some hosting and security tools also report this.
Reviewed
All terms A to Z

Related reading

AI crawler analytics

Load your server access logs for AI crawler analytics, seeing which AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, visit your site and pages.

Guide

AI crawlers and robots.txt

Which AI crawlers visit your site, what each one is for, and how to write robots.txt rules that keep you in AI search while controlling training use.

Glossary

llms.txt

A plain definition of llms.txt, what goes in the file, and an honest look at how much it actually matters for AI search results today and next year.

Glossary

Grounding

A plain definition of grounding in AI answers, how it differs from answering from training, and what it means for your brand.

Glossary

AI Mode

A plain definition of Google AI Mode, how it uses query fan-out, and how to see your AI Mode impressions in Search Console.

Glossary

AI Overviews

A plain definition of Google AI Overviews, how pages get cited in them, and why they matter for your search traffic and your everyday visibility online.

What is an AI crawler? Meaning and examples | Axiom GEO