What does AI crawler analytics answer?
It answers a question most analytics tools cannot. Are AI engines actually reading my site, and which pages? Google Analytics will not tell you, because bots do not run the tracking code. Your server logs will, because they record every request.
The trouble is that logs are long, raw text files. Few marketers want to search them by hand. This feature reads them and pulls out the AI bots for you.
How does it work?
You bring the logs and the platform does the sorting.
- Get your access logs. Download them from your hosting control panel or content delivery network.
- Load them into the platform. Upload the log files for the period you want to look at.
- Match the bots. Each request is checked against the user agents of known AI crawlers.
- Count and group. Visits are grouped by bot and by page, and counted over time.
- Read the results. See which bots came, how often, and which pages they fetched.
Which AI crawlers does it recognise?
It recognises the main AI bots and more besides. The table shows some of the ones you are most likely to see and who runs them.
| Bot | Run by | What it is broadly for |
|---|---|---|
| GPTBot | OpenAI | Collecting pages that may be used to train models |
| OAI-SearchBot | OpenAI | Finding pages for ChatGPT search |
| ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to |
| ClaudeBot | Anthropic | Collecting pages for Claude |
| PerplexityBot | Perplexity | Finding pages for Perplexity answers |
| Meta-ExternalAgent | Meta | Collecting pages for Meta's AI work |
| Bytespider | ByteDance | Collecting pages for ByteDance |
| CCBot | Common Crawl | Building an open web archive used by many AI projects |
| Amazonbot | Amazon | Collecting pages for Amazon services |
Our glossary entry on AI crawlers explains the different types in more detail.
What do you see?
You see each AI bot with its number of visits over time. You can see which pages each one fetched and how often. That quickly shows whether bots reach your important pages or spend their time on old tags, filters and archive pages.
Three patterns come up often. A bot that never appears may be blocked in robots.txt, sometimes by accident. A bot that only fetches the home page may not be finding the rest of the site. And a sudden drop after a site change is often a sign that something in the new setup is turning bots away.
Why does it matter for AI search?
AI engines that search the web can only use pages they can fetch. If OAI-SearchBot or PerplexityBot never reach your service pages, those pages have little chance of being read when a buyer asks a question. Crawler data shows whether that first step is happening.
It is not the whole story. A crawl is not a citation, and a page fetched every day can still be ignored in answers. AI citation tracking shows which pages actually get cited. Crawler data sits one step before that.
It also helps you check your own rules. Many sites changed robots.txt in a hurry when AI crawlers first appeared. Our guide to AI crawlers and robots.txt explains the options. The logs show whether the rules you set are doing what you meant. If you are thinking about an llms.txt file, our llms.txt guide covers what it is and what it is not.
What should you do with what you find?
Most findings lead to one of a few simple actions.
- If a bot you want is missing, check robots.txt and any firewall or bot protection rules.
- If bots skip your key pages, check internal links and your XML sitemap.
- If bots spend time on thin or duplicate pages, tidy those up or keep bots out of them.
- If a bot you do not want is busy, decide whether to block it in robots.txt.
It helps to load a fresh set of logs after each change. That way you can see whether the fix worked rather than assume it did.
What are the limits?
A user agent is only a label, and some scrapers pretend to be well-known bots. So treat odd spikes with some caution. The data is also only as complete as the logs you load. If a content delivery network serves most requests, those logs may sit with the network rather than your server.
Who uses it?
Technical SEO leads use it after site migrations and robots.txt changes. B2B and SaaS teams use it to see whether AI bots reach their documentation and product pages. Agencies use it as part of an AI visibility audit for new clients.
Where it fits
Crawler data is the first link in the chain. Page optimisation checks that the pages bots fetch are well built. AI citation tracking shows which pages get cited in answers. AI traffic analytics shows the visitors those answers send.