What is llms.txt?
Also known as LLMs.txt, llms-full.txt
Definition
llms.txt is a proposed Markdown file placed at a website's root that gives AI tools a short, hand-picked guide to the site's most useful pages and what each one covers.
Key points
- llms.txt is a short Markdown guide to a site's key pages, written for AI tools.
- It is a proposal, not a standard that search engines have committed to.
- Google says no special AI text files are needed to appear in AI Overviews or AI Mode.
What does it mean in practice?
A software company with hundreds of help pages adds a file at /llms.txt. It opens with the company name and a one-line description. Then it lists the key pages under simple headings, each with a link and a short note, such as getting started, pricing and account settings.
The format was proposed by Jeremy Howard in 2024 and is set out at llmstxt.org. It aims to help AI agents find the right content without wading through menus, adverts and scripts.
Why does it matter for AI search?
It helps to be realistic about it. llms.txt is a proposal, not a standard that search engines have committed to. Google says in its guidance on AI features that you do not need AI text files to appear in AI Overviews or AI Mode.
It is quick to add and does no harm. It works best for sites with lots of documentation. For most businesses, clear pages, crawl access and good structured data will matter more.
Our llms.txt guide covers how to write one. AI crawler analytics reads your server logs and shows which AI bots visit and which pages they fetch, so you can see whether the file is ever requested.
Related terms
These terms connect to llms.txt.
- AI crawler, the bots that might read the file.
- Schema markup, a more established way to describe your content.
- Generative engine optimisation, the wider work llms.txt sits within.
Common questions
Do I need an llms.txt file?
What goes in an llms.txt file?
Is llms.txt the same as robots.txt?
Related reading
llms.txt
An honest look at llms.txt. What the file is, how to write one, an example you can copy, and what the AI engines have and have not said about it.
AI crawlers and robots.txt
Which AI crawlers visit your site, what each one is for, and how to write robots.txt rules that keep you in AI search while controlling training use.
AI crawler analytics
Load your server access logs for AI crawler analytics, seeing which AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, visit your site and pages.
AI crawler
A plain definition of AI crawlers such as GPTBot and OAI-SearchBot, what each type does, and why blocking one can cost you visibility.
AI Mode
A plain definition of Google AI Mode, how it uses query fan-out, and how to see your AI Mode impressions in Search Console.
AI Overviews
A plain definition of Google AI Overviews, how pages get cited in them, and why they matter for your search traffic and your everyday visibility online.