Over the past year, llms.txt has been sold as the quick fix for AI search: add one text file to your site and ChatGPT, Gemini and Perplexity will understand you better. Thousands of sites have added one, and SEO tools now flag sites that don't have it.
The data from 2026 tells a different story. This guide explains what llms.txt is, what the evidence says about who actually reads it, where it genuinely helps, and what really decides whether AI search tools can find and cite your site.
Key takeaways
- llms.txt is a Markdown file that gives AI tools a curated map of your site, proposed in September 2024.
- Google says its search ignores llms.txt, and 97% of files got zero requests in May 2026 (Ahrefs).
- It does help AI coding assistants read developer documentation.
- AI search visibility depends on allowing the right crawlers, content in the HTML and strong SEO basics.
- Blocking training bots like GPTBot doesn't remove you from ChatGPT search; blocking OAI-SearchBot does.
The short answer llms.txt doesn't help you rank or get cited in AI search. Google says its search ignores it, and log data shows AI search crawlers almost never fetch it. It is useful for one audience: AI coding assistants reading developer documentation. If you run docs or an API, add one. Otherwise, spend the time on the basics below.
What is llms.txt?
llms.txt was proposed by Jeremy Howard of Answer.AI in September 2024. It's a Markdown file at /llms.txt that gives AI tools a short, curated map of your site: what it is, and links to the pages that matter most, ideally as clean Markdown rather than HTML full of menus and scripts.
The format is simple. Only the H1 title is required:
- an H1 with the site or project name;
- a blockquote with a one-line summary;
- optional paragraphs with extra context;
- H2 sections containing lists of links, each optionally followed by a short note;
- by convention, an Optional section for links an AI tool can skip when it's short on space.
# Example Hosting
> Example Hosting provides managed WordPress and VPS hosting with data centres in the EU and US.
Support is available 24/7. Prices are listed in USD.
## Docs
- [Getting started](https://example.com/docs/start.md): Set up your first site
- [DNS guide](https://example.com/docs/dns.md): Point a domain to your hosting
## Optional
- [Changelog](https://example.com/changelog.md)Version 2 of the spec, published in August 2026, added a few things: Markdown copies of pages can live at page.md as well as page.html.md, files can cover a subfolder such as /docs/llms.txt, and pages can point to their Markdown version and to the llms.txt that covers them with standard Link tags or headers. One common belief is wrong: llms-full.txt, a single file with all your docs, is not part of the spec. It's a convention introduced by documentation platforms such as Mintlify.
Who has adopted it?
Adoption has grown fast. Originality.ai counted 4,088 llms.txt files in June 2025 and 36,120 by May 2026, an 8.8x rise, across more than three million sites it tracks. That's still only around 1% of them. Ahrefs found that 28% of the 137,000 domains using its analytics product publish one, but those are mostly SEO-minded sites.
Yoast SEO, AIOSEO, Wix, Mintlify and GitBook can all generate the file for you, and Chrome's Lighthouse added an llms.txt check to its new "Agentic Browsing" category in May 2026. That check only flags a file that errors; a missing file counts as "not applicable". The AI companies publish llms.txt files too, but only for their developer documentation.
Do AI search engines actually read it?
This is where the hype falls apart.
Google: "Google Search ignores them"
Google's 2026 guide to optimising for its generative AI features lists llms.txt among things you can ignore. It says you don't need new machine-readable files, AI text files or Markdown to appear in Google Search or its AI features, "as Google Search itself doesn't use them". A June 2026 update adds that creating one for other services is fine, but it "will neither harm nor help" your visibility in Google Search. Google's John Mueller has compared it to the old keywords meta tag.
OpenAI, Anthropic and Perplexity: silence
None of the crawler documentation from OpenAI, Anthropic or Perplexity mentions llms.txt. They haven't ruled it out, but no major AI platform has committed to reading it.
The logs: 97% of files got zero requests
The clearest evidence comes from Ahrefs, which analysed llms.txt requests across 137,210 domains in May 2026. Its finding: 97% of llms.txt files received zero traffic that month. Nothing fetched them at all.
Among the 3% that were fetched, most requests came from SEO audit tools, general crawlers and site profilers. AI search and retrieval bots such as OAI-SearchBot and PerplexityBot made up just 1.1% of requests, a couple of hundred fetches in total. And AI bots never asked for llms.txt on sites that don't have one: they don't go looking for it.
Check your own logs
You don't have to take anyone's word for it. If your site has an llms.txt, this shows who has requested it:
# Who fetched llms.txt, by user agent (nginx/Apache combined log format)
grep -h '"GET /llms' /var/log/nginx/access.log* 2>/dev/null \
| awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -20
# cPanel: per-domain logs live under ~/access-logs/
zgrep -h '"GET /llms' ~/access-logs/* | awk -F'"' '{print $6}' | sort | uniq -c | sort -rnWhere llms.txt does help: AI coding assistants
There is one group of tools that uses it: AI coding assistants such as Claude Code and Cursor, and tools built on top of them. In the Ahrefs data, AI agents made 10.5% of llms.txt requests, far more than AI search and assistant bots combined (3.6%), and Claude Code was the most active named AI tool after OpenAI's training crawler. When a developer points one of these tools at your documentation, a clean index of Markdown pages lets it find the right API reference quickly, using far fewer tokens: Cloudflare measured one page at 16,180 tokens as HTML and 3,150 as Markdown, about 80% less.
So the rule of thumb is simple:
- You publish developer docs, an API or a software product: add an llms.txt and Markdown versions of your docs pages. It's a small job, and it makes life easier for developers using AI tools.
- You run a business site, shop or blog: it's harmless, and if your SEO plugin generates it automatically, leave it on. But don't expect any change in AI search traffic.
What actually gets you into AI search
According to Google, to appear as a link in AI Overviews or AI Mode a page simply has to be indexed and eligible for a snippet in normal search. "There are no additional technical requirements." What matters is the same as for good SEO, with a few AI-specific details.
1. Don't block the AI crawlers you want
Each AI company runs several bots, and they do different jobs. Blocking the training bot doesn't remove you from AI search, but blocking the search bot does:
- OAI-SearchBot decides whether you appear in ChatGPT search. GPTBot is for training.
- Claude-SearchBot indexes for Claude's search; ClaudeBot is for training; Claude-User fetches pages when a user asks.
- PerplexityBot powers Perplexity search and isn't used for training.
- Google-Extended isn't a crawler. It's a robots.txt switch for Gemini training and grounding, and it doesn't affect Google Search or AI Overviews. To limit AI Overviews you use
nosnippetormax-snippet. - Applebot-Extended only controls whether Apple can use content for training.
Here's a robots.txt that stays visible in AI search while opting out of model training:
# Allow AI search and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlCheck your firewall and CDN too Google notes that crawling must be allowed in robots.txt and by any CDN or hosting infrastructure. Bot-protection rules on a CDN or security plugin can silently block AI search crawlers even when robots.txt allows them, so check those settings as well.
You can also state your preferences more precisely with Cloudflare's Content Signals, which add a line such as Content-Signal: search=yes, ai-input=yes, ai-train=no to robots.txt. The IETF is working on a standard vocabulary for AI usage preferences, but it's still a draft.
2. Put your content in the HTML
Most AI crawlers don't run JavaScript. A 2024 study by Vercel found that none of the major AI crawlers it measured rendered JavaScript, apart from Google's and Apple's. If your text, prices or product details only appear after scripts run, AI tools may see an empty page. Use server-side rendering, or a CMS like WordPress that outputs the content in HTML. Check what a crawler sees:
# Fetch the raw HTML the way a crawler does, then look for your key text
curl -s -A "OAI-SearchBot/1.0" https://example.com/pricing | grep -i "per month"3. Keep the fundamentals strong
- Clear, specific pages that answer real questions, with headings and lists.
- Structured data that matches what's visible on the page. It's not required for AI search, but helps with rich results.
- A sitemap, good internal links and a fast, reliable site. Google's guidance includes reducing latency as part of good page experience.
- Real expertise and a reputation that other sites cite.
How we handle it at WebGee
We publish an llms.txt on webgee.com, because it's cheap to maintain and helps the AI tools that do read it. Our robots.txt also states our AI preferences with Content Signals. But the work that actually matters is the boring part: pages rendered on the server, fast hosting, clean structured data, and content written by people who run hosting every day.
That's also what our hosting is built for. LiteSpeed web servers with caching keep pages fast for people and crawlers alike, and you have full control over your robots.txt, CDN and security settings, with 24/7 support if you need help reading your logs.
Frequently asked questions
Does llms.txt improve my ranking in Google or AI Overviews?
No. Google states that Google Search, including its AI features, ignores llms.txt, and that having one will neither help nor harm your visibility.
Does ChatGPT read llms.txt?
OpenAI has never said it uses llms.txt, and log data from 2026 shows OpenAI's search crawler almost never requests it. ChatGPT search uses OAI-SearchBot, which you must allow in robots.txt to appear.
Is llms.txt harmful?
No. It's harmless to publish, and useful if you have developer documentation. Just don't expect it to bring AI search traffic.
What is llms-full.txt?
A single file with all of a site's documentation in Markdown. It's a convention from documentation platforms such as Mintlify, not part of the llms.txt specification.
How do I stop AI models training on my content but stay in AI search?
In robots.txt, disallow training bots such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, and allow search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot.
