What Is llms.txt and Do You Need One?
A one-file proposal for telling language models what your site is about. Cheap to add, genuinely useful — and widely misunderstood.
Should you block GPTBot? Check what your CDN already did
There is a decision worth making deliberately: whether AI crawlers may read your site. What happens in practice is that a CDN makes it for you, silently, and you discover months later that you have been invisible to AI search the entire time.
This happened to us on this site, and the diagnosis is worth walking through because the failure has two independent layers and fixing one does nothing for the other.
Layer one: the robots.txt you did not write
Our robots.txt explicitly allowed GPTBot, ClaudeBot, PerplexityBot and the rest. An audit still reported that AI crawlers were blocked. Fetching the file settled it in seconds:
curl -s https://yourdomain.com/robots.txt | head -40
What came back was not the file in our repository. Cloudflare had prepended its own block — a Content Signals policy plus `Disallow: /` for GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot, Applebot-Extended, meta-externalagent and Bytespider — with our file appended underneath. The served file contradicted itself, and the crawlers read the first match.
This is Cloudflare's managed robots.txt, part of AI Crawl Control. Cloudflare began blocking AI crawlers by default for new domains in 2025, so most site owners in this situation never switched it on. It is off by turning off "Set your preference to block training in robots.txt" under Security → Settings → Bot traffic.
Layer two: the 403 that robots.txt cannot fix
We turned the managed robots.txt off. The file came back clean. The audit still complained — because robots.txt is a request, and the firewall is a lock. Testing what crawlers actually receive is one line:
curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://yourdomain.com/
Run that for each crawler you care about. Ours returned this, with a clean, permissive robots.txt in place:
| Crawler | What it does | Response |
|---|---|---|
| Googlebot | Classic search index | 200 |
| GPTBot | OpenAI training | 403 |
| OAI-SearchBot | ChatGPT search retrieval | 403 |
| ClaudeBot | Anthropic training | 403 |
| Claude-User | Live fetch during a chat | 403 |
| PerplexityBot | Perplexity index | 403 |
| CCBot | Common Crawl | 403 |
Googlebot sailed through, so this was not blanket bot protection — it was the AI category specifically, enforced at the edge by a WAF rule. Per-crawler control lives in the AI Crawl Control dashboard, where each bot can be set to allow or block.
Why the OAI-SearchBot row is the alarming one
It is tempting to read "AI crawlers blocked" as "our content will not be used for training", shrug, and move on. That is not what those rows say.
The crawlers split into two groups that get lumped together and should not be. GPTBot, ClaudeBot, CCBot and meta-externalagent build training corpora — blocking them is a defensible business decision. But OAI-SearchBot, Claude-User and PerplexityBot are retrieval crawlers: they fetch your page at the moment a person asks a question, so the answer can cite you.
Blocking those does not protect anything. It removes you from the answer while your competitors are named in it. That is the difference between declining to be training data and declining to be a search result.
How to decide
There is a real trade here, and it depends on what your content is for.
- If your content is the product — a publisher, a course, a research archive — blocking training crawlers protects a genuine asset. Keep the retrieval crawlers.
- If your content exists to make you findable, as a services company's writing does, blocking any of them works against you. Being in the training set is how you get recalled unprompted.
- Retrieval crawlers (OAI-SearchBot, Claude-User, PerplexityBot, ChatGPT-User) should be allowed in almost every case. They are how citations happen.
- Google-Extended controls Gemini grounding, not your normal Google ranking. Blocking it does not remove you from Google Search or AI Overviews, which use the standard index.
Whatever you decide, decide it. The failure mode is not choosing to block — it is discovering a default you never set.
A five-minute audit
- Fetch your live robots.txt with curl and read all of it. Compare it to the file in your repository.
- Request your homepage with each AI crawler's user agent and record the status code. Anything other than 200 is a block.
- Check your CDN's bot settings for an AI-crawler toggle you did not enable.
- Confirm your content is in the raw HTML, not injected after hydration — many AI crawlers run little or no JavaScript.
- Re-run the crawler test after every CDN or WAF change. These defaults change without notice.
Steps one and two take five minutes and are the highest-value SEO check most sites are not running. If the results surprise you, our AI-search SEO work starts exactly here, and what belongs in llms.txt covers what to do once crawlers can actually reach you.
Frequently asked questions
Should I block GPTBot?
Only if your content is itself the product you sell, such as a publication or a paid archive. If your writing exists to make your business findable, blocking GPTBot removes you from the training data that lets models recall you unprompted, with no offsetting benefit. Either way, allow the retrieval crawlers like OAI-SearchBot and PerplexityBot, which fetch pages so answers can cite them.
Why are AI crawlers blocked when my robots.txt allows them?
Two common causes. Your CDN may prepend its own managed robots.txt above yours, so the served file contradicts the one in your repository; and a firewall rule may return 403 to AI crawler user agents regardless of what robots.txt says. Check both by fetching your live robots.txt with curl and by requesting your homepage with each crawler's user agent.
How do I test whether GPTBot can reach my site?
Send a request with the crawler's user agent and read the status code: curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://yourdomain.com/. A 200 means it can read the page. A 403 means something at your edge is blocking it, which robots.txt cannot override. Repeat for OAI-SearchBot, ClaudeBot, Claude-User and PerplexityBot.
Does blocking Google-Extended remove me from Google?
No. Google-Extended controls whether your content is used for Gemini training and grounding. Google Search ranking and AI Overviews both draw on the standard Googlebot index, which is governed separately. Blocking Google-Extended does not remove you from search results or from AI Overviews.
References
- AI Crawl Control — Cloudflare Docs
- Block AI bots, scrapers and crawlers with a single click — Cloudflare Blog
- OpenAI bots and user agents — OpenAI
- RFC 9309: Robots Exclusion Protocol — IETF
Keep reading
Generative Engine Optimisation (GEO)
Ranking first is worth less when the answer appears above the results. Being the source that answer cites is the new position one.
RAG vs Fine-Tuning: How to Choose
Teams reach for fine-tuning when they want the model to know something. That is the one job fine-tuning is worst at.