robots.txt for AI crawlers: invite them by name, and mind the wildcard trap
GPTBot, ClaudeBot, PerplexityBot and Google-Extended each read their own group. How to allow them, and the rule that silently opens pages the wildcard closed.
If you want your pages quoted in assistant answers, the first gate is robots.txt. Most sites either say nothing about AI crawlers or block them all. We invite them by name, and in doing so hit a rule that is easy to get wrong.
Two kinds of AI crawler
| Purpose | Examples |
|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended |
| Search and live answers | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User |
You can treat them differently. We allow both, because for a store selling skills, being cited is worth more than the copy is to withhold. Your answer might differ, and the split lets you decide per purpose.
The trap: a named group ignores the wildcard
A crawler that finds a group naming it uses only that group. Everything under User-agent: * stops applying to it. So this looks safe and is not:
User-agent: *
Disallow: /checkout/
User-agent: GPTBot
Allow: /GPTBot now has no rule about /checkout/ at all. The fix is to repeat the disallows in every named group:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /
Disallow: /checkout/
Disallow: /order/Generate it, do not hand-edit it
We write robots.txt from the build script, next to the sitemap, from one list of crawlers and one list of private paths. A preview build emits Disallow: / for everyone, so a staging URL can never be indexed by accident. Hand-kept files drift; generated ones cannot disagree with the rest of the site.
Then check the edge. A host that blocks AI bots before they reach you makes all of this moot. Pair it with llms.txt and a way to track visits that come from assistants.
Questions
Do I need to name AI crawlers if my robots.txt already allows everyone?
Not strictly, because a crawler with no group of its own falls back to User-agent: *. Naming them makes your intent explicit, lets you treat training and live-answer crawlers differently, and is easy to audit later.
What is Google-Extended?
A separate token Google uses for whether your content may be used for Gemini and Vertex AI, independent of Googlebot. Blocking it does not affect normal Google Search ranking.
Why are AI crawlers blocked even though my robots.txt allows them?
Some CDNs and hosts block known AI bots at the edge before robots.txt is ever read. Check your host's bot settings; on Cloudflare, look at the AI crawler and bot management controls for the zone.