robots.txt for AI Crawlers: GPTBot & ClaudeBot

Which AI crawlers exist, how to allow or block them in robots.txt, and the Google-Extended misconception that costs sites AI Overview citations.

Most sites have never made a decision about AI crawlers. They inherited one — from a copied robots.txt, a security plugin, or a CDN preset that blocks unfamiliar user agents by default. This is a guide to making it a choice.

The crawlers worth knowing about

User-agent tokens change, so treat this as a starting point and check each vendor's current documentation before relying on it:

• OpenAI runs separate agents for separate jobs: one for model training, one for building its search index, and one for fetching a page because a user asked about it right now. Blocking training while allowing search is a coherent, common position — and it's only possible because they're distinct tokens.
• Anthropic (ClaudeBot and related agents) similarly separates crawling for training from user-triggered fetches.
• Perplexity (PerplexityBot, plus a user-triggered agent) crawls to build the index it cites from.
• Google-Extended is not a crawler. It's a robots.txt token that controls whether content Googlebot already fetched may be used for Gemini grounding and model training.
• CCBot (Common Crawl) feeds a public dataset that many models train on indirectly.
• Applebot-Extended, meta-externalagent, Amazonbot and Bytespider cover Apple, Meta, Amazon and ByteDance respectively.

The Google-Extended misconception

This one costs people citations, so it's worth stating plainly: blocking Google-Extended does not remove you from AI Overviews.

AI Overviews are built on Google's ordinary search index, crawled by Googlebot. Google-Extended governs Gemini grounding and training use. The only way to keep yourself out of AI Overviews is to keep yourself out of Google Search — which is a much larger decision than most people blocking that token think they're making.

So if your goal was "don't train on me, but do cite me", blocking Google-Extended is roughly right. If your goal was "keep me out of AI answers entirely", it doesn't achieve that, and nothing short of leaving the index will.

How to allow them

If you want to be cited, the simplest robots.txt is one that doesn't single any of them out:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Then confirm nothing else is blocking them. robots.txt is advisory and it isn't the only gate — a WAF rule, a Cloudflare bot-fight setting, a rate limiter or a plugin can all reject an AI crawler while your robots.txt cheerfully says yes. Check your edge configuration, not just the file.

How to block them

A blanket block, if that's your position:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

Two caveats. robots.txt is a convention, not enforcement — well-behaved crawlers honour it and badly-behaved ones don't, so if you need a guarantee you need authentication or edge rules. And a total block removes you from consideration as a cited source in those systems, which for most commercial sites is the opposite of what they want.

The middle position

Most publishers who've thought about it land in the same place: allow the search and user-triggered agents, block the training ones. You stay citable and keep the referral traffic, while opting out of bulk training use. Because the vendors split these into separate tokens, it's expressible — look up each vendor's current agent names and write the rules accordingly. It's worth ten minutes of checking the primary sources rather than copying a block from a blog post that's a year stale.

Check what you're actually doing today

Read your own /robots.txt first — the number of sites blocking AI crawlers entirely by accident is genuinely high. Then check the rest of the picture: our AI visibility checker looks at crawler access alongside whether your content is extractable once a crawler does get in, since being allowed through and being quotable are two different problems.