AI Crawlers and robots.txt: Which Bots to Allow or Block

AI crawlers fall into three groups: search bots that index pages so answer engines can cite them, training bots that collect data for future models, and fetchers that visit a page when a user asks. Major vendors give these jobs separate robots.txt names, so you can stay citable while opting out of training.

What are the three kinds of AI crawlers?

• Search bots crawl pages ahead of time and build the index an answer engine searches when someone asks a question. Block these and you can't be cited. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
• Training bots collect content that may be used to train future models. Blocking them is about data use, not today's citations. Examples: GPTBot, ClaudeBot.
• User-triggered fetchers visit a specific page because a user asked an assistant to read it, or because the assistant needed that page to answer. Examples: ChatGPT-User, Claude-User, Perplexity-User.

Then there are the classic search crawlers. Googlebot feeds Google Search, including AI Overviews and AI Mode. Bingbot feeds Bing, which grounds Microsoft Copilot's web answers. Block either one and you lose far more than AI visibility.

Finally, some robots.txt names are tokens, not crawlers. Google-Extended and Applebot-Extended never fetch anything. They tell Google and Apple how they may use content their regular crawlers already collected.

Which AI crawlers should you know about?

These are the names that matter most for answer engines as of late 2026. Match on the token in the first column. Full user-agent strings include version numbers that change.

Vendors add and rename bots, so check each vendor's crawler documentation before you ship a policy. Older names such as anthropic-ai and Claude-Web still linger in plenty of robots.txt files. Anthropic's current documentation lists ClaudeBot, Claude-User and Claude-SearchBot.

Which AI crawlers should most brands allow?

If your goal is to be cited, this is a sensible default. Treat it as a starting point, not a rule:

• Always allow: Googlebot and Bingbot. They power search and the AI features built on top of it.
• Allow if you want citations: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
• Allow if you want assistants to read your pages on request: ChatGPT-User, Claude-User and Perplexity-User. Blocking them can stop an assistant from opening your page when a user's question needs it.
• Decide deliberately: GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended. These are training and data-use choices, covered below.
• Block at the firewall: unidentified scrapers that ignore robots.txt. Robots.txt won't stop them.

How do you allow or block each AI crawler in robots.txt?

Give each bot, or each set of bots, its own User-agent group. Under the robots.txt standard (RFC 9309), a crawler that finds a group naming it follows that group and ignores the * group. That has one big consequence:

Here's an example for a brand that wants to be cited by answer engines but keep its content out of model training. Adjust it to your own decision:

# Answer engines: search bots and user-triggered fetchers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /account/
Disallow: /checkout/

# Model training: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Disallow: /account/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

A few details worth knowing:

• Several User-agent lines can share one group, which keeps the file short.
• Crawlers match tokens case-insensitively, but copy them exactly from vendor docs anyway.
• Robots.txt applies per host. blog.example.com needs its own file.
• Changes aren't instant. OpenAI says about 24 hours, and Google generally caches robots.txt for up to a day.
• Robots.txt is a request, not a lock. Well-behaved crawlers honor it. Scrapers that ignore it need firewall rules.

Should you block AI training bots?

There's no universal right answer, whatever anyone selling certainty tells you. It's a business decision with trade-offs on both sides:

Two facts keep the decision honest. Blocking a training bot today doesn't remove you from models already trained. And it doesn't affect citations from search bots, which you control separately.

Does blocking AI crawlers protect your content?

Partly. It's worth being clear-eyed about what a block does and doesn't achieve:

• It works on bots that comply. The major vendors' declared crawlers say they honor robots.txt. Scrapers that don't identify themselves won't.
• It's forward-looking. Content collected before the block may already sit in training sets and in models that have shipped.
• It doesn't stop the conversation. Blocking search bots keeps your pages from being cited, but engines can still describe you using reviews, articles and forums that quote you. You just lose your own voice in the answer.

What does Google-Extended control, and what doesn't it?

Google-Extended is the most misunderstood name in robots.txt. It isn't a crawler. Googlebot does the fetching, and Google-Extended tells Google whether that content may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI.

What it doesn't do:

• It doesn't affect Google Search. Google says Google-Extended has no impact on inclusion or ranking in Search.
• It doesn't remove you from AI Overviews or AI Mode. Those features are part of Search and use Googlebot's index.

To limit how your content appears in AI Overviews and AI Mode, Google points to its standard controls: nosnippet, data-nosnippet, max-snippet and noindex. They apply to regular results too, so using them to hide from AI features also shrinks or removes your classic snippets.

Is your firewall or CDN blocking AI crawlers?

A perfect robots.txt can still be undone at the network edge. Bot-management products, web application firewalls and hosting platforms can block or challenge AI user agents before robots.txt is ever read. Some CDNs now offer one-click settings to block AI crawlers, and on at least some providers that setting can be on by default for new sites.

How to check:

1. Read your logs. Filter server or CDN logs for each AI token. Are the bots arriving? Are they getting 200 responses, or 403s, 429s and challenge pages?
2. Review bot settings. Look for categories like “AI bots”, “AI scrapers” or “verified bots” in your CDN and firewall dashboards, and check custom rules that match on user agent.
3. Probe with the real user agent. Request a few key pages using the full user-agent string from each vendor's docs and compare the response with a normal browser request.
4. Allow by verification, not just by name. Anyone can fake a user agent. OpenAI and Perplexity publish IP ranges for their bots, and Google supports reverse DNS verification, so allowlist the real thing.

Elite Answers' AEO site audit runs these checks for you: it reads your robots.txt rules per AI bot, requests your pages with AI user agents to catch firewall and CDN blocks, and checks your sitemap and llms.txt.

How often should you review your AI crawler policy?

Whenever something changes, and on a schedule even when nothing seems to. Good triggers for a review:

• A CDN, firewall or hosting change, including plan upgrades that switch on new bot features.
• A site migration, a new subdomain or a robots.txt regenerated by your CMS.
• A vendor announcing a new crawler or renaming an old one.
• A sudden drop in AI citations or AI referral traffic.

Outside those events, a quarterly look at logs and settings catches most drift. For the other file people ask about in the same breath, see our guide to llms.txt.