Robots.txt for AI Crawlers: Copy-Paste GPTBot & ClaudeBot Configs (2026)

Robots.txt for AI Crawlers: The Complete 2026 Guide

Your robots.txt file is the front door to your website for every crawler on the internet. In 2026, a new generation of AI crawlers from OpenAI, Anthropic, Perplexity, and others is knocking on that door. Whether you let them in determines whether your content appears in AI-generated answers, citations, and search results across ChatGPT, Claude, Perplexity, and Google AI Overviews.

This guide covers every AI crawler active in 2026, the exact robots.txt syntax to allow or block each one, and complete configuration templates you can copy into your own site.

TL;DR

If you want AI search visibility, you need to allow AI crawlers in your robots.txt. If you want to protect content from training data collection, you can selectively block training-only crawlers while keeping citation crawlers active.

Quick-reference table:

The bottom line: Most businesses wanting AI citations should allow at minimum OAI-SearchBot, ChatGPT-User, PerplexityBot and Anthropic's search and user agents; allowing the training crawlers (GPTBot, ClaudeBot) as well gives maximum visibility. Blocking the search and user-fetch agents makes your content invisible to the AI platforms that drive the Citation Economy.

Which AI Crawlers Exist in 2026

These are the major AI crawlers scanning the web in 2026 (operators add and rename agents, so check each vendor's documentation before relying on a list). Each serves a different purpose, and understanding the distinction between training crawlers and citation crawlers is critical for making informed robots.txt decisions.

Understanding Each Crawler

GPTBot is OpenAI's training crawler. OpenAI documents the three agents separately: GPTBot gathers content that may be used to train future models, OAI-SearchBot builds the ChatGPT search index, and ChatGPT-User fetches pages live. Blocking GPTBot is an opt-out from training; it does not remove you from ChatGPT search, which is governed by OAI-SearchBot. OpenAI respects robots.txt directives for all three.

ChatGPT-User is the user-agent string sent when ChatGPT browses the web in real time during a conversation. When a user asks ChatGPT to look something up, this crawler fetches the page. Blocking it prevents ChatGPT from accessing your content during live browsing sessions.

OAI-SearchBot powers ChatGPT search. It indexes content specifically for search queries routed through ChatGPT's search mode. Blocking it removes your content from ChatGPT search answers, which makes it the most important OpenAI agent to allow if you want citations.

ClaudeBot is Anthropic's training crawler. Anthropic documents two further agents: Claude-SearchBot, which indexes pages to improve Claude's search results, and Claude-User, which fetches a page when a Claude user's question needs it. Anthropic has stated that its agents respect robots.txt, so you can opt out of training (ClaudeBot) while staying citable (Claude-SearchBot, Claude-User).

PerplexityBot indexes content for Perplexity AI, the search-focused AI platform that provides cited answers to user queries. Perplexity is one of the fastest-growing AI search platforms, and its citations drive measurable referral traffic.

Google-Extended is separate from Googlebot (which powers Google Search). Blocking Google-Extended prevents your content from being used for Gemini AI training but does not affect your Google Search rankings or visibility in Google AI Overviews. This is a common point of confusion.

Bytespider is ByteDance's aggressive crawler, known for high crawl rates. It collects data for ByteDance's AI products. Many site operators block it due to high server load and limited citation benefit.

CCBot powers Common Crawl, a nonprofit that maintains an open web archive. Many AI companies use Common Crawl data for model training. Blocking CCBot reduces your content's presence in training datasets but has no direct citation impact.

FacebookBot crawls content for Meta's AI features, including Meta AI in WhatsApp, Instagram, and Facebook. Blocking it may reduce visibility in Meta's AI-generated answers.

Applebot-Extended is separate from standard Applebot (which powers Safari suggestions and Siri web results). It is not a separate crawler but a robots.txt token that controls whether content Applebot collects may be used to train Apple Intelligence models. Blocking it does not affect standard Apple Search or Siri functionality.

cohere-ai crawls for Cohere, an enterprise AI company. Its impact is primarily on enterprise search products rather than consumer-facing citations.

How to Allow or Block AI Crawlers in Robots.txt

The robots.txt file lives at the root of your domain (https://yoursite.com/robots.txt). It uses a simple syntax of User-agent and Allow/Disallow directives.

Basic Syntax

To allow a specific AI crawler:

User-agent: GPTBot
Allow: /

To block a specific AI crawler:

User-agent: GPTBot
Disallow: /

To allow a crawler but block specific paths (e.g., allow your blog but block private pages):

User-agent: GPTBot
Allow: /blog/
Allow: /press-releases/
Disallow: /admin/
Disallow: /account/
Disallow: /api/

Important Rules

1. Each crawler needs its own User-agent block. You cannot list multiple crawlers under one User-agent directive.
2. More specific paths take precedence. If you Disallow: / but Allow: /blog/, the crawler can access /blog/ but nothing else.
3. Robots.txt is a suggestion, not enforcement. Well-behaved crawlers (GPTBot, ClaudeBot, PerplexityBot) respect it. Malicious crawlers ignore it.
4. Changes take effect on the next crawl. There is no instant cache invalidation. It may take days or weeks for crawlers to re-read your robots.txt.
5. The wildcard * matches all crawlers. A User-agent: * block with Disallow: / blocks everything, including AI crawlers, unless you add specific allow rules for named crawlers above it.

Testing Your Configuration

Verify that AI crawlers can access your site:

curl -A "GPTBot/1.0" https://yoursite.com/robots.txt
curl -A "ClaudeBot/1.0" https://yoursite.com/robots.txt
curl -A "PerplexityBot/1.0" https://yoursite.com/robots.txt

Use Pressonify's AI Visibility Checker to test your robots.txt against all major AI crawlers simultaneously.

Sample Robots.txt Configurations

Below are three complete, copy-paste-ready robots.txt templates. Choose the one that matches your business goals.

Configuration 1: Permissive (Allow All AI Crawlers)

Best for businesses that want maximum AI visibility -- appearing in ChatGPT, Claude, Perplexity, Google AI Overviews, and all other AI search products.

# ============================================
# Robots.txt - Permissive AI Configuration
# Maximum AI visibility and citation potential
# ============================================

# Standard search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# OpenAI crawlers
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Google AI
User-agent: Google-Extended
Allow: /

User-agent: GoogleOther
Allow: /

# Apple Intelligence
User-agent: Applebot-Extended
Allow: /

# Meta AI
User-agent: FacebookBot
Allow: /

# Other AI crawlers
User-agent: Bytespider
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

# Default rule for all other crawlers
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /account/

# Sitemap
Sitemap: https://yoursite.com/sitemap.xml

Configuration 2: Selective (Allow Citation Crawlers, Block Training-Only)

Best for businesses that want AI citations without contributing to training datasets. This allows crawlers that power real-time search and citation features while blocking crawlers used primarily for model training.

# ============================================
# Robots.txt - Selective AI Configuration
# Allow citations, block training-only crawlers
# ============================================

# Standard search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# OpenAI - allow search/citation crawlers
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# OpenAI - block training crawler
User-agent: GPTBot
Disallow: /

# Anthropic - allow search and user fetches, block training
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

# Perplexity - allow for citations
User-agent: PerplexityBot
Allow: /

# Google AI training - block
User-agent: Google-Extended
Disallow: /

User-agent: GoogleOther
Disallow: /

# Apple Intelligence training - block
User-agent: Applebot-Extended
Disallow: /

# Meta AI - allow
User-agent: FacebookBot
Allow: /

# Training-only crawlers - block
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

# Default
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /account/

# Sitemap
Sitemap: https://yoursite.com/sitemap.xml

Important caveat: Content that is already in a model's training data stays there, and a training opt-out does not change what a model already "knows" about you. Blocking training crawlers affects future models only. Monitor your AI citation metrics after making any change.

Configuration 3: Restrictive (Block All AI Crawlers)

For businesses that want to prevent all AI crawling -- typically used by publishers concerned about content licensing, or sites with proprietary content.

# ============================================
# Robots.txt - Restrictive AI Configuration
# Block all AI crawlers
# ============================================

# Standard search engines - still allowed
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Block all AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: cohere-ai
Disallow: /

# Default
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /account/

# Sitemap
Sitemap: https://yoursite.com/sitemap.xml

Be aware: This configuration makes your content invisible to the AI search platforms that honor robots.txt. You should expect no citations from ChatGPT, Claude or Perplexity. Traditional Google and Bing search results are unaffected, and Google AI Overviews are governed by Googlebot, not Google-Extended.

Robots.txt vs ai.txt vs llms.txt -- Which Do You Need?

Three files now govern how AI systems interact with your website. They serve different purposes and are complementary, not interchangeable.

How They Work Together

Think of it as a three-layer system:

1. robots.txt is the gatekeeper. It determines which AI crawlers can access your site at all.
2. llms.txt is the tour guide. Once a crawler is allowed in, llms.txt tells it what your site is about, which pages matter most, and how content is organized.
3. ai.txt is the policy document. It communicates your terms for how AI systems may use the content they find.

If you only implement one, make it robots.txt with AI crawler permissions. If you implement two, add llms.txt. For the full stack, add ai.txt as well.

For a detailed guide on llms.txt, see our llms.txt pillar page. For the complete AI discoverability framework, see the AI Discovery Protocol documentation.

How AI Crawlers Differ from Search Engine Crawlers

AI crawlers and traditional search engine crawlers (Googlebot, Bingbot) behave differently in several important ways.

Crawl Frequency and Depth

Traditional search engine crawlers maintain persistent indexes and re-crawl pages on regular schedules based on crawl budgets. AI crawlers tend to be more aggressive in initial crawling but less predictable in re-crawl timing. Some AI crawlers (like ChatGPT-User) fetch pages on demand in real time rather than maintaining a pre-built index.

Content Extraction

Googlebot primarily processes HTML structure, metadata, and links for ranking purposes. AI crawlers extract full text content for semantic understanding. They parse your content the way a human reader would -- paragraph by paragraph -- looking for facts, claims, statistics, and quotable statements that can be cited in AI-generated answers.

Respect for Robots.txt

OpenAI, Anthropic, Perplexity and Google have all publicly committed to honoring robots.txt for their named agents. Compliance has not always been perfect in practice, and robots.txt remains a request rather than an enforcement mechanism, so server-level controls are the backstop if you need a hard block.

Impact of Blocking

Blocking Googlebot removes you from Google Search entirely. Blocking an AI crawler removes you from that specific AI platform's knowledge base and citation pool. The difference is scope: blocking Googlebot affects billions of daily searches; blocking ClaudeBot affects Claude users specifically.

Crawl Rate

Some AI crawlers, particularly Bytespider, are known for aggressive crawl rates that can strain server resources. If you notice performance issues, you can use Crawl-delay directives (supported by some crawlers) or rate-limiting at the server level.

Common Mistakes

1. Accidentally Blocking AI Crawlers with Wildcard Rules

Many legacy robots.txt files include a broad wildcard block:

User-agent: *
Disallow: /

This blocks every crawler, including AI crawlers, unless you add named groups for them. A crawler obeys the most specific User-agent group that matches it, so explicit groups for each AI crawler you want to permit override the wildcard (order in the file does not matter, but listing them first keeps it readable).

Fix:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /

2. Blocking the Wrong OpenAI Agent

The most common mistake runs the other way: sites that want to opt out of training block OAI-SearchBot or ChatGPT-User along with GPTBot, and disappear from ChatGPT search. GPTBot governs training; OAI-SearchBot governs ChatGPT search; ChatGPT-User governs live fetches. Block only what you mean to block. If you want to see which agents actually visit, our log of 108,808 AI-agent visits shows ChatGPT-User as the dominant answer-time fetcher, and an AI crawler audit will show the same for your site.

3. Not Having a Robots.txt at All

If your site has no robots.txt file, AI crawlers will crawl everything by default. While this is technically permissive, it also means you have no control over which pages are crawled. Private pages, admin panels, staging content, and duplicate pages may all be indexed. Always have a robots.txt file, even if your policy is permissive.

4. Using Robots Meta Tags Instead of Robots.txt

HTML meta tags like <meta name="robots" content="noindex"> affect search engines but are not consistently respected by all AI crawlers. For reliable AI crawler control, always use robots.txt as the primary mechanism and treat meta tags as a secondary layer.

5. Forgetting to Include a Sitemap Reference

Your robots.txt should include a Sitemap: directive pointing to your XML sitemap. AI crawlers use sitemaps to discover content efficiently. Without it, crawlers must discover pages by following links, which is slower and may miss orphaned pages.

Sitemap: https://yoursite.com/sitemap.xml

Frequently Asked Questions

How do I allow AI crawlers in robots.txt?

Add a User-agent directive for each AI crawler you want to allow, followed by Allow: /. Each crawler needs its own block. For example:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Place these directives in your robots.txt file at the root of your domain. There is no registration or approval process -- once the directives are in place, crawlers will detect them on their next visit.

Should I block AI crawlers?

It depends entirely on your business goals.

Allow AI crawlers if you want your content cited by ChatGPT, Claude, Perplexity, and other AI platforms. AI citations drive qualified traffic and brand authority in the Citation Economy. For most businesses -- especially those publishing press releases, thought leadership, or product content -- allowing AI crawlers is strongly recommended.

Block AI crawlers if you are a publisher concerned about content being used for AI model training without compensation, or if you have proprietary content that should not be indexed outside traditional search engines. Even in this case, consider a selective approach: block training-only crawlers while allowing citation crawlers.

What AI crawlers exist in 2026?

The major AI crawlers active in 2026 are: GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot (Perplexity AI); Google-Extended and GoogleOther (Google); Bytespider (ByteDance); CCBot (Common Crawl); FacebookBot (Meta); Applebot-Extended (Apple); and cohere-ai (Cohere). See the full comparison table above for details on each crawler's purpose and citation impact.

What is the difference between robots.txt, ai.txt, and llms.txt?

Robots.txt is the access control layer. It tells crawlers which pages they are permitted to visit. It has been the web standard for crawler management since 1994 and is supported by all major AI crawlers.

Llms.txt is the context layer. It provides AI systems with a structured summary of your site: what you do, which pages are most important, and how your content is organized. It helps AI systems understand your site holistically rather than page by page.

Ai.txt is the policy layer. It declares how AI systems may use your content -- whether it can be used for training, whether attribution is required, and any licensing terms. It is a proposed standard that has not yet achieved wide adoption.

All three are complementary. Robots.txt is essential. Llms.txt is strongly recommended. Ai.txt is optional but forward-looking.

Does blocking GPTBot prevent ChatGPT from citing my content?

Not by itself. OpenAI documents GPTBot as its training crawler: blocking it tells OpenAI not to use your content to train future models. ChatGPT search answers are built from the index that OAI-SearchBot maintains, and ChatGPT-User fetches pages live when a user's question calls for it.

So the practical rule is: to stay citable in ChatGPT while opting out of training, block GPTBot and allow OAI-SearchBot and ChatGPT-User. Blocking OAI-SearchBot is what removes you from ChatGPT search.

Conclusion

Your robots.txt file is no longer just about managing Googlebot. In 2026, it is the primary control mechanism for AI crawler access and directly determines whether your content appears in AI-generated citations across ChatGPT, Claude, Perplexity, and other platforms.

For most businesses, the recommended approach is:

1. Allow the search and live-fetch agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) for citation visibility.
2. Make a deliberate decision about training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider) based on your content licensing stance.
3. Add a Sitemap directive to help crawlers discover your content efficiently.
4. Complement robots.txt with an llms.txt file for richer AI context.
5. Test your configuration regularly using the AI Visibility Checker.

The businesses winning in the Citation Economy are the ones making deliberate, informed decisions about AI crawler access. Your robots.txt is where that starts.

Related Resources

• Robots.txt for AI Crawlers -- Quick configuration reference and copy-paste checklist
• AI Visibility Checker -- Test your site's AI crawler accessibility
• How to Get Cited by ChatGPT: The Complete Guide -- Full citation optimization walkthrough
• Does Google Support IndexNow? 2026 Status Guide -- Instant indexing for AI search