robots.txt for AI Crawlers: Quick Configuration Reference

Why robots.txt Matters for AI Discovery

Your robots.txt file is the first thing automated AI crawlers check before accessing your site. A misconfigured robots.txt can make your entire website invisible to AI systems, no matter how well-optimized your content is.

How AI Crawlers Use robots.txt

Unlike traditional search engine crawlers that primarily index for search results, AI crawlers serve multiple purposes:

• Training Data: Content for model training and updates
• RAG Systems: Real-time retrieval for answering user queries
• Citation Sources: Content to cite when generating responses
• Knowledge Graphs: Entity and relationship extraction

If you block AI search crawlers, you're opting out of AI-powered search and citation.

This page is the quick configuration reference. For the full crawler-by-crawler research and deeper strategy, read the complete robots.txt for AI crawlers guide.

The Major AI Crawlers You Need to Know

Here are the key AI crawlers and their purposes:

Key Insight: If you want AI citation, allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot at minimum (Google AI Overviews use Googlebot; Google-Extended governs Gemini apps and Vertex AI). Every release on the Pressonify platform is published on pages that allow all of these crawlers.

Recommended robots.txt Configuration

Here's the recommended robots.txt configuration for maximum AI discoverability:

# =============================================
# robots.txt - AI-Optimized Configuration
# =============================================
# A crawler obeys only the most specific group that names it, so the
# named crawlers share ONE group with the default rule. That way the
# protected paths below apply to every one of them.

# Search engines
User-agent: Googlebot
User-agent: Bingbot

# OpenAI (ChatGPT): search, user fetches, training
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot

# Anthropic (Claude): search, user fetches, training
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot

# Perplexity
User-agent: PerplexityBot
User-agent: Perplexity-User

# AI-use tokens (Gemini, Apple Intelligence) and others
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: CCBot

# Everyone else
User-agent: *
Allow: /

# PROTECTED PATHS (adjust for your site)
Disallow: /admin/
Disallow: /api/internal/
Disallow: /private/

# =============================================
# SITEMAPS
# =============================================
Sitemap: https://yoursite.com/sitemap.xml

Replace yoursite.com with your actual domain. A crawler follows only the most specific group that names it, which is why every crawler here shares one group: give GPTBot its own Allow: / group and it will ignore the Disallow lines under *. To opt out of training, move GPTBot, ClaudeBot, Google-Extended and Applebot-Extended into a separate group with Disallow: /.

Common robots.txt Mistakes That Block AI

These are the most common mistakes that accidentally block AI crawlers:

Mistake 1: Blanket Disallow Rules

# BAD: Blocks ALL crawlers including AI
User-agent: *
Disallow: /

This blocks everything. If you have this, remove it immediately or add specific Allow rules for AI crawlers.

Mistake 2: No AI-Specific Rules

# INCOMPLETE: Only allows Googlebot
User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

This allows Google but blocks GPTBot, ClaudeBot, etc. Always add explicit rules for AI crawlers.

Mistake 3: Blocking Based on Outdated Advice

Some SEO guides from 2023 recommended blocking AI crawlers to "protect content." This advice is outdated: blocking AI crawlers now means missing the Citation Economy.

Mistake 4: Forgetting Multiple OpenAI Crawlers

# INCOMPLETE: Only allows GPTBot
User-agent: GPTBot
Allow: /

OpenAI uses separate agents: GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (user requests). Allow OAI-SearchBot and ChatGPT-User, and decide on GPTBot.

Mistake 5: Blocking the Search Crawler Instead of the Training Crawler

# WRONG: Removes you from ChatGPT search citations
User-agent: OAI-SearchBot
Disallow: /

If your goal is to opt out of AI training, block GPTBot, not OAI-SearchBot. Blocking the search crawler removes you from ChatGPT's cited sources.

Testing and Verifying Your Configuration

After updating robots.txt, verify it's working correctly:

1. Google Search Console robots.txt Report

Use the robots.txt report in Google Search Console to confirm Google can fetch and parse your file.

2. Manual Testing

Visit your robots.txt directly: https://yoursite.com/robots.txt

Verify all AI crawler rules are present and correctly formatted.

3. Pressonify AI Visibility Checker

Our AI Visibility Checker analyzes your robots.txt and reports which AI crawlers are allowed or blocked.

4. Check AI Crawler Logs

Monitor your server logs for these User-Agent strings:

• GPTBot (e.g. GPTBot/1.4; +https://openai.com/gptbot)
• ClaudeBot ([email protected])
• PerplexityBot/1.0

If you're not seeing these crawlers, check your robots.txt for blocking rules.

5. Use the Agentic Audit Tool

Our Agentic Audit tool checks robots.txt as part of comprehensive AI readiness scoring.

When to Selectively Block AI Crawlers

While we recommend allowing AI crawlers, there are valid reasons to block specific ones:

Valid Reasons to Block

• Paywalled Content: Premium content behind subscriptions
• Proprietary Data: Trade secrets or confidential information
• Legal Requirements: GDPR or copyright concerns
• Training Opt-Out: Block training but allow citation (supported by OpenAI, Anthropic and Google)

Selective Blocking Example

# Allow citation-focused crawlers
User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Block training-only crawlers for specific paths
User-agent: GPTBot
Allow: /blog/
Allow: /news/
Disallow: /premium/
Disallow: /members-only/

The Trade-Off

Remember: blocking AI crawlers means opting out of AI-powered discovery. For most businesses seeking visibility, the benefits of allowing crawlers outweigh the risks.

Integration with llms.txt and ADP

robots.txt is just one piece of AI discoverability. For complete optimization, combine with:

robots.txt + llms.txt

While robots.txt tells crawlers what to access, llms.txt tells them how to understand your site:

• robots.txt: Access permissions (Allow/Disallow)
• llms.txt: Site context, key pages, topic authority

The ADP Triumvirate

The AI Discovery Protocol pairs three complementary files:

1. robots.txt: What AI can crawl
2. llms.txt: How AI should understand your site
3. /.well-known/ai.json: Discovery manifest with all endpoints

Example ai.json Reference

{
  "version": "3.0",
  "endpoints": {
    "robots": "/robots.txt",
    "llms": "/llms.txt",
    "llms_full": "/llms-full.txt",
    "sitemap": "/sitemap.xml",
    "feed": "/feed.json"
  },
  "ai_crawlers": {
    "allowed": ["GPTBot", "ClaudeBot", "PerplexityBot"],
    "blocked": []
  }
}