Understanding AI crawler directives in robots.txt has become a non-negotiable skill for anyone managing web properties in 2026 — yet most implementations are either incomplete, inconsistent, or accidentally blocking the wrong bots entirely. This guide walks through every major AI user-agent token, what each one controls, and the exact syntax to deploy directives that actually do what you intend.

Understanding AI Crawler Directives in Robots.txt

Robots.txt has existed for decades as a simple crawl management tool, but the emergence of large language model training pipelines has transformed it into a strategic content-rights instrument. AI crawler directives in robots.txt now determine whether your content feeds into training datasets, retrieval-augmented generation (RAG) systems, or neither — decisions with meaningful commercial and legal implications.

The core mechanism remains unchanged: a plain-text file at the root of your domain lists user-agent strings paired with Allow or Disallow rules. What has changed is the number of distinct AI crawlers that need to be addressed. Where publishers once managed a handful of well-known bots, they now face a growing list of AI-specific agents — each with different purposes, different operators, and different levels of compliance with the standard.

"An analysis of large publishers' robots.txt practices found that 60 of 70 publishers explicitly restricted at least one AI crawler as of July 14, 2026 — signaling that selective blocking, not blanket restriction, has become the dominant strategy."

Before you write a single directive, you need a clear framework: which crawlers you are addressing, what each one does, and what outcome you want for each one. The following sections build that framework step by step.

Prerequisites: You need FTP, SSH, or CMS-level access to modify the robots.txt file at your domain root. You should also have server log access or a crawler-monitoring tool so you can verify bot behavior after deployment. Familiarity with basic robots.txt syntax (user-agent blocks, Allow/Disallow paths) is assumed — if you are starting from scratch, the broader AI robots.txt training controls for publishers implementation guide covers foundational syntax before diving into specific bots.

Map Every AI Bot to Its User-Agent Token

The first concrete step is building an accurate map of AI user-agent strings. Using an incorrect or outdated token means your directive is silently ignored — the crawler simply does not recognize itself as the target. The table below reflects tokens confirmed by their respective operators as of late 2026.

Crawler Name Operator User-Agent Token Primary Purpose
GPTBot OpenAI GPTBot LLM training data collection
ChatGPT-User OpenAI ChatGPT-User Real-time retrieval for ChatGPT browse/plugins
OAI-SearchBot OpenAI OAI-SearchBot SearchGPT / ChatGPT Search index
Google-Extended Google Google-Extended Gemini and Vertex AI training
Googlebot Google Googlebot Search index + AI Overviews retrieval
ClaudeBot Anthropic ClaudeBot Training data and Claude.ai retrieval
anthropic-ai Anthropic anthropic-ai Supplemental Anthropic crawler
PerplexityBot Perplexity AI PerplexityBot Perplexity answer index and citations
cohere-ai Cohere cohere-ai Enterprise LLM training
Bytespider ByteDance Bytespider TikTok and ByteDance AI products
Meta-ExternalAgent Meta Meta-ExternalAgent Meta AI training and products
Applebot-Extended Apple Applebot-Extended Apple Intelligence training

Key actions for this step:

  • Cross-reference each token against the operator's official documentation page — user-agent strings do get revised, and an outdated token does nothing.
  • Note that OpenAI operates three distinct crawlers for different functions; blocking GPTBot alone does not block ChatGPT's live browsing agent (ChatGPT-User) or its search indexer (OAI-SearchBot).
  • Identify which crawlers your server logs have already recorded — this confirms which bots are actively hitting your domain and prioritizes your effort.
  • For a detailed look at how OpenAI's crawlers differ from each other, see the dedicated GPTBot robots.txt controls guide, which covers training versus retrieval separation in full.

Write Precise Directives for Each Crawler

Once you have confirmed user-agent tokens, write explicit directives for each one. Ambiguity in robots.txt is costly — a missing block means assumed permission, while an overly broad block may cut off crawlers you actually want. The right approach is intentional, per-bot configuration.

Key actions for this step:

  • Use separate user-agent blocks for each AI crawler. Combining multiple agents in one block is syntactically allowed but makes future editing error-prone and harder to audit.
  • Distinguish training from retrieval. For OpenAI, you may want to block GPTBot (training) while permitting OAI-SearchBot (search index citations). These require two separate blocks with opposite rules.
  • Use Disallow: / for a full site block and Disallow: (empty value) or explicit Allow: / for full access — do not leave a block with no Disallow rule and assume it means anything.
  • Handle Google carefully. Blocking Googlebot removes you from search and AI Overviews. Google-Extended is the correct token to block Gemini training without touching search rankings — the Google-Extended AI training opt out guide covers this distinction and its implications for AI Overview citations in depth.
  • Address Anthropic's dual tokens. Some operators report that Anthropic has used both ClaudeBot and anthropic-ai; block both to be comprehensive.

A minimal but complete example for a publisher blocking all AI training while permitting search indexers:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

This configuration blocks training crawlers while keeping retrieval and citation crawlers active — a pattern that has become standard among news and content publishers seeking citation visibility without surrendering training rights.

Implement Crawl-Rate and Path-Level Controls

Beyond binary Allow/Disallow decisions, robots.txt supports more granular controls that are particularly useful for large sites with varied content types. Not all of your content may warrant the same treatment — editorial archives, product pages, and gated resources each have different strategic value in the AI context.

Key actions for this step:

  • Use path-level Disallow to protect specific directories. If you want to allow an AI retrieval crawler to cite your public articles but block access to your subscriber-only archive, specify Disallow: /members/ within that bot's block rather than a blanket site ban.
  • Use Crawl-delay to manage server load from compliant crawlers. While not universally honored, adding Crawl-delay: 10 (seconds between requests) to blocks for well-behaved bots reduces resource pressure. Do not rely on this as a security measure — it is advisory only.
  • Protect specific file types with wildcard patterns. Use Disallow: /*.pdf$ to block AI crawlers from indexing downloadable reports or research documents you have not licensed for training use.
  • Separate /api/ and /wp-json/ paths. AI crawlers hitting REST API endpoints can expose structured data in ways robots.txt at the page level does not cover. Explicitly Disallow API paths from AI training agents.
  • Consider sitemap references strategically. Including a Sitemap directive in a block you want retrieval crawlers to follow helps them find content efficiently — but omit it from blocks where you want minimal engagement beyond the Disallow signal.

Industry practitioners increasingly report that path-level differentiation — rather than all-or-nothing blocks — delivers better outcomes: training crawlers are excluded from proprietary content, while citation-generating retrieval crawlers can still surface that content in AI-powered answers and drive referral traffic.

Audit and Validate Your Directives

Writing the directives is only half the work. Validating that they are syntactically correct, properly deployed, and actually being honored by the crawlers they target is what transforms intent into enforced policy.

Key actions for this step:

  • Use Google Search Console's robots.txt tester to check syntax and confirm that specific URLs are blocked or allowed as intended for Googlebot and Google-Extended.
  • Fetch your live robots.txt via curl (curl -I https://yourdomain.com/robots.txt) to confirm the file is returning a 200 status and is not cached at a CDN layer with stale content.
  • Review server access logs for each AI user-agent string. Seeing continued crawl activity from an agent you have blocked is a signal that either the token is wrong, the file is not being served correctly, or — in the case of less compliant crawlers — the directive is being ignored outright.
  • Run periodic re-audits every 60–90 days. New AI crawlers emerge regularly, operator documentation updates token strings, and your own site structure may change in ways that affect path-level rules. Static robots.txt files decay in accuracy over time.
  • Cross-reference against published compliance data. The AI crawler robots.txt compliance benchmark tracks which bots actually honor directives in 2026 — a critical read before deciding how much weight to put on robots.txt versus technical access controls like authentication or IP blocking.

"A 2026 audit of news publishers found that 56.4% blocked at least one AI crawler in robots.txt — but changes in CDN configurations silently removed those blocks from dozens of sites, underscoring why deployment validation is as important as the directives themselves."

Common Mistakes to Avoid

The most expensive errors in AI crawler management are not dramatic — they are quiet misconfigurations that go unnoticed for months while crawlers behave contrary to your intent.

  • Blocking Googlebot instead of Google-Extended. This is the single most consequential mistake: it removes your site from Google Search entirely while doing nothing to restrict Gemini training. These are two completely separate tokens requiring two separate blocks.
  • Assuming one OpenAI block covers all OpenAI activity. GPTBot, ChatGPT-User, and OAI-SearchBot are distinct agents with different purposes. A publisher blocking only GPTBot for training may inadvertently permit or restrict more than intended by not explicitly addressing the other two.
  • Using deprecated or incorrect token names. Some older guides reference tokens that have since been renamed or supplemented. Always verify against the operator's current documentation before deployment.
  • Placing robots.txt at a non-root path. The file must live at https://yourdomain.com/robots.txt — not in a subdirectory. Subdomain properties (e.g., blog.yourdomain.com) each require their own robots.txt file.
  • Relying solely on robots.txt for access control. Robots.txt is an advisory protocol — compliant crawlers respect it, but it provides no technical enforcement. For content that must not be accessible, authentication, server-side access controls, or IP-level filtering are necessary complements.
  • Setting and forgetting. A robots.txt file written in early 2025 is likely missing several crawlers that emerged since then. Treat it as a living document, not a one-time configuration.

Expected Results and Timeline

Realistic expectations help you evaluate whether your implementation is working and avoid mistaking normal crawl behavior for a failure of your directives.

  • Compliant crawlers typically re-fetch robots.txt every 24 hours, so changes propagate within one to two days for well-behaved agents. Some operators cache the file for longer — up to a week — so allow a full week before concluding a directive change has been ignored.
  • Server log reduction for blocked agents should be visible within days. If a blocked crawler's log entries continue at the same volume after two weeks, investigate whether the token is correct and whether the file is being served properly.
  • Citation visibility in AI answers (from permitted retrieval crawlers) typically takes two to four weeks to reflect changes, as retrieval indexes are rebuilt on their own schedules.
  • No impact on Google Search rankings from Google-Extended blocks — this is a training-only signal and does not affect Googlebot's indexing or ranking decisions.
  • Non-compliant crawlers will not change behavior. Industry data consistently shows that a subset of AI crawlers do not honor robots.txt directives regardless of configuration. For these, server-level controls are the only effective option — your robots.txt work still matters for the majority that do comply.

Frequently Asked Questions

What is the correct robots.txt syntax to block all AI crawlers at once?

There is no single wildcard token that reliably targets all AI crawlers as a group — you need individual user-agent blocks for each one you want to restrict. The closest approximation is using User-agent: * with a Disallow: / directive, but this blocks every crawler including search engines, which is almost never the intended outcome. The correct approach is to list each AI user-agent token explicitly in its own block with the appropriate rule, as shown in the directive examples in this guide.

Does blocking GPTBot prevent my content from appearing in ChatGPT answers?

Blocking GPTBot prevents OpenAI from using your content for future training data collection, but it does not remove content that was already crawled before the block was in place. It also does not affect ChatGPT-User, which handles real-time web browsing, or OAI-SearchBot, which powers ChatGPT Search. To restrict all OpenAI access, you need separate Disallow directives for all three tokens.

Will blocking AI crawlers in robots.txt hurt my SEO rankings?

Blocking AI training crawlers like GPTBot, Google-Extended, ClaudeBot, and similar agents has no effect on your traditional search rankings because these are separate from search indexing bots. The critical rule is to never block Googlebot itself, as that controls both search indexing and AI Overviews retrieval. Blocking Google-Extended specifically restricts Gemini model training without touching search visibility.

Do all AI crawlers actually respect robots.txt directives?

No — compliance varies significantly across operators. Major AI companies like OpenAI, Google, Anthropic, and Perplexity have publicly committed to honoring robots.txt and generally do so for their primary crawlers. However, independent audits in 2026 have documented cases where smaller or less established crawlers ignore directives entirely. For content you need to actively protect, robots.txt should be treated as a first layer of control rather than a complete solution.

How often should I update my robots.txt AI crawler directives?

A review every 60 to 90 days is a reasonable baseline, with immediate reviews triggered by news of a new AI crawler launch, an operator publishing a new user-agent token, or changes to your own site's content structure. The AI crawler landscape has expanded considerably in 2025 and 2026, and a file that was comprehensive at the start of the year may be missing several agents by mid-year. Keeping a versioned copy of your robots.txt with dated change notes makes ongoing management significantly easier.