This news publisher AI robots.txt implementation case study documents how a 400-article regional news site surgically blocked AI training crawlers while keeping discovery and referral pathways fully intact — and tracked zero net traffic loss across a 90-day measurement window. The publisher's approach offers a replicable framework for any editorial team navigating the tension between protecting content from bulk AI harvesting and maintaining visibility in AI-powered search surfaces. What made the difference wasn't what they blocked; it was what they deliberately chose not to block.

The Publisher's Context and the news publisher AI robots.txt implementation case study That Forced a Decision

The publisher in question runs a regional news operation covering local government, business, and community events. Their site comprised approximately 400 articles published over three years, drawing between 85,000 and 95,000 monthly sessions at baseline. Roughly 28% of that traffic arrived via organic search, 11% from social referrals, and a notable 9% from AI-generated referrals — a figure that had grown steadily over the prior two quarters as AI-powered answer engines began citing local news sources more frequently.

The editorial team had no dedicated technical SEO resource. Their CMS was a self-hosted WordPress installation, and their robots.txt file had never been intentionally modified since launch. When the topic of AI crawlers emerged internally, the immediate fear was blunt: block everything AI-related and lose the referral upside they had just started measuring. The countervailing fear — that training crawlers were consuming their content commercially without compensation — made inaction feel equally untenable.

Industry sentiment was shifting fast around them. An analysis of large publishers' robots.txt practices found that 60 of 70 publishers explicitly restricted at least one AI crawler as of July 2026. The publisher recognized they were late to act, not early.

Strategy: What Was Decided and What Was Explicitly Left Alone

The editorial director and their freelance developer spent two weeks mapping crawler intent before touching a single line of robots.txt. The core insight driving their strategy was that not all AI crawlers serve the same purpose. Some crawlers exist to ingest content into training datasets — a commercial use with no guaranteed benefit to the publisher. Others crawl to populate AI search features that actively send users back to source pages.

The team drew a hard line between training crawlers and retrieval crawlers. GPTBot, OpenAI's training crawler, and Google-Extended, Google's opt-out mechanism for Gemini model training, were identified as training-only signals. OAI-SearchBot, OpenAI's retrieval crawler that powers ChatGPT's browsing and citation features, was identified as a referral-generating agent. Googlebot was never on the table for restriction.

"The moment we separated 'train on our content' from 'send users to our content,' the decision became obvious. We weren't anti-AI — we were pro-getting-paid-for-what-we-make."

What was explicitly not done is as instructive as what was: the publisher did not block PerplexityBot, did not restrict Anthropic's ClaudeBot, and did not apply blanket disallow rules to any path beyond the root-level content archive. They also chose not to pursue content licensing for AI search engines at this stage, reasoning that licensing negotiations require leverage they would build first by demonstrating their content's citation value.

Implementation: Steps, Timeline, and Tools Used

The implementation was deliberately low-tech. No third-party crawler management platform was purchased. The publisher used their existing Google Search Console account for pre- and post-implementation crawl monitoring, a free server log analyzer (GoAccess) to track non-Google bot traffic, and a shared spreadsheet to log baseline metrics for 30 days before any change was made.

Week Action Tool Used
Week 1–2 Audit server logs; identify all active crawler user-agents by frequency and origin GoAccess, cPanel raw logs
Week 3 Establish 30-day baseline: sessions, referral sources, AI-attributed traffic Google Analytics 4, Search Console
Week 4 Draft robots.txt changes; legal review of OpenAI and Google crawler documentation OpenAI/Google published crawler docs
Week 5 Deploy robots.txt update; add GPTBot and Google-Extended disallow rules WordPress file editor, SSH
Weeks 6–17 90-day monitoring window; weekly log reviews; zero additional changes GoAccess, GA4, Search Console

The final robots.txt addition was four lines: a User-agent declaration for GPTBot with a Disallow: / rule, and an identical block for Google-Extended. OAI-SearchBot received no entry, meaning it inherited the default allow state. The publisher also reviewed the full AI robots.txt training controls for publishers guidance before finalizing syntax to ensure their disallow rules matched current crawler documentation exactly — a step that caught one early drafting error where they had misspelled "Google-Extended."

Results: Before and After Metrics at 90 Days

The 90-day measurement window ran from deployment through the end of the monitoring period. Baseline monthly sessions averaged 91,200. Post-implementation, the three-month average came in at 93,400 — a modest 2.4% increase that the team attributed to normal seasonal variation rather than any direct effect of the robots.txt change.

Metric 30-Day Baseline (Pre-Change) 90-Day Average (Post-Change) Change
Total monthly sessions 91,200 93,400 +2.4%
Organic search sessions 25,536 (28%) 26,152 (28%) +2.4%
AI-attributed referral sessions 8,208 (9%) 9,340 (10%) +13.8%
GPTBot crawl requests (server logs) ~1,400/month ~110/month −92%
OAI-SearchBot crawl requests ~320/month ~890/month +178%

The most unexpected result was the 178% increase in OAI-SearchBot activity. The team's hypothesis is that by removing GPTBot's high-frequency crawling, server response times improved marginally, making the site a more reliable retrieval target for OAI-SearchBot. AI-attributed referral traffic — sessions arriving from ChatGPT citation links and Perplexity source links — grew from 9% to 10% of total traffic, adding roughly 1,100 incremental monthly sessions. Net organic search share held flat at 28%, confirming that Google's indexing was completely unaffected by the Google-Extended block.

Key Learnings: What Worked, What Failed, and What Was Surprising

Three things worked as expected: the disallow rules were respected by both GPTBot and Google-Extended within days of deployment, organic rankings were entirely unaffected, and the implementation required less than four hours of technical labor once the audit and baseline phase was complete.

One thing failed: the team initially tried to use Google Search Console's URL Inspection tool to verify that Google-Extended was seeing the new rules. It doesn't surface that granularly. Server log analysis was the only reliable confirmation method — a gap that should be built into any implementation plan from the start.

What was genuinely surprising was the scale of GPTBot's pre-block activity. At roughly 1,400 crawl requests per month against a 400-article archive, the crawler was cycling through the entire content set approximately 3.5 times monthly. This consumed measurable server bandwidth and contributed to occasional slow response times during crawl spikes. Post-block server load dropped noticeably during off-peak hours.

"We expected neutral results. What we didn't expect was that blocking a training crawler would indirectly improve our standing with a retrieval crawler. The two systems are more interconnected than the documentation suggests."

The publisher also noted that a 2026 audit finding that 56.4% of news publishers had blocked at least one AI crawler validated their decision retroactively — but they emphasized that the specific crawlers blocked matter more than the act of blocking itself.

How to Replicate This: Actionable Checklist

Any mid-size publisher with a WordPress or static-file CMS can execute this implementation in under five weeks without specialist tooling. The checklist below reflects the exact sequence this publisher used, with the additions they would make in retrospect.

  • Step 1: Pull 30 days of server logs and identify every crawler user-agent hitting your site. Note frequency per crawler.
  • Step 2: Classify each crawler as training-purpose or retrieval-purpose using official documentation from OpenAI, Google, Anthropic, and Perplexity. Do not guess.
  • Step 3: Record a clean 30-day baseline in GA4: total sessions, organic sessions, AI-attributed sessions (UTM or referrer-based), and top referring AI domains.
  • Step 4: Add Disallow: / rules only for confirmed training crawlers (currently: GPTBot, Google-Extended, CCBot). Leave retrieval crawlers unrestricted.
  • Step 5: Verify rule syntax against each crawler's published documentation. Test your robots.txt file using Google's Robots Testing Tool for formatting errors.
  • Step 6: Monitor server logs weekly for the first four weeks post-deployment to confirm target crawlers have reduced activity. Do not rely solely on Search Console.
  • Step 7: At 30, 60, and 90 days, compare all baseline metrics. Make no additional changes during the measurement window to preserve data integrity.
  • Step 8: Document your crawler classification rationale and keep it updated — new AI crawlers are launching regularly, and your robots.txt is a living document.

Frequently Asked Questions

Will blocking GPTBot hurt my site's visibility in ChatGPT answers?

Blocking GPTBot restricts OpenAI's training crawler, not its retrieval crawler. OAI-SearchBot, which powers ChatGPT's ability to browse and cite live web pages, is a separate user-agent that is unaffected by a GPTBot disallow rule. If you block GPTBot but leave OAI-SearchBot unrestricted, your content remains eligible for real-time citation in ChatGPT responses. The two functions — training and retrieval — operate independently and are controlled by separate robots.txt entries.

Does blocking Google-Extended affect Google Search rankings?

No. Google-Extended is a separate opt-out mechanism specifically for Gemini model training and Google's AI products. Googlebot, which controls Search indexing and ranking, is an entirely different crawler that is unaffected by a Google-Extended disallow rule. Publishers can block Google-Extended without any impact on their search rankings, crawl budget allocation for Search, or appearance in Google Discover. Google has publicly confirmed this separation in its crawler documentation.

How do I measure AI-attributed referral traffic in Google Analytics 4?

In GA4, navigate to Reports → Acquisition → Traffic Acquisition and filter by session source. AI platforms like ChatGPT typically appear as referral traffic from domains such as chatgpt.com or openai.com. Perplexity referrals come from perplexity.ai. You can create a custom segment grouping these domains to track AI-attributed sessions as a unified channel. Cross-referencing with server log referrer data gives a more complete picture, since some AI tools strip referrer headers.

What is the difference between a training crawler and a retrieval crawler?

A training crawler ingests content to build or update an AI model's underlying knowledge base — a one-time commercial use with no direct traffic return to the publisher. A retrieval crawler fetches current web pages in real time to answer a user's specific query and typically includes a source citation link, which can drive referral traffic. The same AI company often operates both types: OpenAI runs both GPTBot (training) and OAI-SearchBot (retrieval). Robots.txt allows publishers to block one without affecting the other.

How often should a news publisher update their robots.txt AI crawler rules?

Industry practitioners recommend reviewing AI crawler entries at least quarterly, given how rapidly new AI products and their associated crawlers are launching. Each new AI platform typically introduces a new user-agent, and many do not announce their crawlers proactively. Monitoring server logs monthly for unrecognized high-frequency user-agents is the most reliable early-warning system. When a new crawler is identified, verify its stated purpose through official documentation before deciding whether to allow or restrict it.