Welcome to the Pulse: updates impact who can crawl your site, what an AI index records about your pages, and how Google will compare your campaigns with similar businesses. Here’s what you need to know for your work. Cloudflare’s AI Blocking Will Cover Googlebot In September Starting September 15, 2026, Cloudflare will block crawlers that […]
OpenAI Says Robots.txt May Not Apply To ChatGPT’s Fetch Bot
ChatGPT’s page-fetching bot is disallowed by more sites than any other AI bot of its kind. It also reached disallowed pages on more sites than any other bot. OpenAI says robots.txt rules may not apply to it because a person asked for the page. TollBit’s latest State of the Bots report has the numbers for […]