30 N Gould St Ste N, Sheridan, WY 82801
+212 681 53 04 05
contact@skyweb3agency.com
ChatGPT’s page-fetching bot is disallowed by more sites than any other AI bot of its kind. It also reached disallowed pages on more sites than any other bot. OpenAI says robots.txt rules may not apply to it because a person asked for the page. TollBit’s latest State of the Bots report has the numbers for […]
The single largest AI crawler on my website over the past day was not an AI crawler. It arrived roughly 1,500 times under Common Crawl’s name; it sent back nothing, and what it wanted was my SSH keys. I went looking because of a number. Cloudflare’s CFO Told Analysts Machine Traffic Could Reach 1,000 Times […]
Cloudflare announced two products on August 4, 2026, during its Agents Week: Cloudflare Wallets, which will let AI agents hold stablecoins and pay for what they use, and cloudflare.pay, an optional identity handle that tells a merchant which account an agent acts for. Handle reservation opened the same day, and the payment features are future […]
Every month, a non-profit called Common Crawl fetches over 2 billion webpages and gives the copy away to anyone who wants it. That free archive is where most AI training data starts. When Mozilla audited the LLMs released between 2019 and 2023, 64% had trained on Common Crawl data in some form, and GPT-3 was […]
Alphabet’s CEO Sundar Pichai announced that Google DeepMind co-founder and CEO Demis Hassabis is now Chief Scientist of Alphabet and Chair of Google DeepMind. In bigger news, Jeff Dean has announced that he is stepping away from Google and is founding a new AI company called Discovery Loop. These changes may be attributable to the […]
In my previous Search Engine Journal article, I argued that organizations need to build Brand Sovereignty by becoming the most authoritative source of truth for AI. As AI increasingly serves as the intermediary between businesses and customers, the organizations most likely to be recommended will be those that provide the highest-confidence evidence about their products, […]
There’s a report that blocking AI training bots with Cloudflare is also blocking legitimate Googlebot and Bingbot from indexing a website. Cloudflare recently announced that this would happen beginning in September. It’s unclear whether this is a fluke or user error but it does serve as a reminder to be aware of the pending change […]
Google updated its crawl budget documentation with new information about how Google’s different crawlers all share a website’s crawl capacity and added a recommendation to use a 304 server response to reduce the amount of server resources used by crawlers. Both of these additions are to Google’s Optimize Your Crawl Budget Documentation that is for […]
Google’s newly updated crawl budget optimization guide has been updated to be more precise, with emphasis given to the different parts of crawling in order to provide greater detail that is more precise than the previous version of the documentation. Aside from improvements in precision and understandability, Google adds an entirely new section. The previous […]
If entity mapping feels familiar, that is because you already ran this play. It is the knowledge graph conversation you were having in 2021, wearing a 2026 outfit. The vocabulary migrated almost intact: entities, not strings; disambiguation; sameAs; relationships between nodes; feed the structure; and the machine understands you. Pull a client deck from four […]