Google’s John Mueller answered a question on Reddit about a link to an internal web page that was automatically created by Squarespace, a closed-source platform. The link to the web page was also blocked from crawling by robots.txt, apparently serving no purpose for the Redditor’s client. The person asking the question was frustrated because the […]
How Publishers Can Monetize AI Visibility
Right now, most publishers block AI bots. According to Vince Nero’s excellent research, 79% of top news sites block AI training bots via robots.txt. Only 18% block no AI crawlers at all. Not a perfect science, as the robots.txt is so easily circumvented, but still. No one is playing nice anymore. Multiple publisher x AI […]