Publishers are locked in a very visible negotiation with Google over AI licensing, blocking, and lawsuits. Reddit spent part of its July 30 earnings call discussing exactly that trade-off. Every move gets covered like a summit. Meanwhile, the biggest machine reader of the open web now belongs to Meta — it reads more of the internet than the crawlers everyone talks about blocking, sends almost nothing back in return, and barely comes up in the conversation about who should be blocked.
Meta’s crawlers became the majority while GPTBot stayed the most blocked
Bot-defense vendor DataDome reports 17.7 billion AI agent requests across its network in the second quarter of 2026, up 45% from the first quarter. That growth did not come from Google or OpenAI. Meta-ExternalAgent grew 74% quarter over quarter, Meta-WebIndexer grew 163%, and together Meta’s two crawlers now account for the majority of the AI agent traffic DataDome observes — with close to nothing returned in referral traffic. It’s worth noting DataDome sells bot protection, so its numbers arrive with a commercial interest attached; treat the exact figures as one vendor’s view of its own network rather than the whole web. The direction is harder to dismiss, though, especially against robots.txt data: by the same company’s count, GPTBot remains the most-blocked AI crawler on the web. The crawler the web organized its defenses around is not the one doing most of the reading.
Google earned the meeting by paying in traffic — Meta never owed anyone a visit
The gap makes sense given each company’s history. For two decades, the deal with Google was straightforward: it crawled your site, and it sent visitors back. When AI summaries started keeping more of those visitors inside Google’s own results, publishers experienced it as a broken promise — which is why the response looks like a renegotiation, with blocking, licensing, and lawsuits as the available levers.
Meta has never worked that way. Keeping the user inside its own properties has always been the model, so when Meta’s crawlers became the heaviest readers of the open web, there was no prior promise for anyone to feel broken. It doesn’t register as a loss the way Google’s shift does — even though the company that perfected the keep-the-user model is now also among the largest consumers of everyone else’s content, while the company being renegotiated with is the one actively copying that model.
If platforms keep the user, there’s less left to negotiate for
The traffic that made the old Google arrangement work is the thing now shrinking, which leaves publishers negotiating mainly over payment for the content that trains and feeds AI systems rather than for referral traffic itself. That’s a weak position for most publishers — Meta does write licensing checks, but only at the very top of the market. In March 2026 it signed a deal with News Corp worth up to $50 million a year, covering both Meta AI answers and model training, alongside similar arrangements with CNN, Fox News, USA Today, and a handful of others. That table seats a few dozen of the largest media brands, while Meta’s crawlers read everyone. A smaller publisher, a niche trade site, or an independent blog has no equivalent seat at that table — and separate publisher-side data confirming OpenAI, Meta, and ByteDance as the leading sources of AI bot traffic shows just how broadly that reading happens across the rest of the web, which is simply read, not negotiated with.
That leaves one realistic path for smaller publishers: build a direct enough relationship with an audience that a platform can’t dilute it, whether that means email lists, owned communities, or a distribution mix that doesn’t depend on any single crawler-driven channel. None of that happens quickly, and none of it runs through the meeting rooms where Google’s negotiations are taking place.
Meta’s reading costs you almost nothing — but that’s not really the point
Because Meta never sent traffic and never promised any, its crawlers reading billions of pages don’t take away something you had. In narrow economic terms, it costs most sites almost nothing per page. But that framing misses the underlying issue: a company is building products on the back of other people’s work without compensating the people who made it — a dynamic covered in more detail in the paradox of brands blocking AI crawlers, then paying to get seen anyway.
DataDome’s numbers likely describe only the first phase of this pattern. The same crawlers reading your site today are plausible candidates to eventually pay for access, or buy through it, and the terms the web sets now — largely by brand recognition and negotiating leverage rather than any consistent policy — are likely to be the terms it lives with later, a tension explored further in whether to block AI crawlers or measure their value first.
What an ordinary website should actually do
For most site owners, the honest answer is: nothing drastic today, but start watching. Watching means checking server logs to see who is actually reading your site, rather than relying on which AI company is dominating headlines — increasingly, those are two different lists. More sites are choosing to block AI crawlers by default, a shift tracked in the move toward defaulting to blocking AI crawlers, but that decision should be based on which bots are actually hitting your infrastructure, not on which company made the most recent headline. The public negotiation everyone can see is with Google. The reader almost no one is watching closely enough is Meta’s, and building genuinely machine-readable sites — a broader shift covered in why AI agents are here and most websites aren’t ready — matters regardless of which specific crawler is doing the reading. Your log files already know the difference. The headlines mostly don’t.