In early August, an SEO named Anastasia Kourou spotted something strange in her Search Console report: queries that weren’t searches at all. “Yes.” “Yes go on.” “Yes, pricing.” She posted a screenshot and asked Google’s John Mueller directly whether Search Console was somehow logging what people say to an AI chatbot. Mueller confirmed it was.

That confirmation opened up a genuinely useful, if slightly unsettling, discovery for anyone who reads Search Console data: your query report has quietly become part conversation transcript, and there’s a way to tell the two apart.

How a chat reply becomes a Search Console query

AI Mode looks and feels like a chatbot, but underneath, every message — including every follow-up in a multi-turn conversation — gets processed as a fresh Google search. When a user follows up on an AI Mode answer with something like “yes, go on” or “what about the pricing,” that follow-up is treated as a brand-new query, and everything in the AI’s response gets attributed to it. If your page shows up in that response, Search Console logs an impression for your page against the query “yes go on” — a string nobody would ever type into a search box.

The proof isn’t just anecdotal. Position data gives it away. A page can show an average position of 4.5 for the query “yes” — a ranking that’s meaningless on the open web, where “yes” belongs to song lyrics and grammar pages, not commercial content. But inside an AI-generated answer, position data reflects where a link sat within the answer block, not where it ranked on a results page. A position of 4.5 for a one-word reply means the link was embedded inside a conversational AI response, not competing for a search results slot.

This matters because Google’s separate Generative AI performance report — which shows how often pages appear in AI Overviews and AI Mode — deliberately withholds query and click data. It has no API support either, and the only way the data leaves Google is a manual CSV export from the interface. The ordinary Search Console performance report anyone already reads, by contrast, has been quietly absorbing these AI conversation fragments the whole time, unfiltered and unlabeled.

Seven recognizable kinds of leaked fragment

Sorting genuine search queries from AI conversation leakage turns out to be possible by pattern-matching a handful of consistent signatures. Analysis of 16 months of Search Console data on one SEO’s own site — roughly 1,127 non-search queries and 20,300 impressions, against millions of ordinary impressions — sorted the fragments into seven categories:

• Reply artifacts — bare responses like “yes,” “sure,” “really?” or “show me,” where a person answered an AI mid-conversation and a page appeared in what came back.
• Pivot follow-ups — comparison requests such as “what about [competitor]?” where someone already has an answer and asks the AI to check an alternative — a strong signal of exactly what the person is deciding between.
• Conversational questions — phrasing addressed to a listener rather than typed into a search box, like “can you show me” or “is it free.”
• Tracker probes — synthetic, repeated prompts from AI-visibility monitoring tools, identifiable because the same phrasing recurs on a schedule that no human would replicate.
• Agent harnesses — full instruction sets from automated agents, logged verbatim, such as a prompt template telling a bot to “search the web for… return the 3 most relevant results… do not invent results or urls.”
• Pasted strings — error messages, CSV headers, or other text pasted wholesale and searched as-is.
• Long, uncategorized — strings of ten-plus words with no other clear marker, filed for manual review rather than forced into a category.

The distinguishing test isn’t length alone — long-tail queries have always existed and aren’t new. The real difference is whether a query is addressed to someone. A typed long-tail query names its own subject and stands alone; a conversational fragment reads like half of a dialogue, missing context that only makes sense if there was a preceding exchange.

What this looks like at scale

In the sampled dataset, reply artifacts were essentially nonexistent for eight months and then became a persistent, recurring pattern starting in spring 2026 — a behavior shift with a visible start date, coinciding with wider AI Mode adoption. The conversational-question bucket was the largest of the seven: hundreds of queries generating thousands of impressions but only a handful of clicks, which is roughly what “being read inside an AI answer instead of clicked through to” looks like in the data. Tracker probes and agent harnesses, meanwhile, showed up as clearly mechanical: the same prompt repeating daily for weeks, or the same multi-hundred-word instruction block firing thousands of times in a few days — evidence that AI-visibility monitoring tools and automated research agents are themselves generating a meaningful share of what looks like “search” activity in aggregate reporting.

One important caveat: what’s visible in Search Console is a floor, not the full picture. Google anonymizes rare queries, and conversational fragments are close to rare by definition — almost nobody phrases a follow-up exactly the way anyone else does. Bulk export data suggests the majority of AI-surface impressions carry no visible query string at all, meaning the classifiable fragments are only the tip of a much larger, hidden pool.

Why this matters for SEO reporting

The practical risk is a reporting one. Any keyword-opportunity tool or manual query review that doesn’t account for this leakage risks recommending that a site “optimize for the query ‘yes.'” Machine-generated fragments — probes and harnesses in particular — need to be excluded from opportunity analysis entirely; they represent monitoring software, not human demand, and treating them as query volume will distort any keyword strategy built on top of that data.

Reply artifacts and conversational questions, on the other hand, are genuinely useful signal. They confirm which pages are being read and cited inside multi-turn AI conversations rather than clicked. For those pages, headline and title-tag optimization does very little, since the person reading the AI’s answer never sees the title — what gets pulled into the response is the passage that directly answers the question, which puts the emphasis back on clear, extractable answers near the top of a page rather than on classic on-page SEO signals. That’s consistent with the broader shift already visible in how AI search users have moved past keyword-style queries while much site content hasn’t caught up.

Pivot follow-ups are arguably the most directly actionable category. A query like “what about [competitor]?” appearing against a page is a direct signal that the comparison a reader wants isn’t currently on that page — a gap that can be closed by adding the missing comparison section.

A path around the data Google won’t hand over directly

Because Google’s Generative AI report only shows visibility, not the queries or clicks behind it, and because the query-leakage fragments only reveal what people asked, combining both gives a more complete page-level picture than either report alone. Exporting the Generative AI report’s page-level CSV and cross-referencing it against classified query fragments shows, for each page, both how much of its visibility comes from AI features and what people were actually asking when they arrived there. Pages with high AI visibility and rich conversational fragments skew toward AI Mode’s back-and-forth conversations; pages with high AI visibility and few or no fragments skew toward one-shot AI Overview citations. That’s an inference rather than a hard measurement, since Google reports AI Overviews and AI Mode together under the same “web” search type, but it’s currently the only available signal into that split.

There are real limits here. Classification is pattern-based, so it judges how a query looks rather than what a person meant — a short, odd question could still be a genuine search. It’s also English-centric for now, and because Google won’t expose this data through an API, extracting it requires a manual export. Google’s own Generative AI report, notably, is already drawing scrutiny for how little it tells marketers — this leakage is one of the few ways to fill in what that report leaves out.

The bigger picture

None of this changes the fundamentals of AI search visibility, but it does hand SEOs a genuinely new diagnostic: proof, inside data most teams already collect, that specific pages are being cited and read inside live AI conversations — not just theoretical AI Overview appearances. Google has been testing dedicated AI search reporting inside Search Console for months, and this kind of query leakage is a reminder that the reporting gap between what AI systems do with a page and what publishers are told about it is still wide. Reading the fragments already sitting in a standard performance report is, for now, one of the more reliable ways to close part of that gap.

Frequently asked questions

Why do AI Mode conversations show up in Search Console at all?

Every message inside AI Mode, including follow-up replies, is processed internally as a new Google search and folded into the same “web” search type as ordinary queries. If a page is cited in the AI’s response, that follow-up gets logged as a query with an impression attributed to the page.

Does Google’s Generative AI report show this same data?

No. That report shows impressions by page, country, device, and date, but deliberately excludes queries and clicks, and has no API support. The manual CSV export is currently the only way to get even the page-level data out.

Are all unusual-looking queries actually AI conversation fragments?

Not necessarily. Classification is pattern-based and can misjudge edge cases — a short, odd-looking query could still be a genuine human search. Fragments should be treated as directional evidence, not a precise measurement.

Should these fragments be included in keyword opportunity analysis?

Machine-generated fragments, such as repeated tracker probes or agent instruction sets, should be excluded entirely since they reflect software activity, not human demand. Genuine conversational fragments from real users are useful as qualitative signal about what people are asking, not as query-volume data.