A growing share of your search impressions in 2026 aren’t coming from people typing into a search box. They’re coming from AI agents researching a question on someone’s behalf, and those agents don’t care about your keyword rankings. They care whether your HTML loads in under 200 milliseconds, whether a product page is reachable within four clicks of the homepage, and whether your content actually answers a nine-word question that never once showed up in a keyword research tool.

Server log data across hundreds of enterprise sites backs this up consistently since mid-2025. Here’s what a technical SEO audit needs to look like now that a meaningful portion of your traffic is machine-initiated.

What’s actually happening in your search data

The clearest signal is query length. Ten-word queries grew 161% year over year in Google Search Console data — a rate that has nothing to do with people suddenly typing longer searches. It’s AI agents decomposing a single user prompt into dozens of parallel sub-queries, a process researchers call “fan-out.” By October 2025, queries of seven-plus words reached nearly 1% of total query volume, roughly triple their historical share, and the growth curve tracks machine behavior far more cleanly than it tracks human typing habits.

The click-through data tells the other half of the story. While impressions for ten-word queries spiked 161%, click-through rate on those same queries collapsed to 2.26%, down from 8-11% in 2023. The pattern is straightforward: an AI reads the page, extracts the answer, and hands it to the user directly. The site gets the impression and never gets the visit. Call these phantom impressions — real evidence your content is being pulled into an AI reasoning chain, even though nobody clicked through. If your reporting filters these out because they don’t drive sessions, you’re discarding the exact signal that shows whether AI systems can find and use your content, a blind spot covered further in why AI search skips your content and how to diagnose where it’s failing.

Not all AI bots are the same bot

Treating every AI crawler as one category is the most common mistake in this kind of audit. Training bots crawl broadly without regard to click depth — a training visit only confirms the AI knows your content exists, not that any user will ever see it referenced. AI search bots go deeper but drop off sharply beyond two or three clicks from the homepage, typically visiting a given page once a month at most. AI user bots are the ones that matter for visibility: they fire only when a real person asks ChatGPT, Perplexity, or Claude a question and the model researches the answer live. A site can be crawled heavily by training and search bots and still be entirely absent from AI-generated answers, so segmenting bot traffic by type in your log analysis isn’t optional — without it, you don’t know which third of the picture you’re actually measuring.

Which technical signals still matter

Robots.txt remains the primary lever. ChatGPT, Claude, and Gemini generally follow robots.txt directives. Perplexity is a partial exception — PerplexityBot respects it, but Perplexity-User, the bot triggered by an actual user question, does not, a distinction Cloudflare confirmed independently. Most sites have never audited robots.txt with AI crawlers specifically in mind, and that’s worth fixing first. XML sitemaps are also broadly used by ChatGPT, Claude, and PerplexityBot for URL discovery, so keeping them accurate still pays off.

Other familiar signals have stopped mattering for this purpose. Canonical tags and noindex directives don’t influence AI crawlers at all — they aren’t building a search index, so content hidden from Google with noindex remains fully visible to ChatGPT’s crawler. Llms.txt files also show almost no effect in log data; major AI bots don’t appear to read them, which is a point worth pairing with the deeper structural issues covered in why Lighthouse fails your llms.txt without markdown links. The bigger blind spot is JavaScript rendering: most AI crawlers, including ChatGPT, Claude, and Perplexity, don’t render JavaScript, so a product page that loads key content client-side is read as an empty shell. Server-side rendering is the one architecture that works reliably across all of them — the sole exception is Google’s Gemini, which uses the same Web Rendering Service as Googlebot.

Getting AI crawlers to actually reach your best content

Because AI search bots drop off beyond three clicks from the homepage, the pages with the most specific, directly answerable information are often the hardest for them to reach. The fix is internal linking: elevate your highest-value deep pages so they sit within four clicks of the homepage. Pages that training bots reach but user bots never touch are your clearest priority fixes. Pages that AI user bots do visit frequently are telling you exactly what to build more of — additional content covering the same topic cluster at similar depth.

Building content around fan-out queries

Roughly 95% of the queries driving AI citations show zero monthly search volume in traditional tools, because they’re synthetic sub-queries an AI model generated on the fly. They still show up in Search Console as impressions with no clicks, at query lengths no one would target intentionally through conventional research. Filtering Search Console data for queries longer than seven words, under 50 impressions, and zero clicks over a recent three-month window surfaces exactly these opportunities — call it a fan-out opportunity matrix, since it maps directly onto the sub-questions AI agents are generating about your content.

List and comparison formats capture a disproportionate share of this fan-out traffic. If your content doesn’t structure information as explicit rankings, pros and cons, or side-by-side specs, you’re leaving the largest share of that surface area unaddressed. The scale here is significant: “product review” intent queries went from 239 in June 2025 to more than 40,000 by September 2025, a 16,000% increase driven by AI agents systematically harvesting structured opinion content. Product pages without that depth are effectively invisible to that harvest.

Running the audit step by step

Start with your server logs. Pull raw Apache or Nginx logs and isolate every line containing the relevant AI user agents — OAI-SearchBot and ChatGPT-User, PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User — then group hits by user agent and endpoint. Distinguishing training bots from user bots requires maintaining your own classification list, since agent behavior and naming shift often; dedicated log analysis platforms can automate this segmentation if manual tracking doesn’t scale for your site.

Next, audit technical accessibility on a sample of deep URLs: check HTML payload size, view raw HTML to confirm content isn’t injected via JavaScript, count clicks from the homepage to estimate crawl depth, and test load time with Lighthouse or Chrome DevTools. Check specifically whether important content sits behind accordions or “view more” elements — anything requiring a click or JavaScript execution is invisible to bots that don’t interact with the page.

Then review robots.txt line by line for every user agent’s Allow and Disallow rules, and manually confirm key URLs aren’t accidentally blocked. This takes about thirty minutes and prevents two opposite failures — locking out crawlers you want and exposing content you’d rather keep private.

Map your phantom impressions by exporting Search Console performance data filtered to impressions with zero clicks, working around the 1,000-row UI limit via the API or by exporting in date chunks and merging the results. Factor in query frequency too — a long query that recurs daily is more likely a genuine recurring pattern than a one-off fan-out artifact.

Finally, put a recurring monitoring process in place: pull Search Console data monthly, re-run log analysis to catch shifts in bot activity, and track Core Web Vitals separately through PageSpeed Insights or CrUX. Stitching these sources together manually is tedious but catches regressions before they compound — the same discipline that underpins ongoing AI search performance benchmarking.

Technical accessibility is the new baseline KPI

SEO in 2026 increasingly comes down to one question: can an AI agent crawl, reach, and extract a fact from your deepest product page in under 200 milliseconds? If the answer is no, your rankings, backlinks, and content quality stop mattering for a growing share of search interactions, no matter how strong they look by traditional metrics. The starting point is always your logs — everything else in this audit builds on what they show you.

Frequently asked questions

What is a “phantom impression”?

A phantom impression is a Search Console impression with no accompanying click, generated when an AI system reads and extracts an answer from your page without a user ever visiting it. It’s a real signal of AI visibility even though it looks like a dead metric in standard reporting.

Does blocking AI bots in robots.txt actually work?

Mostly yes. ChatGPT, Claude, and Gemini generally respect Disallow directives. Perplexity is the exception: PerplexityBot honors robots.txt, but Perplexity-User, which fires when a real person asks a question, does not.

Should I still maintain an llms.txt file?

Log data from hundreds of enterprise sites shows major AI bots largely aren’t reading llms.txt files today, so it shouldn’t be a priority investment. Robots.txt, sitemaps, and server-side rendering deliver far more measurable impact.

Why does JavaScript rendering matter so much for AI visibility?

Most AI crawlers, including ChatGPT, Claude, and Perplexity, do not render JavaScript. If key content on a page loads client-side, these crawlers see an empty shell rather than the actual page. Server-side rendering is the only approach that works reliably across all major AI platforms except Gemini, which shares Googlebot’s rendering service.