The “are you a bot” check used to block bad traffic can sometimes cause pages to fall out of Google. On the latest episode of Search Off the Record, Google’s John Mueller explained why this happens, noting that it might cause Google to consider your page a duplicate of another site’s page.
The Problem
When a site’s security flags a visitor as suspicious, it shows an “are you a bot” page instead of the real content. Sometimes, this page is returned to Google as if it were normal, leading Google to index it. This can cause the site’s real content to drop from the index or be replaced by this interstitial.
Given that the ‘are you a bot’ screen appears on other sites, Google encounters several similar pages. When it detects duplicate-looking pages, Google selects one as the canonical version and considers the others as duplicates.
In addition to affecting indexing, this issue could also affect which page Google views as the main version of your content. It may select a page from another website, causing your page to be marked as a duplicate.
Mueller explained that it’s difficult to trace because it requires examining the page Google selected and then working backward to determine the cause. Additionally, since a typical visitor might never encounter the “are you a bot” prompt, there seems to be nothing wrong when you review the site yourself.
Why It’s Hard To See
Normal browser checks may not allow you to see the “are you a bot” screen, because it only appears for visitors flagged as suspicious. For you, the page probably loads normally.
You’ll have to go to Search Console to confirm there’s a problem. The page indexing report flags pages as duplicates or as canonicalized elsewhere.
Additionally, the Search Console’s URL Inspection tool reveals which address Google has selected as the main version. If that address belongs to a site that isn’t yours, it’s a good idea to look into it.
Why This Matters
Sometimes, issues can arise without any obvious signs on the site itself because the “are you a bot” screen might be triggered by a CDN, host, or bot-protection layer, rather than the pages directly. Mueller mentioned that this type of protection can activate when crawling picks up.
The request goes through successfully, so the common instinct to look for a broken page isn’t helpful here. Since Google is able to reach the site and get a valid response, the issue is that it’s receiving the wrong content. This is why the failure isn’t caught by checks that don’t recognize what Googlebot actually sees.
I’ve covered a similar issue where Mueller described the “Page Indexed Without Content” error. In that case, a website’s security settings silently blocked Googlebot but allowed regular visitors, causing Google to load the page without receiving any content.
Looking Ahead
If you run into this issue, contact whoever manages your security service, CDN, or web hosting to look into a solution. After fixing it, ask Google to re-crawl via Search Consoles Validate Fix. Google may also detect the correction during its next crawl.
Featured Image: Rokas Tenys/Shutterstock