Google’s John Mueller answered a question about robots.txt and explained an easy-to-miss mistake that can impact your SEO and website indexing goals. The specific issue was related to search box spam getting indexed by Google, but this mistake can happen to anyone in general under any context.

Website Search Box Spam

The person who asked the question on Reddit was suffering from a search bar spam attack. What spammers do is search with a query that reflects their spammy niche, and they add a link or a website name. What happens next is that the search bar generates a URL that can be referenced to generate the spammy search result.

And that’s what was happening to the person who was asking the question. Their response was to add a line in the robots.txt file to prevent Google from indexing the file. But Google was indexing those spammy search-generated URLs anyway.

Google Indexed Pages Blocked By Robots.txt

Someone posted on Reddit that their client’s Shopify search box was generating spammy web pages in response to spammer queries and that Google was indexing them despite a robots.txt file prohibiting Google from indexing those pages. What the client did was redirect those spammy URLs to another web page. The person asking the question didn’t ask how to stop the pages from being indexed (which is what they should have been asking); they asked if those redirected URLs should be marked 404 instead.

The person asked:

“Working on a client’s Shopify store where we have the /search added as a disallow in robots.txt, however these search results are still indexed inside of Google.

However, if I try to open one of these pages, they have a redirect set and they redirect to another collection page on the store. Should we display a 404 page instead? What is the easiest way to fix this?”

Why Robots.txt File Caused Spam To Be Indexed

Google’s John Mueller took the extra step to identify and review the client’s robots.txt file and identified an error that was causing Google to ignore the directive prohibiting Googlebot from indexing search results pages.

Mueller responded:

“Also, not sure if it’s your site, but the one I found with similar indexed URLs had sections for “user-agent: Googlebot” (in the “START: Custom Rules” block in comments) as well as a lot more in the “user-agent: *” section further down. With robots.txt, the more specific rules win, so if you have a user-agent: Googlebot section, it will *only* use that section. If you want to apply all the rules in the “user-agent: *” section, you need to copy them. Also, if that’s your site, then you can just list all the user-agents that you want to have shared rules for together, eg:

user-agent: googlebot

user-agent: otherbot

user-agent: imgsrc

user-agent: somethingpt

disallow: /fishes

disallow: /orange-cats

… etc …”

User-Agent Specific Directives Take Precedence

What happened is that the client was relying on Google to follow the directives in a line that’s aimed at all user agents, “user-agent: *”, but because there’s another section of the robots.txt file that’s specific to Googlebot, Google ignored the “user-agent: *” directives and obeyed the one that specifically addressed Googlebot.

That may sound like a quirk in the way robots.txt works, but it actually makes sense because this enables users to target specific crawlers with unique rules and target everyone else with a different set of rules.

How To Be Safe From Search Box Spam

WordPress and Shopify both have ways to mitigate search box spam.

WordPress and Shopify both have ways to mitigate search box spam.

Shopify Search Box Spam Mitigation

Shopify’s website has a tutorial on how to automatically add a noindex directive to all search results. This will effectively prevent indexing of all search results pages. However, it’s necessary to not block search pages with robots.txt for this to work.

Using a noindex directive is more effective than using robots.txt because robots.txt does not control indexing; it only controls crawling.

Shopify instructs:

“You can hide pages that aren’t included in your robots.txt.liquid file by customizing the

section of your store’s theme.liquid layout file. You need to include some metatag code to stop the indexing of particular pages.

From your Shopify admin, go to Online Store > Themes.

Find the theme you want to edit, click the … button to open the actions menu, and then click Edit code.

In the layout folder, click the theme.liquid file.

To exclude the search template, paste the following code on a blank line in the

section:

{% if template contains ‘search’ %}

{% endif %}”

WordPress Search Box Spam Mitigation

It’s quite easy to mitigate search box spam with WordPress. Users of the Yoast, Rank Math, and AIOSEO SEO plugins have their search results pages automatically set to noindex by default. Additionally, some page builders and themes like Divi (and its Extra theme) will automatically not generate spammy words and URLs in response to searches and instead will inject a few sentences saying that the search produced no results.

Robots.txt Knowledge

Effective SEO requires a wide range of knowledge. Robots.txt contains some quirks that can cause it to be less effective than intended, so it’s useful to read up on the official specifications in order to keep up to date.

Featured Image by Shutterstock/Stockinq