Block your site’s search result pages

Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed.

Google considers these search results pages to be of lower quality than your actual informational pages. That doesn’t mean these internal search pages are useless, but it makes sense to block these internal search pages.

Back in 2007

Many years ago, Google, or more specifically Matt Cutts, told us that we should block these pages in our robots.txt. The reason for that:

Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index. (Not all URLs that contains things like “/results” or “/search” are search results, of course.)
– Matt Cutts (2007)

Nothing changed, really. Even after more than 10 years of SEO changes, this remains the same. The Google Webmaster Guidelines still state that you should “Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.” Furthermore, the guidelines state that webmasters should avoid techniques like automatically generated content, in this case, “Stitching or combining content from different web pages without adding sufficient value”.

However, blocking internal search pages in your robots.txt doesn’t seem the right solution. In 2007, it even made more sense to simply redirect the user to the first result of these internal search pages. These days, I’d rather use a slightly different solution. 

Blocking internal search pages in 2018

I believe nowadays, using a noindex, follow meta robots tag is the way to go instead. It seems Google ‘listens’ to that meta robots tag and sometimes ignores the robots.txt. That happens, for instance, when a surplus of backlinks to a blocked page tells Google it is of interest to the public anyway. We’ve already mentioned this in our Ultimate guide to robots.txt.

The 2007 reason is still the same in 2018, by the way: linking to search pages from search pages delivers a poor experience for a visitor. For Google, on a mission to deliver the best result for your query, it makes a lot more sense to link directly to an article or another informative page.

Yoast SEO will block internal search pages for you

If you’re on WordPress and using our plugin, you’re fine. We’ve got you covered. Your internal search results pages are noindexed by default in Yoast SEO. In addition, you can set a template for the page title of those pages, so the correct title will display in your browser tab or if your title is used in your theme.

You can set the template in the Search Appearance section of Yoast SEO. Just click on SEO › Search Appearance › Archives › Special pages in the left hand menu, and you’ll get this. This is the default template, but you can amend it, if you consider it necessary:

Search Appearance special pages

Most other content management systems allow for templates for your site’s search results as well, so adding a simple line of code to that template will suffice:
<meta name="robots" content="noindex,follow"/>

Meta robots AND robots.txt?

If you try to block internal search pages by adding that meta robots tag and disallowing these in your robots.txt, please think again. Just the meta robots will do. Otherwise, you’ll risk losing the link value of these pages (hence the follow in the meta tag). If Google listens to your robots.txt, they will ignore the meta robots tag, right? And that’s not what you want. So just use the meta robots tag!

Back to you

Did you block your internal search results? And how did you do that? Go check for yourself! Any further insights or experiences are appreciated; just drop us a line in the comments.

Read more: Robots.txt: the ultimate guide »


7 Responses to Block your site’s search result pages

  1. Anil Agarwal
    Anil Agarwal  • 3 months ago

    Hi Michiel

    Please help us in knowing how we can get rid of thousands of search pages that are getting indexed into Google. You can check the same from here: https://www.google.com/search?q=site:https://bloggerspassion.com/search/&ei=VZeDW-GNLZGo9QPeyLqAAg&start=10&sa=N&filter=0&biw=1280&bih=726

    I’m unable to understand how they are getting created and they are not part of website (website layout), so unable to understand how I can add the meta noindex tag in these pages. Then I can remove the Disallow: */search/* command from robots.txt file.

    Please help here.

    Thanks

  2. Desmond SEO
    Desmond SEO  • 3 months ago

    Very Useful info, and yes it is not something that is always on the top of my mind for the very fact that I do use the Yoast SEO plug-in for every word press site we manage, but a good reminder for those sites without WP.
    Thanks for sharing.

  3. StudioWasabi
    StudioWasabi  • 3 months ago

    Very important not only for newbie but also for “experts” , who forgot these best practices…

  4. Pradeep Singh
    Pradeep Singh  • 3 months ago

    Hello,
    If I Search My blog name then Google show Page 1, Page 2 and so on in Snippet, I don’t want it, How to Disable It.
    Thanks

  5. Marie
    Marie  • 3 months ago

    But what about the fact that Google said that on long-term noindex on a page will lead to them not following links on that page (you have a good article about this https://yoast.com/pagination-seo-best-practices) ?

  6. Brad Neuendank
    Brad Neuendank  • 3 months ago

    What about adding rel=canonical (to the canonical version) to all these low-value internal search pages?

    • Michiel Heijmans

      I wouldn’t use canonicals for that. Noindex makes the most sense. Search results linking to search results is just not what your visitor would find the best experience either.


Check out our must read articles about Crawl directives