block internal search pages

Block your site’s search result pages

Block your site’s search result pages

May 10th, 2017 – 14 Comments

Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed.

Google considers these search results pages to be of lower quality than your actual informational pages. That doesn’t mean these internal search pages are useless, but it makes sense to block these internal search pages.

Back in 2007

10 Years ago, Google, or more specifically Matt Cutts, told us that we should block these pages in our robots.txt. The reason for that:

Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index. (Not all URLs that contains things like “/results” or “/search” are search results, of course.)
– Matt Cutts (2007)

Nothing changed, really. Even after 10 years of SEO changes, this remains the same. The Google Webmaster Guidelines still state that you should “Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.” Furthermore, the guidelines state that webmasters should avoid techniques like automatically generated content, in this case, “Stitching or combining content from different web pages without adding sufficient value”.

However, blocking internal search pages in your robots.txt doesn’t seem the right solution. In 2007, it even made more sense to simply redirect the user to the first result of these internal search pages. These days, I’d rather use a slightly different solution. 

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

Blocking internal search pages in 2017

I believe nowadays, using a noindex, follow meta robots tag is the way to go instead. It seems Google ‘listens’ to that meta robots tag and sometimes ignores the robots.txt. That happens, for instance, when a surplus of backlinks to a blocked page tells Google it is of interest to the public anyway. We’ve already mentioned this in our Ultimate guide to robots.txt.

The 2007 reason is still the same in 2017, by the way: linking to search pages from search pages delivers a poor experience for a visitor. For Google, on a mission to deliver the best result for your query, it makes a lot more sense to link directly to an article or another informative page.

Yoast SEO will block internal search pages for you

If you’re on WordPress and using our plugin, you’re fine. We’ve got you covered:

Block internal search pages

That’s located at SEO › Titles & Metas › Archives. Most other content management systems allow for templates for your site’s search results as well, so adding a simple line of code to that template will suffice:
<meta name="robots" content="noindex,follow"/>

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training Info

Meta robots AND robots.txt?

If you try to block internal search pages by adding that meta robots tag and disallowing these in your robots.txt, please think again. Just the meta robots will do. Otherwise, you’ll risk losing the link value of these pages (hence the follow in the meta tag). If Google listens to your robots.txt, they will ignore the meta robots tag, right? And that’s not what you want. So just use the meta robots tag!

Back to you

Did you block your internal search results? And how did you do that? Go check for yourself! Any further insights or experiences are appreciated; just drop us a line in the comments.

Read more: ‘Robots.txt: the ultimate guide’ »


14 Responses to Block your site’s search result pages

  1. Daniel Meyer
    By Daniel Meyer on 18 May, 2017

    Is this feature in the premium version only? I am testing the module right now, and there definitely is no Titles & Metas › Archives tab in the SEO section.

    Other than that, great module, very useful.

  2. Blogger Dashboard
    By Blogger Dashboard on 17 May, 2017

    It is a amzing article to blog my unwanted pages.

  3. K Patel
    By K Patel on 16 May, 2017

    Thanks for sharing this with us. Yes we are using Yoast but we are not aware of this. I am using Yoast but till date i was not aware how to block internal search page. Thanks for sharing this with us.
    and yes of course we are fully satisfied with your wordpress plugin

  4. Amit
    By Amit on 14 May, 2017

    That’s absolutely a right thing you have mentioned. We too use yoast and its really good!

  5. Frank Okun
    By Frank Okun on 11 May, 2017

    Thanks for the article on robots.txt and meta robot. Most of our new and potential clients are not aware of the technical aspects of rankings, and don’t have the time or desire to get educated – which is why they hire a web company to manage and improve their sites. It helps sometimes for us web guys and gals to be reminded of the puzzle pieces that affect rankings. Thanks again.

    • Michiel
      By Michiel on 15 May, 2017

      You’re welcome, Frank!

  6. T Man
    By T Man on 11 May, 2017

    The field in my Yoast SEO is blank – should it be filled, if so how do I know what to populate it with to block indexing search results?

    • Michiel Heijmans
      By Michiel Heijmans on 17 May, 2017

      at “special pages”? That only adds a template for your title. Blocking is done by default.

  7. Michael Romano
    By Michael Romano on 11 May, 2017

    Does Yoast’s plugin do this automatically for WordPress?

    • Willemien Hallebeek
      By Willemien Hallebeek on 12 May, 2017

      Yes it does!

  8. Pattipie
    By Pattipie on 11 May, 2017

    This is so useful! Thanks! Going to block it now!

  9. Doug Smith
    By Doug Smith on 11 May, 2017

    I blocked my search results with robots.txt recently because sploggers and scammers were abusing search engines to hit search queries on my site. With it blocked, they can’t get the results they need for their nefarious purposes.

    • Michiel Heijmans
      By Michiel Heijmans on 17 May, 2017

      Please read this post on why just using robts.txt isn’t sufficient. But if you are using our plugin, you are fine.

  10. Rupesh Kumar
    By Rupesh Kumar on 11 May, 2017

    This was an issue for me before but not anymore, since I am using Yoast plugin on my website. Anyways, thanks Michiel for reminder.


Check out our must read articles about Technical SEO