CYBER MONDAY SALE! Last chance to get a 30% discount on ALL products!

Block your site’s search result pages

Why should you block your internal search result pages for Google? Well, how would you feel if you are in need of an answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed.

Google considers these search results pages to be of lower quality than your actual informational pages. That doesn’t mean these internal search pages are useless, but it makes sense to prevent Google from sending visitors to them.

Back in 2007

In 2007, Google’s representative Matt Cutts told us:

Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index. (Not all URLs that contains things like “/results” or “/search” are search results, of course.)
– Matt Cutts (2007)

Google’s Webmaster Guidelines backed this up, by stating that you should “Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.

It was best practice to completely block Google from seeing or accessing these types of pages.

Modern best practices for blocking internal search pages

Google’s guidelines no longer mention search results pages specifically, but it’s still good practice to help Google to avoid crawling and indexing ‘infinite spaces’ like search results.

However, blocking internal search pages via your robots.txt isn’t the best way to do this. It’s much better to allow search engines to crawl these kinds of pages, but to prevent them from indexing them by using a meta robots tag with a value of noindex, follow.

Yoast SEO will block internal search pages for you

If you’re on WordPress and using our plugin, you’re fine. We’ve got you covered. Your internal search results pages are noindexed by default in Yoast SEO. In addition, you can set a template for the page title of those pages, so the correct title will display in your browser tab or if your title is used in your theme.

You can set the template in the Search Appearance section of Yoast SEO. Just click on SEO › Search Appearance › Archives › Special pages in the left hand menu, and you’ll get this. This is the default template, but you can amend it, if you consider it necessary:

Search Appearance special pages

Most other content management systems allow for templates for your site’s search results as well, so adding a simple line of code to that template will suffice:
<meta name="robots" content="noindex,follow"/>

Meta robots AND robots.txt?

If you try to block internal search pages by adding that meta robots tag and disallowing these in your robots.txt, please think again. Just the meta robots will do. Otherwise, you’ll risk losing the link value of these pages (hence the follow in the meta tag). If Google listens to your robots.txt, they will ignore the meta robots tag, right? And that’s not what you want. So just use the meta robots tag!

Back to you

Did you block your internal search results? And how did you do that? Go check for yourself! Any further insights or experiences are appreciated; just drop us a line in the comments.

Read more: Robots.txt: the ultimate guide »


7 Responses to Block your site’s search result pages

  1. Anil Agarwal
    Anil Agarwal  • 2 years ago

    Hi Michiel

    Please help us in knowing how we can get rid of thousands of search pages that are getting indexed into Google. You can check the same from here: https://www.google.com/search?q=site:https://bloggerspassion.com/search/&ei=VZeDW-GNLZGo9QPeyLqAAg&start=10&sa=N&filter=0&biw=1280&bih=726

    I’m unable to understand how they are getting created and they are not part of website (website layout), so unable to understand how I can add the meta noindex tag in these pages. Then I can remove the Disallow: */search/* command from robots.txt file.

    Please help here.

    Thanks

  2. Desmond SEO
    Desmond SEO  • 2 years ago

    Very Useful info, and yes it is not something that is always on the top of my mind for the very fact that I do use the Yoast SEO plug-in for every word press site we manage, but a good reminder for those sites without WP.
    Thanks for sharing.

  3. StudioWasabi
    StudioWasabi  • 2 years ago

    Very important not only for newbie but also for “experts” , who forgot these best practices…

  4. Pradeep Singh
    Pradeep Singh  • 2 years ago

    Hello,
    If I Search My blog name then Google show Page 1, Page 2 and so on in Snippet, I don’t want it, How to Disable It.
    Thanks

  5. Marie
    Marie  • 2 years ago

    But what about the fact that Google said that on long-term noindex on a page will lead to them not following links on that page (you have a good article about this https://yoast.com/pagination-seo-best-practices) ?

  6. Brad Neuendank
    Brad Neuendank  • 2 years ago

    What about adding rel=canonical (to the canonical version) to all these low-value internal search pages?

    • Michiel Heijmans

      I wouldn’t use canonicals for that. Noindex makes the most sense. Search results linking to search results is just not what your visitor would find the best experience either.