Why should you block your internal search result pages for Google? Well, how would you feel if you are in need of an answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed.
Google considers these search results pages to be of lower quality than your actual informational pages. That doesn’t mean these internal search pages are useless, but it makes sense to prevent Google from sending visitors to them.
Back in 2007
In 2007, Google’s representative Matt Cutts told us:
Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index. (Not all URLs that contains things like “/results” or “/search” are search results, of course.)
– Matt Cutts (2007)
Google’s Webmaster Guidelines backed this up, by stating that you should “Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.“
It was best practice to completely block Google from seeing or accessing these types of pages.
Modern best practices for blocking internal search pages
Google’s guidelines no longer mention search results pages specifically, but it’s still good practice to help Google to avoid crawling and indexing ‘infinite spaces’ like search results.
However, blocking internal search pages via your robots.txt isn’t the best way to do this. It’s much better to allow search engines to crawl these kinds of pages, but to prevent them from indexing them by using a meta robots tag with a value of
Yoast SEO will block internal search pages for you
If you’re on WordPress and using our plugin, you’re fine. We’ve got you covered. Your internal search results pages are noindexed by default in Yoast SEO. In addition, you can set a template for the page title of those pages, so the correct title will display in your browser tab or if your title is used in your theme.
You can set the template in the Search Appearance section of Yoast SEO. Just click on SEO › Search Appearance › Archives › Special pages in the left hand menu, and you’ll get this. This is the default template, but you can amend it, if you consider it necessary:
Most other content management systems allow for templates for your site’s search results as well, so adding a simple line of code to that template will suffice:
<meta name="robots" content="noindex,follow"/>
Meta robots AND robots.txt?
If you try to block internal search pages by adding that meta robots tag and disallowing these in your robots.txt, please think again. Just the meta robots will do. Otherwise, you’ll risk losing the link value of these pages (hence the
follow in the meta tag). If Google listens to your robots.txt, they will ignore the meta robots tag, right? And that’s not what you want. So just use the meta robots tag!
Back to you
Did you block your internal search results? And how did you do that? Go check for yourself! Any further insights or experiences are appreciated; just drop us a line in the comments.
Read more: Robots.txt: the ultimate guide »