Crawl directives archives - Page 3 of 3

Recent Crawl directives articles

Preventing your site from being indexed, the right way

5 June 2017 | 9 Comments | Joost de Valk

We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result, their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt …

Read: "Preventing your site from being indexed, the right way"
robots meta


Nofollow layered navigation links?

30 January 2017 | 4 Comments | Joost de Valk

If you have a big eCommerce site with lots of products, layered navigation can help your users to narrow down their search results. Layered or faceted navigation is an advanced way of filtering, by providing groups or filters for (many) product attributes.  In this filtering process, you might create a lot of URLs though. The …

Read: "Nofollow layered navigation links?"


Playing with the X-Robots-Tag HTTP header

3 January 2017 | 2 Comments | Joost de Valk

Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains or other content search engines will be allowed to crawl. But did you know that there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss what the possibilities are and how this might be a better …

Read: "Playing with the X-Robots-Tag HTTP header"


Google Panda 4, and blocking your CSS & JS

19 June 2014 | 79 Comments | Joost de Valk

A month ago Google introduced its Panda 4.0 update. Over the last few weeks we’ve been able to “fix” a couple of sites that got hit in it. These sites both lost more than 50% of their search traffic in that update. When they returned, their previous position in the search results came back. Sounds too good to be …

Read: "Google Panda 4, and blocking your CSS & JS"
noindex a post with meta robots noindex