Crawl directives

There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:

  • tell a search engine to not crawl a page at all;
  • not to use a page in its index after it has crawled it;
  • whether to follow or not to follow links on that page;
  • a lot of “minor” directives.

We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.

Must read articles about Crawl directives

  • rel=canonical: the ultimate guide

    rel=canonical: the ultimate guide »

    The canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Learn how to use rel=canonical!

  • hreflang: the ultimate guide

    hreflang: the ultimate guide »

    This guide discusses what hreflang is, what it is for and gives in-depth information on how to implement it for your multilingual websites.

  • robots.txt: the ultimate guide

    robots.txt: the ultimate guide »

    The robots.txt file is a file that follows a strict standard. With the robots.txt you can tell search engines where they can and cannot go on your site. Learn how to use the robots.txt to your advantage to block the search engine from spidering parts of your site it doesn't need to visit!



Ask Yoast: Changes to your site and the search results

12 January 2018 by Joost de Valk - 17 Comments

Whenever you make some big changes to your website, for instance to your brand name, you’re probably eager for these changes to show in the search results. Unfortunately, it can take a while for Google to crawl your site again and until then, it will show the indexed version of your site in the results, …

Read: "Ask Yoast: Changes to your site and the search results"
ask yoast: changes visible in search results

Closing a spider trap: fix crawl inefficiencies

12 October 2017 by Joost de Valk - 4 Comments

We recently made some changes to how yoast.com is run as a shop and how it’s hosted. In that process, we accidentally removed our robots.txt file and caused a so-called spider trap to open. In this post, I’ll show you what a spider trap is, why it’s problematic and how you can find and fix …

Read: "Closing a spider trap: fix crawl inefficiencies"
spider traps

SEO for a new website: the very first things to do

11 July 2017 by Marieke van de Rakt - 36 Comments

How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. …

Read: "SEO for a new website: the very first things to do"
seo for a new website - first things to do

Preventing your site from being indexed, the right way

5 June 2017 by Joost de Valk - 9 Comments

We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt …

Read: "Preventing your site from being indexed, the right way"
googlebot prevent your site being indexed

Block your site’s search result pages

10 May 2017 by Michiel Heijmans - 14 Comments

Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal …

Read: "Block your site’s search result pages"
block internal search pages

SEO basics: What is crawlability?

20 February 2017 by Marieke van de Rakt - 6 Comments

Ranking in the search engines requires a website with flawless technical SEO. Luckily, the Yoast SEO plugin takes care of (almost) everything on your WordPress site. Still, if you really want to get most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, …

Read: "SEO basics: What is crawlability?"
what is crawlability

Ask Yoast: Should I redirect my affiliate links?

6 February 2017 by Joost de Valk - 1 Comment

There are several reasons for cloaking or redirecting affiliate links. For instance, it’s easier to work with affiliate links when you redirect them, plus you can make them look prettier. But do you know how to cloak affiliate links? We explained how the process works in one of our previous posts. This Ask Yoast is …

Read: "Ask Yoast: Should I redirect my affiliate links?"
Ask Yoast stopwords in focus keywords

Ask Yoast: Nofollow layered navigation links?

30 January 2017 by Joost de Valk - 4 Comments

If you have a big eCommerce site with lots of products, layered navigation can help your users to narrow down their search results. Layered or faceted navigation is an advanced way of filtering by providing groups of filters for (many) product attributes. In this filtering process, you might create a lot of URLs though, because the user …

Read: "Ask Yoast: Nofollow layered navigation links?"
Ask Yoast stopwords in focus keywords

How to cloak your affiliate links

24 January 2017 by Joost de Valk - 11 Comments

We regularly consult for sites that monetize, in part, with affiliate links. We usually advise people to redirect affiliate links. In the past, we noticed that there wasn’t a proper script available online that could handle this for us, so we created one to tackle this problem. In this post, I explain how you can get …

Read: "How to cloak your affiliate links"


Browse other categories