NEW Basic SEO training! Get $30 discount now!

Crawl directives

There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:

  • tell a search engine to not crawl a page at all;
  • not to use a page in its index after it has crawled it;
  • whether to follow or not to follow links on that page;
  • a lot of “minor” directives.

We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.

Must read articles about Crawl directives

  • rel=canonical: the ultimate guide

    rel=canonical: the ultimate guide »

    The canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Learn how to use rel=canonical!

  • hreflang: the ultimate guide

    hreflang: the ultimate guide »

    This guide discusses what hreflang is, what it is for and gives in-depth information on how to implement it for your multilingual websites.

  • robots.txt: the ultimate guide

    robots.txt: the ultimate guide »

    The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!




Media / attachment URL: what to do with them?

30 May 2018 by Joost de Valk - 412 Comments

In our major Yoast SEO 7.0 update, there was a bug concerning attachment URLs. We quickly resolved the bug, but some people have suffered anyhow (because they updated before our patch). This post serves both as a warning and an apology. We want to ask all of you to check whether your settings for the …

Read: "Media / attachment URL: what to do with them?"
attachment pages fix


Yoast SEO & Ryte: Checking your site’s indexability

28 March 2018 by Edwin Toonen - 9 Comments

Your site needs to be up and running if you want to be found in search engines. If you aren’t blocking anything — deliberately or accidentally — search engine spiders can crawl and index it. You probably know that Yoast SEO has lots of options to determine what does and doesn’t need to be indexed, but …

Read: "Yoast SEO & Ryte: Checking your site’s indexability"
Crawl efficiency

Pagination & SEO: best practices

15 March 2018 by Joost de Valk - 40 Comments

Paginated archives have long been a topic of discussion in the SEO community. Over time, best practices for optimization have evolved, and we now have pretty clear definitions. This post explains what these best practices are. It’s good to know that Yoast SEO applies all these rules to every archive with pagination. Indicate that an …

Read: "Pagination & SEO: best practices"
Category pages

Which pages should I noindex or nofollow?

Some of the pages of your site serve a purpose, but that purpose isn’t ranking in search engines or even getting traffic to your site. These pages need to be there as glue for other pages, or simply because whatever regulations require them to be accessible on your website. As a regular visitor to our website, …

Read: "Which pages should I noindex or nofollow?"
noindex robots


Ask Yoast: Changes to your site and the search results

12 January 2018 by Joost de Valk - 17 Comments

Whenever you make some big changes to your website, for instance to your brand name, you’re probably eager for these changes to show in the search results. Unfortunately, it can take a while for Google to crawl your site again and until then, it will show the indexed version of your site in the results, …

Read: "Ask Yoast: Changes to your site and the search results"

Closing a spider trap: fix crawl inefficiencies

12 October 2017 by Joost de Valk - 4 Comments

We recently made some changes to how yoast.com is run as a shop and how it’s hosted. In that process, we accidentally removed our robots.txt file and caused a so-called spider trap to open. In this post, I’ll show you what a spider trap is, why it’s problematic and how you can find and fix …

Read: "Closing a spider trap: fix crawl inefficiencies"
spider traps


Browse other categories