Want to learn about Crawl directives?

Take our Technical SEO training!

  • Which technical SEO errors hurt your site?
  • Solve them and climb the rankings!
  • Improve your site speed on the go
  • On-demand SEO training by Yoast
More info

Or just read some posts about crawl directives

rel=canonical: the ultimate guide »

The canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Learn how to use rel=canonical!


Must read articles about Crawl directives

Recent Crawl directives articles


Block your site’s search result pages

16 August 2018 by Michiel Heijmans - 7 Comments

Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal …

Read: "Block your site’s search result pages"
block internal search pages




Media / attachment URL: what to do with them?

30 May 2018 by Joost de Valk - 412 Comments

In our major Yoast SEO 7.0 update, there was a bug concerning attachment URLs. We quickly resolved the bug, but some people have suffered anyhow (because they updated before our patch). This post serves both as a warning and an apology. We want to ask all of you to check whether your settings for the …

Read: "Media / attachment URL: what to do with them?"
attachment pages fix


Yoast SEO & Ryte: Checking your site’s indexability

28 March 2018 by Edwin Toonen - 9 Comments

Your site needs to be up and running if you want to be found in search engines. If you aren’t blocking anything — deliberately or accidentally — search engine spiders can crawl and index it. You probably know that Yoast SEO has lots of options to determine what does and doesn’t need to be indexed, but …

Read: "Yoast SEO & Ryte: Checking your site’s indexability"
Crawl efficiency



Crawl directives

There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:

  • tell a search engine to not crawl a page at all;
  • not to use a page in its index after it has crawled it;
  • whether to follow or not to follow links on that page;
  • a lot of “minor” directives.

We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.