24-hr offer! $50 discount on our Content SEO training bundle! Get it TODAY!

Crawl directives

There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:

  • tell a search engine to not crawl a page at all;
  • not to use a page in its index after it has crawled it;
  • whether to follow or not to follow links on that page;
  • a lot of “minor” directives.

We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.

The ultimate guide to robots.txt »

The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!


Must read articles about Crawl directives


Recent Crawl directives articles

What’s technical SEO? 8 technical aspects everyone should know

An SEO Basics post about technical SEO might seem like a contradiction in terms. Nevertheless, some basic knowledge about the more technical side of SEO can mean the difference between a high ranking site and a site that doesn’t rank at all. Technical SEO isn’t easy, but here we’ll explain – in layman’s language – …

Read: "What’s technical SEO? 8 technical aspects everyone should know"
Ask Yoast: technical SEO





Block your site’s search result pages

16 August 2018 | 7 Comments | Michiel Heijmans

Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal …

Read: "Block your site’s search result pages"
block internal search pages


Media / attachment URL: what to do with them?

In our major Yoast SEO 7.0 update, there was a bug concerning attachment URLs. We quickly resolved the bug, but some people have suffered anyhow (because they updated before our patch). This post serves both as a warning and an apology. We want to ask all of you to check whether your settings for the …

Read: "Media / attachment URL: what to do with them?"
attachment pages fix


Yoast SEO & Ryte: Checking your site’s indexability

28 March 2018 | 9 Comments | Edwin Toonen

Your site needs to be up and running if you want to be found in search engines. If you aren’t blocking anything — deliberately or accidentally — search engine spiders can crawl and index it. You probably know that Yoast SEO has lots of options to determine what does and doesn’t need to be indexed, but …

Read: "Yoast SEO & Ryte: Checking your site’s indexability"
Crawl efficiency

Crawl directives

There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:

  • tell a search engine to not crawl a page at all;
  • not to use a page in its index after it has crawled it;
  • whether to follow or not to follow links on that page;
  • a lot of “minor” directives.

We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.