CYBER MONDAY SALE! Last chance to get a 30% discount on ALL products!

Want to learn about Crawl directives?

Take our Technical SEO training!

  • Part of our SEO academy Premium subscription!
  • Which technical SEO errors hurt your site?
  • Solve them and climb the rankings!
  • Improve your site speed on the go
More info

Or just read some posts about Crawl directives

The ultimate guide to robots.txt »

The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!


Must read articles about Crawl directives


Recent Crawl directives articles

Which pages to noindex or nofollow on your site?

Some of the pages of your site serve a purpose, but that purpose isn’t ranking in search engines or even getting traffic to your site. These pages need to be there, as glue for other pages or simply because regulations require them to be accessible on your website. If you regularly read our blog, you’ll know …

Read: "Which pages to noindex or nofollow on your site?"
noindex robots

What are sponsored, nofollow and ugc links, and why use them?

Links are an important part of SEO. Without links, Google (or other search engines) may not discover your pages, or might not think that they’re important. Sometimes, though, you might want Google not to follow a link. Or you might want to tell them a particular is sponsored, or added to your page by a …

Read: "What are sponsored, nofollow and ugc links, and why use them?"
outbound links sponsored nofollow and UGC

Noindex a post or page in WordPress, the easy way!

22 June 2020 | 18 Comments | Joost de Valk

Some posts and pages should not show up in search results. To make sure they don’t show up, you should tell search engines to exclude them. You do this with a meta robots noindex tag. Setting a page to noindex makes sure search engines never show it in their results. Here, we’ll explain how easy it …

Read: "Noindex a post or page in WordPress, the easy way!"
noindex a post with meta robots noindex

Yoast SEO & Ryte: Checking your site’s indexability

17 April 2020 | 28 Comments | Edwin Toonen

Your site needs to be up and running if you want to be found in search engines. If you aren’t blocking anything — deliberately or accidentally — search engine spiders can crawl and index it. You probably know that Yoast SEO has lots of options to determine what does and doesn’t need to be indexed, but …

Read: "Yoast SEO & Ryte: Checking your site’s indexability"
Crawl efficiency


Help, I’ve accidentally noindexed a post. What to do?

It can happen to anyone: You’re working on your site, fiddling on some posts here and there, and hit update when you’re done. After a while, you check back on how a post is doing and, to your dismay, it disappeared completely from the search engines! It turns out you’ve accidentally set a post or …

Read: "Help, I’ve accidentally noindexed a post. What to do?"
accidentally noindex post

WordPress robots.txt: Best-practice example for SEO

7 November 2019 | Jono Alderson

Your robots.txt file is a powerful tool when you’re working on a website’s SEO – but it should be handled with care. It allows you to deny search engines access to different files and folders, but often that’s not the best way to optimize your site. Here, we’ll explain how we think webmasters should use their …

Read: "WordPress robots.txt: Best-practice example for SEO"
noindex a post with meta robots noindex


What’s technical SEO? 8 technical aspects everyone should know

An SEO Basics post about technical SEO might seem like a contradiction in terms. Nevertheless, some basic knowledge about the more technical side of SEO can mean the difference between a high ranking site and a site that doesn’t rank at all. Technical SEO isn’t easy, but here we’ll explain – in layman’s language – …

Read: "What’s technical SEO? 8 technical aspects everyone should know"
Ask Yoast: technical SEO


Crawl directives

There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:

  • tell a search engine to not crawl a page at all;
  • not to use a page in its index after it has crawled it;
  • whether to follow or not to follow links on that page;
  • a lot of “minor” directives.

We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.