Crawl directives

There are multiple ways to tell search engines how to behave on your site. These are called “crawl directives”. They allow you to:

  • tell a search engine to not crawl a page at all;
  • not to use a page in its index after it has crawled it;
  • whether to follow or not to follow links on that page;
  • a lot of “minor” directives.

We write a lot about these crawl directives as they are a very important weapon in an SEO’s arsenal. We try to keep these articles up to date as standards and best practices evolve.


Must read articles about Crawl directives

SEO basics: What is crawlability?

20 February 2017 by Marieke van de Rakt » - 5 Comments

what is crawlability

Ranking in the search engines requires a website with flawless technical SEO. Luckily, the Yoast SEO plugin takes care (of almost) everything on your WordPress site. Still, if you really want to get most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, »

Category: Technical SEO
Tag:

Ask Yoast: should I redirect my affiliate links?

6 February 2017 by Joost de Valk » - 2 Comments

Ask Yoast stopwords in focus keywords

There are several reasons for cloaking or redirecting affiliate links. For instance, it’s easier to work with affiliate links when you redirect them, plus you can make them look prettier. But do you know how to cloak affiliate links? We explained how the process works in one of our previous posts. This Ask Yoast is »

Category: Technical SEO
Tags: , , ,

Ask Yoast: nofollow layered navigation links?

30 January 2017 by Joost de Valk » - 4 Comments

Ask Yoast stopwords in focus keywords

If you have a big eCommerce site with lots of products, layered navigation can help your users to narrow down their search results. Layered or faceted navigation is an advanced way of filtering by providing groups of filters for (many) product attributes. In this filtering process, you might create a lot of URLs though, because the user »

Categories: eCommerce, Technical SEO
Tags: , ,

How to cloak your affiliate links

24 January 2017 by Joost de Valk » - 11 Comments

cloak affiliate links

We regularly consult for sites that monetize, in part, with affiliate links. We usually advise people to redirect affiliate links. In the past, we noticed that there wasn’t a proper script available online that could handle this for us, so we created one to tackle this problem. In this post, I explain how you can get »

Category: Technical SEO
Tags: , ,

Playing with the X-Robots-Tag HTTP header

3 January 2017 by Joost de Valk » - 2 Comments

cloak affiliate links

Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains or other content search engines will be allowed to crawl. But did you know that there’s also such a thing as the X-Robots-Tag HTTP header? In this post we’ll discuss what the possibilities are and how this might be »

Category: Technical SEO
Tags: , ,

Don’t block CSS and JS files

2 January 2017 by Michiel Heijmans » - 4 Comments

you should not block your CSS and JS files

In 2015, Google Search Console already started to warn webmasters actively not to block CSS and JS files. In 2014, we told you the same thing: don’t block CSS and JS files. We feel the need to repeat this message now. We’re currently working on the websites of our first Yoast SEO Care customers, and this »

Category: Technical SEO
Tag:

Crawl budget optimization

5 July 2016 by Joost de Valk » - 3 Comments

crawl budget

What is a crawl budget? Crawl budget is the number of pages Google will crawl on your site on any given day. This number varies slightly from day to day, but overall it’s relatively stable. The number of pages Google crawls, your “budget”, is generally determined by the size of your site, the “health” of your »

Category: Technical SEO
Tag:

robots.txt: the ultimate guide

17 May 2016 by Joost de Valk » - 5 Comments

noindex a post with meta robots noindex

The robots.txt file is one of the primary ways of telling a search engine where it can and can’t go on your website. All major search engines support the basic functionality it offers. There are some extra rules that are used by a few search engines which can be useful too. This guide covers all »

Category: Technical SEO
Tag:

rel=canonical: the ultimate guide

10 May 2016 by Joost de Valk » - 38 Comments

canonical urls as a solution for duplicate content

A canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Sometimes you have products or content that is accessible under multiple URLs, or even on multiple websites. Using a canonical URL (an HTML link tag with attribute rel=canonical) these can exist without harming your rankings. What »

Categories: Content SEO, Technical SEO
Tags: , ,

WordPress robots.txt example for great SEO

26 April 2016 by Joost de Valk »

noindex a post with meta robots noindex

The robots.txt file is a very powerful file if you’re working on a site’s SEO. At the same time, it also has to be used with care. It allows you to deny search engines access to certain files and folders, but that’s very often not what you want to do. Over the years, especially Google changed »

Categories: Technical SEO, WordPress
Tags: ,