Crawl directives archives


Crawl budget optimization

5 July 2016 by Joost de Valk » - 3 Comments

crawl budget

What is a crawl budget? Crawl budget is the number of pages Google will crawl on your site on any given day. This number varies slightly from day to day, but overall it’s relatively stable. The number of pages Google crawls, your “budget”, is generally determined by the size of your site, the “health” of your »

Category: Technical SEO
Tag:

robots.txt: the ultimate guide

17 May 2016 by Joost de Valk » - 5 Comments

noindex a post with meta robots noindex

The robots.txt file is a file you can use to tell search engines where they can and cannot go on your site. Learn how to use it to your advantage!

Category: Technical SEO
Tag:

rel=canonical: the ultimate guide

10 May 2016 by Joost de Valk » - 38 Comments

canonical urls as a solution for duplicate content

WordPress robots.txt example for great SEO

26 April 2016 by Joost de Valk »

noindex a post with meta robots noindex

The robots.txt file is a very powerful file if you’re working on a site’s SEO. At the same time, it also has to be used with care. It allows you to deny search engines access to certain files and folders, but that’s very often not what you want to do. Over the years, especially Google changed »

Categories: Technical SEO, WordPress
Tags: ,

hreflang: the ultimate guide

5 April 2016 by Joost de Valk » - 31 Comments


Google Panda 4, and blocking your CSS & JS

19 June 2014 by Joost de Valk » - 79 Comments

noindex a post with meta robots noindex

A month ago Google introduced its Panda 4.0 update. Over the last few weeks we’ve been able to “fix” a couple of sites that got hit in it. These sites both lost more than 50% of their search traffic in that update. When they returned, their previous position in the search results came back. Sounds too good to be »

Category: WordPress
Tags: , ,

The ultimate guide to the meta robots tag

12 October 2007 by Joost de Valk »

With the meta robots tag you can control what search engine spiders do on your site. Learn what it does right here!

Category: Technical SEO
Tag: