WordPress robots.txt Example

10 February 2012 by Joost de Valk » - 36 Comments

WordPress Robots.txt suggestions by Yoast

This post is outdated and while it might still be good reading, you can read our latest advice on WordPress robots.txt files here. Robots.txt is a way to tell a search engine which pages it’s allowed to spider, to “see”, and which pages it cannot “see”. Because of that, robots.txt differs from meta name=”robots” tags, »

Category: WordPress

Web Designer Mag should fix its SEO

30 December 2009 by Joost de Valk » - 57 Comments

Ok I just had to post this, as it’s too funny. I got a Google Alert this afternoon for this post, which mentioned one of my plugins as being listed by Web Designer Magazine. So, I Googled them, as the post didn’t link to them, and got this result: Notice something? I know I did: »

Category: WordPress

Preventing your site from being indexed, the right way

17 December 2009 by Joost de Valk » - 36 Comments

It keeps amazing me that I keep seeing people use robots.txt files to prevent sites from being indexed and thus showing up in the search engines. You know why it keeps amazing me? Because robots.txt doesn’t actually do the latter, even though it does prevent your site from being indexed. Let’s go through some terms »

Category: General SEO

PageRank sculpting – Siloing and more

22 March 2008 by Joost de Valk » - 81 Comments

PR sculpting seems to be “all the rage” at the moment. Tracking the conversation back seems to get me to an article by Dan Thies of september 4th last year, pointing back to an interview with Matt on SEOmoz. It’s been a whole load of buzz lately, coming up at SES again a few times »

Category: General SEO

Playing with the X-Robots-Tag HTTP header

20 January 2008 by Joost de Valk » - 31 Comments

Ever since the announcement on the Google Blog and more recently Yahoo’s announcement that they’ve enhanced their support for it, I’ve been meaning to play with the X-Robots-Tag header. This HTTP header allows you to do what you’d normally do in a robots meta tag, in an HTTP header, which has some pretty cool appliances. »

Category: General SEO

Extreme SEO Q&A

26 October 2007 by Joost de Valk » - 4 Comments

So yesterday I was on the panel for the last session of the day: an Extreme SEO Q&A, moderated by Ciaran Norris, and on that panel with me were Dixon Jones, Marcus Tandler and Jason Duke. We had a lot of fun, and loads of good questions came up. Some of the topics we touched »

Category: General SEO

The ultimate guide to the meta robots tag

12 October 2007 by Joost de Valk » - 11 Comments


With the meta robots tag you can control what search engine spiders do on your site. Learn what it does right here!

Category: General SEO

Feeds in the search results?

3 October 2007 by Joost de Valk » - 15 Comments

There was some discussion on one of the biggest Dutch online marketing sites as to whether RSS feeds were any good for your SEO. I gave a quite lengthy reaction to that in Dutch which I wanted to share with all of you as well. My opinion is that the fact that feeds are showing »

Category: General SEO