robots.txt


WordPress robots.txt Example

Robots.txt is a way to tell a search engine which pages it’s allowed to spider, to “see”, and which pages it cannot “see”. Because of that, robots.txt differs from meta name=”robots” tags, which tell search engines on those individual pages, whether they can include them in their index or not. The difference is subtle, but…

Web Designer Mag should fix its SEO

Ok I just had to post this, as it’s too funny. I got a Google Alert this afternoon for this post, which mentioned one of my plugins as being listed by Web Designer Magazine. So, I Googled them, as the post didn’t link to them, and got this result: Notice something? I know I did:…

Preventing your site from being indexed, the right way

It keeps amazing me that I keep seeing people use robots.txt files to prevent sites from being indexed and thus showing up in the search engines. You know why it keeps amazing me? Because robots.txt doesn’t actually do the latter, even though it does prevent your site from being indexed. Let’s go through some terms…

PageRank sculpting – Siloing and more

PR sculpting seems to be “all the rage” at the moment. Tracking the conversation back seems to get me to an article by Dan Thies of september 4th last year, pointing back to an interview with Matt on SEOmoz. It’s been a whole load of buzz lately, coming up at SES again a few times…

Playing with the X-Robots-Tag HTTP header

Ever since the announcement on the Google Blog and more recently Yahoo’s announcement that they’ve enhanced their support for it, I’ve been meaning to play with the X-Robots-Tag header. This HTTP header allows you to do what you’d normally do in a robots meta tag, in an HTTP header, which has some pretty cool appliances.…

Extreme SEO Q&A

So yesterday I was on the panel for the last session of the day: an Extreme SEO Q&A, moderated by Ciaran Norris, and on that panel with me were Dixon Jones, Marcus Tandler and Jason Duke. We had a lot of fun, and loads of good questions came up. Some of the topics we touched…

Feeds in the search results?

There was some discussion on one of the biggest Dutch online marketing sites as to whether RSS feeds were any good for your SEO. I gave a quite lengthy reaction to that in Dutch which I wanted to share with all of you as well. My opinion is that the fact that feeds are showing…