Google sometimes lists URLs that it's not allowed to spider, because it's blocked by robots.txt, because a lot of links point to a URL. A good example of this is a search for [RTL Nieuws] (disclosure: RTL is a client of mine). rtlnieuws.nl 301 redirects to the news section of rtl.nl. But... rtlnieuws.nl/robots.txt exists... And has the following content:

User-agent: *
Disallow: /

Because of that, the links towards rtlnieuws.nl don't count toward the news section on rtl.nl, and Google displays rtlnieuws.nl in the search results. This is unwanted behavior that we're trying to fix but for now it's a good example of what I wanted to explain. By blocking /wp-admin/ and /trackback/ in your robots.txt, you're not preventing them from showing up.

Unfortunately, recently the /wp-admin/ block was added to WordPress core, because of this Trac ticket. In the discussion on that ticket, I've proposed another solution in this patch. This solution involves sending an X-Robots-Tag header, which is the HTTP header equivalent of a meta name="robots" tag. This would in fact remove all wp-admin directories from Google search results.

WordPress Robots.txt blocking Search results and Feeds

There are two other sections which are blocked in the suggested robots.txt, /*?, which blocks everything with a question mark and as such all search results, and */feed/, which blocks all feeds. The first is not a good idea because if someone were to link to your search results, you wouldn't benefit from those links.

A better solution would be to add a <meta name="robots" content="noindex, follow"> tag to those search results pages, as it would prevent the search results from rankings but would allow the link "juice" to flow through to the returned posts and pages. This is what my WordPress SEO plugin does as soon as you enable it. It also does this for wp-admin and login and registration pages.

I'm aware that that is different from Google's guidelines on this topic at the moment, which state:

Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines.

I've reached out to Google to get clarification on whether they would say my solution is acceptable as well, or perhaps even better :) .

Blocking /feed/ is a bad idea because an RSS feed is actually a valid sitemap for Google. Blocking it would prevent Google from using that to find new content on your site. So, my suggested robots.txt for WordPress is actually a lot smaller than the Codex one. I only have this:

User-Agent: *
Disallow: /wp-content/plugins/

I block the plugins directory because some plugin developers have the annoying habit of adding index.php files to their plugin directories that link back to their websites. For all other parts of WordPress, there are better solutions for blocking.

The other WordPress Robots.txt suggestions

The other sections of the robots.txt as suggested are a bit old and no longer needed. Digg mirror is something for us old guys who remember when Digg used to send loads of traffic, Googlebot Image and Media Partner are still there but if you only have the above in your robots.txt you don't need specific lines for them in your WordPress robots.txt file.

WordPress robots.txt Example is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Notice something? I know I did: there's no description there. So I clicked on and checked the page to see what could cause that, finding it was easy:

<meta name='robots' content='noindex,nofollow' />

It was listed right above the EditURI line that WordPress puts in by default, telling me they forgot to uncheck a box (that get's checked by default on some auto installers, I've been told on Twitter):

One issue though... If Google had actually seen that tag, it wouldn't have listed the site at all. So there had to be something else. And of course, there is: meet Webdesigner Magazine's robots.txt:

User-agent: *
Disallow: /

Sitemap: http://www.webdesignermag.co.uk/sitemap.xml.gz

The first two lines there prevent Google from indexing the site entirely. But it just became funnier... The fourth line was added by a plugin they're running, pointing search engine bots at their XML sitemap. Yes, the same bots they just forbade entry to their entire site... If you check out their XML sitemap, you'd notice they're running Arne Brachold's Google XML Sitemap Generator...

So: they installed an XML sitemap plugin, let's see what else they did: ah... Cool! They're running All In One SEO too! Now AIOSEO is "ok", except: it doesn't warn you for stupidities like these... Might be a darn good feature for an SEO plugin to check whether your site is actually allowing crawlers to come in, don't you think? Ah well, nothing can be perfect.

Anyway, this tells me one thing: Web Designer Mag needs help with their WordPress install. And they need it bad. If you work for them, and read this, feel free to contact me on Twitter or through the contact form. I'd be happy to help!

Web Designer Mag should fix its SEO is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Let's go through some terms here:

Indexed / Indexing
The process of downloading a site or a page's content to the server of the search engine, thereby adding it to it's "index".

Ranking / Listing / Showing
Showing a site in the search result pages (aka SERPs).

So, while the most common process goes from Indexing to Listing, a site doesn't have to be indexed to be listed. If a link points at a page, domain or wherever, that link will be followed. If the robots.txt on that domain prevents the search engine from indexing that page, it'll still show the URL in the results if it can gather from other variables that it might be worth looking at.

If my explanation above doesn't make sense, have a look at Matt Cutt's video explanation:

Click here to view the embedded video.

So, if you want to effectively hide pages from the search engines, and this might seem contradictory, you need them to index those pages. Why? Because when they index those pages, you can tell them not to List them. There's two ways of doing that: by using robots meta tags, like this (and I've got an article on robots meta tags that's more extensive):

<meta name="robots" content="noindex,nofollow"/>

The issue with a tag like that is that you have to add it to each and every page. That's why the search engines came up with the X-Robots-Tag HTTP header. This allows you to specify an HTTP header called X-Robots-Tag, and set the value as you would the meta robots tags value. The cool thing about this is that you can do it for an entire site. So, if your site is running on Apache, and mod_headers is enabled (it usually is), you could add the following single line to your .htaccess file:

Header set X-Robots-Tag "noindex, nofollow"

And it'd have the effect that that entire site can be indexed, but will never be shown in the search results. So, get rid of that robots.txt file with Disallow: / in it, and use the X-Robots-Tag instead!

Preventing your site from being indexed, the right way is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

In the interview that started the whole discussion again, Matt said:

The nofollow attribute is just a mechanism that gives webmasters the ability to modify PageRank flow at link-level granularity. Plenty of other mechanisms would also work (e.g. a link through a page that is robot.txt'ed out), but nofollow on individual links is simpler for some folks to use. There's no stigma to using nofollow, even on your own internal links; for Google, nofollow'ed links are dropped out of our link graph; we don't even use such links for discovery. By the way, the nofollow meta tag does that same thing, but at a page level.

This inspired Dan Thies to write his article, which is, in my opinion, both very good and also a bit flawed. Dan says:"That's the key point. Getting more of your important pages indexed." And I simply, do not agree, most of the other stuff is very true and valuable though. Let's get back to the basic theory of how you should create a site structure and theme it correctly: siloing.

Siloing

photo credit: twob

One of the oldest articles I found on siloing is on Bruce Clay's site, in an article from 2005.

The idea (and yes I'm oversimplifying a bit now) is that you only link to pages on your site with the same theme, to make it easier to rank for keywords and keyword groups. A quote:

Siloing resolves this problem by allowing you to achieve high search engine placement both for general and targeted keyword phrases through themed vertical page linking and/or construction.

This article talks about only linking to pages that you really should be linking to. Of course, this is still the best practice, and if you totally lived by that, you wouldn't need to nofollow any links. However if you, for any reason whatsoever (like management that doesn't get it, weird laws / lawyers, or conversion / up-selling reasons), have to link to another, unrelated, page, nofollow is the tool you could use to still abide by those siloing laws.

Nofollow != untrusted

In the beginning of this nofollow discussion, some people I really like and respect, like Greg Boser, Todd Friesen and Dave Naylor, were saying that you should really not use nofollow internally, using the argument: "why would you want to tell search engines that you don't trust certain pages on your site?"

Well, it's not about trust (anymore), as Greg admitted in an interview with Mike McDonald at SES recently, now that Matt has come out and said that. Others are still whining though, in part I think because either they don't get it, or they don't think other people will get it and want people to focus on different things that are more important, like site structure. In part, I agree. PageRank sculpting like that is not something for the faint of heart, or the SEO rookie. It IS however a valuable tool when you actually know what you're doing and have a lot of juice to play around with.

So what is it about then?

As my buddy Michael Gray said: why would Apple want to rank for [contact us]? Doesn't that single ranking imply a wasted opportunity to rank for a few more products they actually sell?

A quote from Matt at DaveN's blog is very important here:

Nofollowing your internals (PageRank sculpting - JdV) can affect your ranking in Google, but it's a 2nd order effect.

Anyway, at Onetomarket we've been doing this for at least 4 years. We used javascript links before nofollow was around, and we know that it works because we tested it. It's not specifically about getting more pages indexed, it's about getting those pages indexed that matter to you, and about, as Greg also pointed out in the video interview above, getting as much pages indexed as your overall PageRank can handle. You don't want to "spread yourself too thin".

PageRank sculpting is more then Nofollow

You really need to know that at some point, you'll need more then nofollow. You'll need page per page control of the robots meta tag, and you will probably be using it to noindex, follow category / tag / archive pages, in favor of single pages. That, combined with not giving too much linklove to these pages, is the essence of third level push.

It's also very important to be honest with yourself: "Do I really need this page / this set of pages in the index right now?" It might be wiser to make sure you have a smaller amount of pages in the index, throwing the deeper pages out and allowing yourself to actually rank with the pages higher up in your site's structure.

Those two uses of nofollow, together with the use in siloing of nofollowing links to unrelated pages, make it a very powerful tool. You can be a good SEO without using it, and a lot of times you can probably make more money by focussing on other things, but anyone saying that it's bad advice or nonsense, doesn't know what he or she is talking about, and should think twice before writing openly that people should not follow that advice.

PageRank sculpting – Siloing and more is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

As Sebastian explained in an excellent post on SEOmoz, there are two different kinds of directives: crawler directives and indexer directives.

Crawler directives
The robots.txt file only contains the so called Crawler directives, telling search engines, identified by their User-agent:, where they are not allowed to go by using Disallow: and where they can (and should) go by using Allow:, and by pointing them at a Sitemap:.

As Sebastian pointed out and explains thoroughly in another brilliant post, pages that search engines aren't allowed to spider, can still show up in the search results, when they have enough links pointing at them. This basically means that if you want to really hide something from the search engines and thus from people using search, robots.txt just isn't good enough.

Indexer directives
Indexer directives are directives that are, even with the birth of the X-Robots-Tag, set on a per page or even per element basis. Up until July 2007, there were two: the microformat rel="nofollow", which means that that link should not pass authority / PageRank, and the Meta Robots tag.

With the Meta Robots tag, you can really prevent search engines from showing the pages you block in the search results. You can reach the same with the relatively new X-Robots-Tag HTTP header. If you don't know what an HTTP header is, I'd suggest reading the Wikipedia page on it, but in short: look at it as the envelope around your content. This HTTP header is better than the meta robots tag for a couple of reasons, one of them is that you can send those headers for other documents too. So, let's get into some examples.

Example uses of the X-Robots-Tag

If you want to prevent search engines from showing files you've generated with PHP, add the following in the header file:

header("X-Robots-Tag: noindex", true);

This would not prevent search engines from following the links on those pages, if you want to do that, do the following:

header("X-Robots-Tag: noindex, nofollow", true);

But doing it in PHP is probably not the easiest use for this kind of thing. I myself greatly prefer setting headers in Apache, when possible. Consider, for instance, preventing search engines from caching / showing a preview for all .doc files on your domain, you would only have to do the following:

<FilesMatch "\.doc$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or, if you'd want to do this for both .doc and .pdf files:

<FilesMatch "\.(doc|pdf)$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or another case, your robots.txt file itself is showing up in the search results. Adding this to your Apache config or your .htaccess file would solve that:

<FilesMatch "robots\.txt">
Header set X-Robots-Tag "noindex"
</FilesMatch>

I had a slight uncomfortable feeling when writing this down, so I e-mailed Matt Cutts asking the following: "<snip> would that mean that you will still fetch it for robots.txt purposes, but won't show it in the index?". I'm waiting for him to answer, and will add his response here once I have it.

Tools
I've quickly created a bookmarklet which shows all the headers for a page (works in Moz browsers only I think, and a Greasemonkey script which pops up when a page is using an X-Robots-Tag header.

Conclusion
As you can see, if you combine the examples above with the stuff you can learn from for instance AskApache's .htaccess tutorial, the X-Robots-Tag HTTP header becomes a very powerful tool. Use it wisely and with caution, as you won't be the first to block your entire site by accident, but it's a great addition to your toolset if you know how to use it.

Playing with the X-Robots-Tag HTTP header is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Buying links or traffic?
We had an interesting discussion on paid links, and whether we'd buy links or not to start a site. Dixon said he looked at it as buying traffic, not pagerank, which is probably a smart way of looking at it.

What to do with affiliate links
Either cloak them, or use a redirect script on your site, which you're blocking for bots through robots.txt. To be absolutely certain, make sure that if an SE bot opens that script anyway, it redirects into your own site and not to the merchant.

Hosting
If you want to rank in the UK (or in Germany or Spain for that matter) host the pages you want to rank within the country you're aiming for. Just a .co.uk isn't good enough, you have to host it in the UK as well.

"I got some bad links and my site's rankings went down"
This was an interesting question from someone from the public, who said his rankings went down after getting links from some trade organizations. We were all highly doubting that that was his problem, until I found a link directory hiding on his site, which had nothing to do with the subject of the rest of his site. That directory went up round about the same time as he got those links. We advised him to remove that immediatly.

Session ID´s
We had some questions about sessionid's and of course we all agreed: get rid of those, and 301 redirect those indexed out.

Starting with a freshly registered domain
The three other guys where quite open about this: don't. Buy an old site, and use it, or 301 redirect it into your new domain.

I probably forgot quite a bit, but as you can see we touched on loads of stuff, and it was really loads of fun.

Extreme SEO Q&A is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

My opinion is that the fact that feeds are showing up in the search results is a bug, and it's completely not user friendly at all. In my opinion they do lead to duplicate content problems too, so there's really only one thing you can do, and that's block 'em. But I'd like to block those feeds without loosing the nice side effect of their links going into my posts.

The problem with using a noindex tag in your feed's head section is that the combination noindex,follow does not seem to be supported in that at the moment. I've been trying to get confirmation on that but have had no luck so far. The other problem is that you'll have to adapt the platform that creates your RSS feeds for you for that and that might be troublesome.

Introducing the Robots-X-Tag HTTP header

Google has said though, that you can use "any supported META tag" as a value for the X-Robots-Tag HTTP header. With that HTTP header you should be able to control whether a file can be indexed, has to have a snippet, etc. So you should also be able to add a noindex,follow HTTP header to your feed, indicating to Google that they should just follow the links in the feed, and not index the feed itself. You could arrange that from within your server config, which would look something like this in Apache:

<Directory /feed/>
Header append X-Robots-Tag "noindex,follow"
</Directory>

Here's somewhat more info on Header append.

If you're afraid blocking indexation of your feed might cause you to loose traffic from Google Blogsearch and/or Technorati, it won't. Google Blogsearch uses FeedFetcher, which doesn't observe robots.txt, and neither does Technorati. They both seem to be under the impression that pinging a blog search engine is enough consent to get it indexed, while others have suggested that pinging Technorati on behalf of others might be a nice way of improving your Technorati authority.

In the end, the X-Robots-Tag seems to be quite promising. There's a catch though, FeedBurner does not support it yet at this moment, which makes it pretty hard for everyone, including me, serving their feed through FeedBurner.

Feeds in the search results? is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!