But today, I can tell you that we've got something new, and better. Tracking rankings is hard, especially because Google isn't too fond of people scraping their search results. Because of that, running daily rankings was a pain, but the data we all secretly wanted was in there, we've all known it for ages: visitor data, combined with rankings, day for day, week for week.

Let me show you what I wanted, all this time:

This isn't some mockup, this is real stuff. A rankings tracker called SEO Rank Monitor, that uses the Google Analytics API to combine rankings data with visitor data, showing you what the difference really is between position 1 and position 2. It's also the fastest rankings tracker I've ever seen, ranking a few dozen keywords for me in a few minutes, and then on a daily basis after that, giving some awesome ranking charts:

It also allows you to add your competitors, so you can easily track their goings on your keywords as well. Now all experienced SEO's know they should work on keyword groups more than they should work on single keywords, and thus this new ranking tool allows you to tag your keywords, easily allowing you to check rankings for the group:

As you can see it also allows you to export the keywords to a CSV file. There's more: tracking the number of backlinks to your domain according to the search engines you selected, the number of indexed pages etc. Also, because this isn't a rank tracker that runs on your own machine, it's not clogging up resources and you can check your rankings from everywhere you want.

The tool is paid, of course, but you can try it for free for a month. The normal pricing is $39 a month for the Pro package and $19 a month for the Basic package. With the Pro package you can track 2,500 keywords and 10 domains, with the Basic package you're limited to 1 domain and 100 keywords. In other words: the Pro package is well worth the money. Now there's something even better: I've negotiated a discount code with which you'll get $10 a month off, for as long as you decide to keep your subscription! The discount code should be easy to remember: "yoast" is all you need.

So go check it out, and should you decide to sign up, don't forget to use the discount code "yoast"!

Rankings that make sense is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

It's pretty simple to use and I really suggest you go check it out!

SEO Blogging Tools is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Click here to view the embedded video.

SEOBook SEO Toolbar is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

So I started searching for a way to disable. First I found this post , via Saad Kamal, on Google Operating System, but that didn't work to well for me. This greasemonkey based solution is a tad bit better, but still only disables the CSS, it doesn't remove the functionality, just hides it.

Then I stumbled upon this thread.

That thread told me to add hl=all to the query string, and that would indeed disable SearchWiki, at least for now. So I've immediately created a new instance of my Google Disable Personalized Search plugin, which will disable SearchWiki as well.

If you already have the previous version of that search plugin installed, do the following in Firefox:

• click on the favicon next to the search box, and see the following appear:
  
• Click on Manage, and remove the Google.com Unpersonalized search one.
• Now you can go to the search plugin page and install the new one.

This hack works only for english results, I'll try to find a more lasting solution, which works for the other languages as well!

Disabling SearchWiki is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Google Tutor has released a very cool Google Insights bookmarklet:

Once installed you should see a bookmark cleverly titled ‘Google Insights Bookmarklet’ on your bookmarks menu. To use the bookmarklet simply do a search at Google, Yahoo, or Live Search, then click Google Insights Bookmarklet. The bookmarklet will automatically open Google Insights showing the stats for the keyword you were just looking up at the search engine.

Katz Web Design has released an SEO Stylesheet, which gives you numbered results even without plugins / extensions. Best thing? It works in my favorite browser: Safari.

Google Tutor has told me that he'll be releasing more nice bookmarklets, so subscribe to his feed!

Cool SEO tools roundup is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

SEO Link Analysis 1.0.3! is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Crawl prioritization: Search engine robots can't sign in or register as a member on your forum, so there's no reason to invite Googlebot to follow "register here" or "sign in" links. Using nofollow on these links enables Googlebot to crawl other pages you'd prefer to see in Google's index. However, a solid information architecture — intuitive navigation, user- and search-engine-friendly URLs, and so on — is likely to be a far more productive use of resources than focusing on crawl prioritization via nofollowed links

Now I've always said the same about good site architecture etc. But Google basically tells us here that it's ok to use nofollow for PageRank sculpting. Now I saw from the notes of the Bot Herding session at SMX Advanced yesterday that Adam Audette complained that PR sculpting was aimed way too much at Google. Well, Adam, I know you people don't, but we live in a Google world here in western Europe, where Google has a 95% share of all searches.

So: use nofollow, use it wisely, it's one of the authorized tools in our arsenal as SEO's.

PageRank Sculpting with nofollow, the final words is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

So that's 1 or 2 clicks a day... Now check out the stats for css3.info on that term, which holds the #1 and #2 position, with sitelinks:

6,600 visitors where using the term css3 as or as a part of their search query, and clicked on the organic results for css3.info in the last month. That's 220 a day. So either the ratio of a #1 and #2 ranking with sitelinks vs. an AdWords ad is 110-220 to 1, or the data in the traffic estimator is crap...

Traffic estimation with Google’s traffic estimator is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Click here to view the embedded video.

Cool new Rank Checker by SEObook is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

In the interview that started the whole discussion again, Matt said:

The nofollow attribute is just a mechanism that gives webmasters the ability to modify PageRank flow at link-level granularity. Plenty of other mechanisms would also work (e.g. a link through a page that is robot.txt'ed out), but nofollow on individual links is simpler for some folks to use. There's no stigma to using nofollow, even on your own internal links; for Google, nofollow'ed links are dropped out of our link graph; we don't even use such links for discovery. By the way, the nofollow meta tag does that same thing, but at a page level.

This inspired Dan Thies to write his article, which is, in my opinion, both very good and also a bit flawed. Dan says:"That's the key point. Getting more of your important pages indexed." And I simply, do not agree, most of the other stuff is very true and valuable though. Let's get back to the basic theory of how you should create a site structure and theme it correctly: siloing.

Siloing

photo credit: twob

One of the oldest articles I found on siloing is on Bruce Clay's site, in an article from 2005.

The idea (and yes I'm oversimplifying a bit now) is that you only link to pages on your site with the same theme, to make it easier to rank for keywords and keyword groups. A quote:

Siloing resolves this problem by allowing you to achieve high search engine placement both for general and targeted keyword phrases through themed vertical page linking and/or construction.

This article talks about only linking to pages that you really should be linking to. Of course, this is still the best practice, and if you totally lived by that, you wouldn't need to nofollow any links. However if you, for any reason whatsoever (like management that doesn't get it, weird laws / lawyers, or conversion / up-selling reasons), have to link to another, unrelated, page, nofollow is the tool you could use to still abide by those siloing laws.

Nofollow != untrusted

In the beginning of this nofollow discussion, some people I really like and respect, like Greg Boser, Todd Friesen and Dave Naylor, were saying that you should really not use nofollow internally, using the argument: "why would you want to tell search engines that you don't trust certain pages on your site?"

Well, it's not about trust (anymore), as Greg admitted in an interview with Mike McDonald at SES recently, now that Matt has come out and said that. Others are still whining though, in part I think because either they don't get it, or they don't think other people will get it and want people to focus on different things that are more important, like site structure. In part, I agree. PageRank sculpting like that is not something for the faint of heart, or the SEO rookie. It IS however a valuable tool when you actually know what you're doing and have a lot of juice to play around with.

So what is it about then?

As my buddy Michael Gray said: why would Apple want to rank for [contact us]? Doesn't that single ranking imply a wasted opportunity to rank for a few more products they actually sell?

A quote from Matt at DaveN's blog is very important here:

Nofollowing your internals (PageRank sculpting - JdV) can affect your ranking in Google, but it's a 2nd order effect.

Anyway, at Onetomarket we've been doing this for at least 4 years. We used javascript links before nofollow was around, and we know that it works because we tested it. It's not specifically about getting more pages indexed, it's about getting those pages indexed that matter to you, and about, as Greg also pointed out in the video interview above, getting as much pages indexed as your overall PageRank can handle. You don't want to "spread yourself too thin".

PageRank sculpting is more then Nofollow

You really need to know that at some point, you'll need more then nofollow. You'll need page per page control of the robots meta tag, and you will probably be using it to noindex, follow category / tag / archive pages, in favor of single pages. That, combined with not giving too much linklove to these pages, is the essence of third level push.

It's also very important to be honest with yourself: "Do I really need this page / this set of pages in the index right now?" It might be wiser to make sure you have a smaller amount of pages in the index, throwing the deeper pages out and allowing yourself to actually rank with the pages higher up in your site's structure.

Those two uses of nofollow, together with the use in siloing of nofollowing links to unrelated pages, make it a very powerful tool. You can be a good SEO without using it, and a lot of times you can probably make more money by focussing on other things, but anyone saying that it's bad advice or nonsense, doesn't know what he or she is talking about, and should think twice before writing openly that people should not follow that advice.

PageRank sculpting – Siloing and more is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

I liked the tool back then, and provided some feedback to Dan. Today Dan e-mailed me that Crawl Score was now released in version 1, and immediately ordered a sponsored review from me to show it to you, my readers. I of course happily said yes to that, and started looking at the tool again.

What does it do?
Well, it crawls your site, and creates pretty reports from the data it gathers while doing that. When I started doing this review, I made it crawl this site again, to see what it came up with and be able to make screenshots for you people.

First off, this is the menu that it gives you, which immediately shows you that is has a very slick interface (made it a bit smaller to fit):

The report that is now highlighted, the 200 report, shows all "normally" accessible pages within your site, and looks like this:

You can then click on a page title to get a summary for that page, which looks like this:

As you can see there's a small bug there, it's mixing up the keywords and the description, but the information is very complete and useful.

Another nice feature is the reporting of duplicate titles and descriptions, for instance this duplicate title report for my blog:

This is only a small sample of the reports it gives, and I think this can be a very useful tool. It has some bugs, and it will probably improve a lot over the coming months. You should probably check the features page if this interests you.

Pricing
If you use this service to take care of a few websites, the basic subscription will be good enough, and will only set you back $99.99 for 3 months, which translates into like €67 at the moment.

So, is it worth it?
If you consider how easy this service makes some stuff, I think it's well worth it! You might say "Google Webmaster Tools can give me this info too". First of all: not all of it, and secondly: have you found the crawl now button in Google Webmaster Tools yet? No? You won't. There is none. You can use this tool when you want it, and get the data at the time you need. Of course this tool isn't finished yet, but I seriously think this could be a good addition to the webmasters' toolbox.

If you want your product or service to be reviewed like this as well, see the sponsored reviews section of my advertising page!

Finding crawl issues with Crawl Score is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

1. First of all, and most hardened SEO's know this: Google Blog search. The link data in there is just fenomenal, as they're not, like the normal Google link: command results, filtered. When you want a complete overview, make sure you disable the dupe content filter, that will show you all those pesky scrapers. If you need this data to use it in a tool, it offers RSS feeds too.
2. Second: Technorati. If you click on a blog's authority, you get to see it's reactions, and you can also drag these out via the API.
3. Blogpulse, you don't even need to add link: here, you can just throw in a URL, just like the others, Blogpulse offers RSS feeds.
4. IceRocket. Offers RSS feeds for the link: command queries as well.
5. WASAlive, not too much data, but sometimes it offers some unique stuff. RSS feeds for the results as well.

Now you should also consider, when you're querying those engines, to query for two things: the blog URL, and it's feed URL. In my blog's case: yoast.com and feeds.joostdevalk.nl. Especially scrapers will usually just link to your feed URL.

Now if you've read my post on the Technorati Authority booster, you know why I needed this myself...

Find out who’s scraping you! is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

As Sebastian explained in an excellent post on SEOmoz, there are two different kinds of directives: crawler directives and indexer directives.

Crawler directives
The robots.txt file only contains the so called Crawler directives, telling search engines, identified by their User-agent:, where they are not allowed to go by using Disallow: and where they can (and should) go by using Allow:, and by pointing them at a Sitemap:.

As Sebastian pointed out and explains thoroughly in another brilliant post, pages that search engines aren't allowed to spider, can still show up in the search results, when they have enough links pointing at them. This basically means that if you want to really hide something from the search engines and thus from people using search, robots.txt just isn't good enough.

Indexer directives
Indexer directives are directives that are, even with the birth of the X-Robots-Tag, set on a per page or even per element basis. Up until July 2007, there were two: the microformat rel="nofollow", which means that that link should not pass authority / PageRank, and the Meta Robots tag.

With the Meta Robots tag, you can really prevent search engines from showing the pages you block in the search results. You can reach the same with the relatively new X-Robots-Tag HTTP header. If you don't know what an HTTP header is, I'd suggest reading the Wikipedia page on it, but in short: look at it as the envelope around your content. This HTTP header is better than the meta robots tag for a couple of reasons, one of them is that you can send those headers for other documents too. So, let's get into some examples.

Example uses of the X-Robots-Tag

If you want to prevent search engines from showing files you've generated with PHP, add the following in the header file:

header("X-Robots-Tag: noindex", true);

This would not prevent search engines from following the links on those pages, if you want to do that, do the following:

header("X-Robots-Tag: noindex, nofollow", true);

But doing it in PHP is probably not the easiest use for this kind of thing. I myself greatly prefer setting headers in Apache, when possible. Consider, for instance, preventing search engines from caching / showing a preview for all .doc files on your domain, you would only have to do the following:

<FilesMatch "\.doc$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or, if you'd want to do this for both .doc and .pdf files:

<FilesMatch "\.(doc|pdf)$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or another case, your robots.txt file itself is showing up in the search results. Adding this to your Apache config or your .htaccess file would solve that:

<FilesMatch "robots\.txt">
Header set X-Robots-Tag "noindex"
</FilesMatch>

I had a slight uncomfortable feeling when writing this down, so I e-mailed Matt Cutts asking the following: "<snip> would that mean that you will still fetch it for robots.txt purposes, but won't show it in the index?". I'm waiting for him to answer, and will add his response here once I have it.

Tools
I've quickly created a bookmarklet which shows all the headers for a page (works in Moz browsers only I think, and a Greasemonkey script which pops up when a page is using an X-Robots-Tag header.

Conclusion
As you can see, if you combine the examples above with the stuff you can learn from for instance AskApache's .htaccess tutorial, the X-Robots-Tag HTTP header becomes a very powerful tool. Use it wisely and with caution, as you won't be the first to block your entire site by accident, but it's a great addition to your toolset if you know how to use it.

Playing with the X-Robots-Tag HTTP header is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

[code lang="js"]
javascript:(function(){var%20r=escape(window.location);document.location='http://technorati.com/ping/?url='+r;})();
[/code]

Now you can easily ping those blogs linking to you, and increase your Technorati ranking!

Technorati Ping Bookmarklet is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

The idea is simple: compile a list of blogs that link to you, and that aren't indexed by Technorati. Then ping Technorati with all those pages. The first post wanted you to do a manual ping on all of them, which is way too much work.

Sebastian created a tool with which you can ping multiple URL's at once, making the process a lot faster. Now the next step is obvious: if you can automate pinging, all you have to do is scrape other search engines, and ping Technorati with all the content it's unaware of. That gives you a pretty big advantage.

You've guessed it: I've done that. It will be available within a few weeks. Something will be different though: this will be a for pay tool part of a kit called the Blog Marketer's Toolkit. Why? Because if everybody has this plugin, there's no advantage to the people using it anymore, which is why I will only be making it available to a limited number of people. And of course, I like to make a bit of money as well sometimes :)

Technorati Authority Booster is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

If you're not convinced yet you should go get it now, here are some screenshots:

So here is the SEO Link Analysis extension!

Update: just updated the script to 1.0.1 to fix a change in Yahoo!'s interface.
Another update: 1.0.2 is now available for download.

SEO Link Analysis Firefox Extension is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

I'll try to get some more evidence, but it seems like you have to be logged out of Google to get "real" results now, I hope this is just a glitch, I'll try and get some confirmation on whether this is supposed to happen or not...

Personalized search disabling not working? is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

[code language="php"]
function FrontpageDiggs($url) {
$result['source'] = "http://digg.com/rss_search?search=".$url."&area=promoted&type=url&section=all";
$output = file_get_contents($result['source']);
preg_match_all("/.*().*/",$output,$matches);
$result['result'] = count($matches[1]);
return $result;
}
[/code]

Now if you echo $result['result'], you'll get the number of posts for $url that made the frontpage.

Get the number of popular digg posts for a URL is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Big hunkin’ list of Mac Tools for SEO’s and Webdevelopers is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

It's quite easy for checking rankings if you don't have a big rank tracking system at your SEO company, and I even know some pretty big companies that use this tool all the time... For $47 it's cheap as dirt, it works on Windows, Mac and Linux, so I suggest you buy it now before it sells out!

Update: the deal is gone now of course, but you can still buy the tool directly from Link-Assistant for $87, or download a trial version.

Good deal on DealDotCom is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!