Comments on: Google Webmaster Tools Content Analysis shows Google breaks the rules.

By: gissit

gissit — Mon, 07 Jan 2008 20:12:55 +0000

Joost
I can't really argue with someone with a homepage pr like yours :) but experience has shown me that less is very often more in terms of raw number of pages Vs SERP positions.
If I want to create discovery links I do it in a html sitemap in a page formatted for users.
Using this type of philosophy I have much better results than most competitiors with far fewer inbound links and visible (toolbar) PR.

I know that a lot of talk has gone into the whole nofollow thing but I have also seen the effect of using it as intended. A bit of link management work on a friends forum has increased Google traffic more than 20 fold in less than 9 months.

Anyway, I think this may all be a bit off-topic for this thread but if you don't mind I will bookmark your site and return when I have more time as it looks interesting here.

By: Joost de Valk

Joost de Valk — Mon, 07 Jan 2008 19:22:53 +0000

@gissit: First of all, what you call "current wisdom" does not apply here, I've removed nofollows. Second: the pages are just there for discovery, just like XML sitemaps are, there's nothing wrong with that.

By: gissit

gissit — Mon, 07 Jan 2008 19:17:09 +0000

I read the article (I was searching for something else and this caught my eye). The behaviour of the Googlebot is exactly as I would expect, it was not prohibited from viewing the page in robots.txt, it has not indexed the page as directed by the meta tag and 'maybe' it followed the links as requested.
As this is an SEO blog my comment was more related to the wisdom of having such pages in todays link spam climate. Google has specifically made a point of saying on many occaisions that pages should be for visitors and not for search engines. If I were an algorithm writer this would certainly raise some alarms with me.
If the pages are not for visitors to see then what else are they for? If it is to lead a search engine to other pages then they are indeed doorway pages and may suffer the wrath of Google.
As for posting just to get a link? I really do not need a link from a blog to help my seo. If this page is written in accordance with current wisdom the link will either have a rel="nofollow" or will be wrapped in javascript. If it is not then it would have little or no value anyway as this is not related to my site topic.

By: tom

tom — Mon, 07 Jan 2008 15:32:50 +0000

Did you read the article gissit? or do want a link to your site...

By: gissit

gissit — Mon, 07 Jan 2008 15:25:55 +0000

So you have pages that you want google to follow links from but do not want it to send visitors there? So it is a doorway page of some sort?

By: ed

ed — Wed, 19 Dec 2007 09:46:31 +0000

Thanks, got confused by the term analysis, which is not analytics...

By: Joost de Valk

Joost de Valk — Tue, 18 Dec 2007 19:31:25 +0000

@ed: this is in Google Webmaster Tools, not analytics.

By: ed

ed — Tue, 18 Dec 2007 14:51:31 +0000

Where can I find this duplicate title tag section in google analytics?

By: » Tracking URLs that pass link juice Dixon Jones: Echoes from a quiet space

» Tracking URLs that pass link juice Dixon Jones: Echoes from a quiet space — Mon, 17 Dec 2007 13:09:46 +0000

[...] but by colleague Andy has some issues. Let’s be honest Google - you are ruining standards. Joost gave an example of ignoring standards they’ve agreed by indexing “no index” [...]

By: Joost de Valk

Joost de Valk — Mon, 17 Dec 2007 09:13:47 +0000

@tingeltangeltill: noindex without a nofollow impicitly means, noindex, follow. Hence they need to spider those pages to know which links are on there, that's logical. They won't show them in their index, yet they DO give me some sort of advice on them. I'd rather have them not do that, but they don't "put them in their index", they spider them because they need to follow the links.

By: tingeltangeltill

tingeltangeltill — Mon, 17 Dec 2007 09:03:32 +0000

If someone is linking to your "noindex" sites, google will put them in their index, as Matt said in a Google Video

By: Sint

Sint — Sun, 16 Dec 2007 13:06:19 +0000

@Sergey: I hope the reason for this is not that you called your file really tobots.txt ;-)
Do you have a noindex-tag in these files? Or are they all non-HTML so this isn't possible?

Alternatively you could also block all traffic coming from Googlebot using .htaccess on these folders.

By: Sergey Rusak

Sergey Rusak — Sun, 16 Dec 2007 11:08:22 +0000

For my blog I block all /trackback/ and /feed/ pages through tobots.txt but for some reason Google still index them every time new page appear. Only one good thing, it appear in index only for 1-3 days and later dissapear. Wierd.

By: nam

nam — Sat, 15 Dec 2007 19:20:05 +0000

Content Analysis is a excellent tool to determine a duplicated content, I've used my file robots.txt to to restrict access to Google on these pages, and it's working well for me :
For exemple :
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$

By: Sint

Sint — Sat, 15 Dec 2007 19:09:59 +0000

I agree with some people here that the Google Webmaster Tool can used for more things than just SEO, so providing this information from pages that should be no-indexed can still be useful. But it also raises questions about how Google is interpreting the noindex-metatag and what the status of this tag is related to non-SERP-functionalities of search engines. Maybe it would be wise if Google would include some information on this issue in the documentation/help section of WM-Tools.

By: Sebastian

Sebastian — Sat, 15 Dec 2007 10:18:30 +0000

From Google's POV that's usability advice, not crawling, indexing or ranking advice. I see where you're coming from but I don't think that's really "breaking rules". :)

By: Joost de Valk

Joost de Valk — Sat, 15 Dec 2007 07:59:14 +0000

Sebastian: that's not new to me either, believe me :) but it's just that since I don't want these pages to appear in the SERPs, have made that VERY clear to Google, and YET they still give me that stupid advice :)

By: Sebastian

Sebastian — Sat, 15 Dec 2007 00:53:49 +0000

Oops ... Disallow: /*?=s that is

By: Sebastian

Sebastian — Sat, 15 Dec 2007 00:52:22 +0000

Noindex means "crawl but don't display it on SERPs". Since those URLs aren't indexed Google obeys the REP tag. To follow the links Google needs to keep a copy, so why shouldn't they provide that info in your GWC acct where nobody else can view it? It's extracted from the crawl cache, not the visible cache. You could complain when you add a Disallow: /*?p= to your robots.txt's Googlebot section ;)

By: André Scholten

André Scholten — Fri, 14 Dec 2007 23:00:17 +0000

I someone creates a link to that second page, a visitor that clicks on that link immediately sees he's on page 2. "sense of place".

But I agree with you: nobody will invest time in those little details. You have to be a idealist to do that.