Comments on: Playing with the X-Robots-Tag HTTP header http://yoast.com/x-robots-tag-play/#utm_source=rss&utm_medium=rss&utm_campaign=x-robots-tag-play Tweaking Websites Tue, 16 Mar 2010 20:50:13 +0000 http://wordpress.org/?v= hourly 1 By: Baby Pushchairshttp://yoast.com/x-robots-tag-play/#comment-73973 Baby Pushchairs Mon, 05 Oct 2009 11:55:56 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-73973 Im having real trouble with my site due to this problem. At <a href="http://www.pushchairsonline.com/" rel="nofollow">Baby Pushchairs</a> we have multiple routes to s=the same product and therefore have dissallowed many of the filters to stop a duplicate content issue. However as you point out they can still be found through internal linking. If anyone has any tips please let us knowKind RegardsBaby Pushchairs Im having real trouble with my site due to this problem. At Baby Pushchairs we have multiple routes to s=the same product and therefore have dissallowed many of the filters to stop a duplicate content issue. However as you point out they can still be found through internal linking. If anyone has any tips please let us know

Kind Regards

Baby Pushchairs

]]> By: Matthew Austinhttp://yoast.com/x-robots-tag-play/#comment-73694 Matthew Austin Wed, 23 Sep 2009 12:43:50 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-73694 How do you set the header only for HTTPS/SSL requests? How do you set the header only for HTTPS/SSL requests?

]]> By: Всеволодhttp://yoast.com/x-robots-tag-play/#comment-69998 Всеволод Mon, 08 Jun 2009 15:20:08 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-69998 Согласен естественно :) Для меня это точно очень актуально :) Согласен естественно :) Для меня это точно очень актуально :)

]]> By: SpanishSEO (SpanishSEO)http://yoast.com/x-robots-tag-play/#comment-66528 SpanishSEO (SpanishSEO) Sun, 01 Feb 2009 19:12:21 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-66528 El plugin sobre X-Robots-Tag HTTP header de <a rel="nofollow" href="http://twitter.com/jdevalk">@jdevalk</a> acaba de ser mejorado y revisado. Este es el link http://is.gd/eNws El plugin sobre X-Robots-Tag HTTP header de @jdevalk acaba de ser mejorado y revisado. Este es el link http://is.gd/eNws

]]> By: SEO Bookmarklets Collection: On-Page SEO and Domain Statshttp://yoast.com/x-robots-tag-play/#comment-38294 SEO Bookmarklets Collection: On-Page SEO and Domain Stats Mon, 01 Dec 2008 12:51:59 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-38294 [...] Page Headers - shows all headers for a current page (by Yoast.com); [...] [...] Page Headers - shows all headers for a current page (by Yoast.com); [...]

]]> By: Using X-Robots HTTP Header To Sneakily Delete Your Link Farmshttp://yoast.com/x-robots-tag-play/#comment-38112 Using X-Robots HTTP Header To Sneakily Delete Your Link Farms Wed, 19 Nov 2008 13:42:30 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-38112 [...] solution is to make use of the X-Robots HTTP header that Googlebot is now supporting. This allows you to add the noindex meta information to a page [...] [...] solution is to make use of the X-Robots HTTP header that Googlebot is now supporting. This allows you to add the noindex meta information to a page [...]

]]> By: Link Shaping and Bot Herding | SEO Blog | Search and Socialhttp://yoast.com/x-robots-tag-play/#comment-36804 Link Shaping and Bot Herding | SEO Blog | Search and Social Mon, 22 Sep 2008 15:29:40 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-36804 [...] Playing with the X-Robots-Tag HTTP Header - by Joost de Valk [...] [...] Playing with the X-Robots-Tag HTTP Header - by Joost de Valk [...]

]]> By: 5ubliminalhttp://yoast.com/x-robots-tag-play/#comment-35195 5ubliminal Fri, 11 Jul 2008 00:06:30 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-35195 I'd take a look at the page linked here in my comment. You've got all you need there to put your link exchange partners our of business. Crawler Detection + X-Robots-Tag = Undetectable Link <strike>Love</strike> Hate !If crawler detection is in place no greasy script will do anything. And when someone is smart enough to implement this, he goes all the way. So ... <b>no greasy countermeasures</b>! I'd take a look at the page linked here in my comment. You've got all you need there to put your link exchange partners our of business.
Crawler Detection + X-Robots-Tag = Undetectable Link Love Hate !

If crawler detection is in place no greasy script will do anything. And when someone is smart enough to implement this, he goes all the way. So ... no greasy countermeasures!

]]> By: Bretthttp://yoast.com/x-robots-tag-play/#comment-35053 Brett Tue, 01 Jul 2008 18:30:31 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-35053 Hi,You asked if MSNBot supported X-Robots tags. They do. <a href="http://blogs.msdn.com/webmaster/archive/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation.aspx" rel="nofollow">Read the blog post</a> Hi,

You asked if MSNBot supported X-Robots tags. They do. Read the blog post

]]> By: R. Richard Hobbshttp://yoast.com/x-robots-tag-play/#comment-31543 R. Richard Hobbs Tue, 25 Mar 2008 17:19:52 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-31543 howbout the notion of using noindex,follow (or maybe noindex just by itself so (hopefully) you dont get penalised for dupe content, but allow the Googlebot (or other spider) make the most of all the linky love? Would also seem to take care of a problem of say, finding your feed your highest ranked page? (i.e. instead of disallow: for /feed/ use noindex,follow???)Thanks for any thoughts or ideas. howbout the notion of using noindex,follow (or maybe noindex just by itself so (hopefully) you dont get penalised for dupe content, but allow the Googlebot (or other spider) make the most of all the linky love? Would also seem to take care of a problem of say, finding your feed your highest ranked page? (i.e. instead of disallow: for /feed/ use noindex,follow???)

Thanks for any thoughts or ideas.

]]> By: John S. Britsioshttp://yoast.com/x-robots-tag-play/#comment-31416 John S. Britsios Fri, 21 Mar 2008 15:37:12 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-31416 Google have indexed anchor links i.e http://www.example.com/example/example.html#post100Is there a X-Robots workaround in the htaccess for that, since I cannot get rid of them in their Webmaster Tools URL removal tool? Google have indexed anchor links i.e http://www.example.com/example/example.html#post100

Is there a X-Robots workaround in the htaccess for that, since I cannot get rid of them in their Webmaster Tools URL removal tool?

]]> By: quasigoalhttp://yoast.com/x-robots-tag-play/#comment-29021 quasigoal Tue, 12 Feb 2008 11:02:40 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-29021 Joost, interesting blog, useful! I've subscribed the feed. :0) Greetings from Italy. Joost, interesting blog, useful! I've subscribed the feed. :0)
Greetings from Italy.

]]> By: X-Robots Tag für HTTP Header - Centralising the management of crawlers in the .htaccess. - webideas.dehttp://yoast.com/x-robots-tag-play/#comment-28350 X-Robots Tag für HTTP Header - Centralising the management of crawlers in the .htaccess. - webideas.de Sat, 26 Jan 2008 15:50:20 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28350 [...] Sehr gute Beispiele was man mit dem X-Robots Tag HTTP Header anstellen kann gibt es aus meiner Sicht unter: http://yoast.com/x-robots-tag-play/#examples [...] [...] Sehr gute Beispiele was man mit dem X-Robots Tag HTTP Header anstellen kann gibt es aus meiner Sicht unter: http://yoast.com/x-robots-tag-play/#examples [...]

]]> By: Coen Jacobs » De 11 blogs die een webdeveloper niet kan (en mag) missenhttp://yoast.com/x-robots-tag-play/#comment-28348 Coen Jacobs » De 11 blogs die een webdeveloper niet kan (en mag) missen Sat, 26 Jan 2008 13:58:01 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28348 [...] en handige SEO tools. Een van zijn meest interessante artikelen van de laatste tijd is die over de X-Robots-Tag HTTP Header. Wellicht een beetje zware kost voor beginnende webdevelopers, maar zeker de moeite [...] [...] en handige SEO tools. Een van zijn meest interessante artikelen van de laatste tijd is die over de X-Robots-Tag HTTP Header. Wellicht een beetje zware kost voor beginnende webdevelopers, maar zeker de moeite [...]

]]> By: Sinthttp://yoast.com/x-robots-tag-play/#comment-28313 Sint Thu, 24 Jan 2008 10:08:53 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28313 Because the people you exchange these links with won't know you are nofollowing them? Because the people you exchange these links with won't know you are nofollowing them?

]]> By: Sebastianhttp://yoast.com/x-robots-tag-play/#comment-28302 Sebastian Wed, 23 Jan 2008 19:52:52 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28302 Chloe, when you add a "nofollow" to the "noindex" X-Robots-Tag you're fine with Google. But then, why trade links at all? Chloe, when you add a "nofollow" to the "noindex" X-Robots-Tag you're fine with Google. But then, why trade links at all?

]]> By: Joost de Valkhttp://yoast.com/x-robots-tag-play/#comment-28301 Joost de Valk Wed, 23 Jan 2008 18:45:19 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28301 Chloe: that sounds like a black idea :-) Chloe: that sounds like a black idea :-)

]]> By: Baby Chloehttp://yoast.com/x-robots-tag-play/#comment-28299 Baby Chloe Wed, 23 Jan 2008 18:06:59 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28299 Hi Joost, what are you thoughts on getting people to do link exchanges with a 'links' page and then for that page sending via php, a x-robots noindex nofollow header.Would google consider my site to have a huge influx of one way links ?Chloe. Hi Joost, what are you thoughts on getting people to do link exchanges with a 'links' page and then for that page sending via php, a x-robots noindex nofollow header.

Would google consider my site to have a huge influx of one way links ?

Chloe.

]]> By: SEO Bloghttp://yoast.com/x-robots-tag-play/#comment-28296 SEO Blog Wed, 23 Jan 2008 11:56:18 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28296 Didn't know about this tag. But seems very usefull. I alse downloaded your grease monkey script to be sure that pages where I place my links on have no x-robot nofollow header ;). And it's also very usefull to detect in time that you blocked your entire site :).Thx! Didn't know about this tag. But seems very usefull. I alse downloaded your grease monkey script to be sure that pages where I place my links on have no x-robot nofollow header ;). And it's also very usefull to detect in time that you blocked your entire site :).

Thx!

]]> By: links for 2008-01-23http://yoast.com/x-robots-tag-play/#comment-28287 links for 2008-01-23 Wed, 23 Jan 2008 01:23:13 +0000 http://www.joostdevalk.nl/x-robots-tag-play/#comment-28287 [...] Playing with the X-Robots-Tag HTTP header - SEO - Joost de Valk’s SEO Blog Ever since the announcement on the Google Blog and more recently Yahoo’s announcement that they’ve enhanced their support for it, I’ve been meaning to play with the X-Robots-Tag header. (tags: 2008 article Engine HTTP learn robots.txt Search) [...] [...] Playing with the X-Robots-Tag HTTP header - SEO - Joost de Valk’s SEO Blog Ever since the announcement on the Google Blog and more recently Yahoo’s announcement that they’ve enhanced their support for it, I’ve been meaning to play with the X-Robots-Tag header. (tags: 2008 article Engine HTTP learn robots.txt Search) [...]

]]>