Ok I just had to post this, as it's too funny. I got a Google Alert this afternoon for this post, which mentioned one of my plugins as being listed by Web Designer Magazine. So, I Googled them, as the post didn't link to them, and got this result:
Notice something? I know I did: there's no description there. So I clicked on and checked the page to see what could cause that, finding it was easy:
<meta name='robots' content='noindex,nofollow' />
It was listed right above the EditURI line that WordPress puts in by default, telling me they forgot to uncheck a box (that get's checked by default on some auto installers, I've been told on Twitter):
One issue though... If Google had actually seen that tag, it wouldn't have listed the site at all. So there had to be something else. And of course, there is: meet Webdesigner Magazine's robots.txt:
User-agent: * Disallow: / Sitemap: http://www.webdesignermag.co.uk/sitemap.xml.gz
The first two lines there prevent Google from indexing the site entirely. But it just became funnier... The fourth line was added by a plugin they're running, pointing search engine bots at their XML sitemap. Yes, the same bots they just forbade entry to their entire site... If you check out their XML sitemap, you'd notice they're running Arne Brachold's Google XML Sitemap Generator...
So: they installed an XML sitemap plugin, let's see what else they did: ah... Cool! They're running All In One SEO too! Now AIOSEO is "ok", except: it doesn't warn you for stupidities like these... Might be a darn good feature for an SEO plugin to check whether your site is actually allowing crawlers to come in, don't you think? Ah well, nothing can be perfect.
Anyway, this tells me one thing: Web Designer Mag needs help with their WordPress install. And they need it bad. If you work for them, and read this, feel free to contact me on Twitter or through the contact form. I'd be happy to help!
Considering the privacy setting there is super-important for SEO, yeah all thsoe plugins shoudl totally be checking the status. ;)
Thanks for the LOLs.
Sadly, that's hilarious on so many levels.
Allow me some more gripes:
- they're running WordPress 2.8.4,
- they're asking me to follow them on Twitter but haven't done a decent update in over a month other than just posting their blog posts
- they don't have an about page
- they don't have a contact page
In all, it pretty much sucks.
Looks like Web Designer Mag needs a -ician.
Actually it's common knowledge that Google will list sites that it isn't allowed to index. For example eBay used to disallow Google's crawler. That's why there's no description text for the site. If you don't want your site listed in Google's index you have to submit a request for removal. Noindex and related methods just stop your stuff from being crawled. :)
You're not right there Doug, you should read my recent post on the topic :)
I should clarify, Google asks for robots.txt before it asks for the page & so never gets the noindex directive. This video from Matt Cutts explains why the URL still shows in the search results: http://www.youtube.com/watch?v=KBdEwpRQRD0 What I stated before was incorrect because Google does respect the noindex tag, just can't see that tag in this case. I had to refresh my memory on this, they'll still show a page if they can't crawl it but they won't if it's either manually removed from the index or contains the noindex directive. So my previous comment should read "Google will list sites it isn't allowed to
indexcrawl" :)Still wonder why anyone would want their website not listed in search but hey their loss.
Doug, I can think of a lot of reasons that you would not want the content of your site listed.
For one, a lot of forums do not want their content placed into the search engine. I run a few forums and one of them is deliberately NOT listed (specifically due to the reason listed next.)
For another site that I used to run, they are seriously considering blocking search due to spammers. They don't need to drive traffic to the forum (the rest of the site is searchable) yet due to their 'popularity' of traffic to the forum there are up to 1,000 spammer registration attempts per day.
For some sites, having major reduction in spammers is worth the 'loss' of not being in a search engine.
Joost,
Do you realize how large of a company they are? http://www.imagine-publishing.co.uk/index.html
Kinda scary :)
Ey Nick, yeah I know... It IS scary.
Even more scary considering that magazine specializes in web development
Lol, that's hillarious.
I like the magazine, but this finding sure make em looks stupid.
The top list after installing wordpress I always installing Google Sitemap plugin and All SEO. I want google index my new site ASAP!
I dont think you really want to help those guys! Dam the are non SEO minded.
I do, actually :)
In defense of Web Designer Mag, it is a site about web design, not blog design, which are like two entirely separate things. Right guys? lol.
As a side question, what exactly would constitute a "decent update" on Twitter besides "just posting their blog posts"? Like a link to another blog or interesting article, or a brief tip on web design or something? Just curious.
Well they have articles on the topic ;)
To answer your side question: Twitter is a conversational medium, not a "push out your RSS feed" kind of medium, that's the only thing they're doing now it seems.
Hehe i've had cases that your Meta Robots put those lines in there after upgrading WP without updating the plugin (and even then) but that robots.txt is just plain stupid ;)
I'm interested if they already tried to contact you..
Not yet ;)
"By Joost de Valk Thu, December 31st, 2009 at 15:29Reply
Not yet ;)"
They are probably in an emergy-meeting: "I want your input on how to increase traffic! I want a planning and budget on new design - who is the hottest designer out there? I want Twitter and Facebook! And it should look good on my iPhone!"
Wow! That's funny - that's so unserious.
I love Yoast's website, but before we laugh and point the finger at others there are still loads of things that need to be fixed here on this website. See how the text flows outside of the textarea? Let's fix that before you try and fix other websites that aren't your own.I'm using Chrome and it happens as well in Safari. See jaytillery.com/yoast.gif for a screenshot.
Hmm I'll fix that Jay, but let's be serious: the issue above is a bit more serious than this one :)
Jay - That's so petty.
Tmik,
How am I being petty? Please explain. Joost wants to help one site and I want to help him. What is wrong with that?
I actually appreciated the comment Jay (not that I solved the issue yet, btw ;) )
RE: "Let's fix that before you try and fix other websites that aren't your own."
Jay - We're likely all developers here and most all of use appreciate feedback, but we all know how it's nearly impossible to create a completely flawless website. To say that Yoast shouldn't constructively critique someone's website because he has spacing or margin issues is definitely petty.
It makes you wonder if there may be some sabotage from an ex-employee. Surely they wouldn't have done this on purpose... would they!
thanks.. it was great help.. i had installed mine with fantastico.. so
Something tells me that basic errors like this in big design companies is more common that most of their clients realize.
This is a web design mag from the UK. Not a design firm. But I do agree with your premise. Many of the services that web design/development firms offer are many times not present on their own sites. Accessibility being the first.
I am waiting to see if they get back to you and decide to fix things, and then to see how much difference it makes to their traffic. If their visitors go through the roof after they fix the Robots indexing, what more proof do you need.
WOW! I am sure they will be fixing that soon! How bad is that! I would be totally embarrassed as well.
Interesting post, but keep in mind just because there is a noindex request for the website - do not mean Google will not add it within their index. Google will add such websites within index by picking up links point to it from other sources, such as ODP and links from other websites. At least, this is my experience...
This actually happened to one of my websites, http://www.ghosthuntingequipment.net. I noticed this in
my google webmaster tools so I deleted my robot.txt plugin thinking it was a gliche but I still
cant seem to fix this... anyone willing to show me how to fix ? Thanks!
I had a quick look at your site and can't find anything wrong, is your Google Webmaster Tools showing any errors?
LOL!
That's the funniest post I've read in ages - and a web design magazine to - oh dear. lolololololol
I just had to add this:
I've just done an webpage optimisation test at http://webpagetest.org and their results are really bad!
They score:
A F F F A F
They've got no SEO, rubbish optimisation - I'm left wondering what on earth they have to write about? It's not as though they know anything about web design....
Hi Joost (and everyone else)... Let me see if I understand this right? Someone is using one of your plugins (read: a show of respect to you and your work), you find they have what is best described as an unfortunate human oversite (pub intended). So you proceed to stab them in the back? Dude, that's not cool.
Yes, I agree they should be more careful. We all should be more careful. But I'm not so sure stabbing them in the back was the best approach here. Honestly, it kinda makes you look "small". As my father used to say, "It's not what you say, it's how you say it.". Or there's the more contemporary Dr. Frank Luntz version, "It's not what you say, it's what they hear."
Did they screw up? Sure they did. I'm only suggesting the approach to, and tone of the "conversation" here could have been wiser.
Finally, Twitter is a communications tool. For some it's for speaking, for others it's for listening and still others it's (as you noted) conversational. It is not the same thing to all people all the time. IMHO, of course ;)
Those are not my plugins Mark :) And yes, I'm publicly telling people about this, because they have an example role, as a web designer magazine. I actually wanted to contact them, but I couldn't find a way of reaching out to them on their website, nor have they been active on twitter in any conversational mean, so there's no chance for me to actually start a conversation.
Secondly, this is a darn fine example of what CAN happen to websites, it happens a LOT, and an example like this is way more likely to teach people.
So you might be right in your tone argument, but I truly think this post serves a broader goal. Last but not least, I have offered them my help, and I truly would be happy to help them out. I actually subscribe to the Dutch version of web designer magazine, and like it a lot.
Btw: just checked again, web designer magazine has now fixed the issue, kudos to them! :)
These are terrible mistakes, and not just that - a online maazine about web site development! Great find Yoast!
That is hilarious, how can they screw up on something so basic as this, did they actually contact you?
"Btw: just checked again, web designer magazine has now fixed the issue, kudos to them! :)"
That's all I was trying to say. We're all trying our best. Loads of balls to juggle, etc. and one can only assume that the publishing industry is getting the worst of it. Joost, no doubt you do great work. I'm only suggesting that you believe that too. The guy at the top of the food chain doesn't need to eat minnows, eh? Yes, it was a great example. We all agree to that. It as the tone/approach I'm calling into questions. That's all.
Btw, I emailed you about your taxonomies plugin. Fancy getting back to me? Cheers.
I used to read web designer magazine a few years ago until I got bored and found a more informed read = .net magazine.
I read your blog.I impress your blog. your blog is very good and very attractive.So I like your every post.Thanks for good tips web design marketing.
Aargh. I feel stupid. Only now reading this blog did I see I had the very same issue! Aw shucks. Anyway thanks Joost, this post helped immensely!
Hahaha, that's so funny. What's sad is I just recently discovered this on my own site, the privacy settings in particular.
Yeah your blog is Awesome ! i am a regular visitor since 10 months. Hope you will keep writing such great post.
They have corrected it now!
AND they don't pass the W3C validator at all... 177 errors? Nice touch :)
Great post and a great discussion. Something to think about and learn
Thanks