1. Put up a robots.txt on the new domain with the following contents:

   User-agent: *
   Disallow: /

2. Copy the database and files to the new domain.
3. Edit wp-config.php to have the right database settings, and add the following lines, replacing sample.com with the right domain:

   define('WP_SITEURL', 'http://www.example.com');
   define('WP_HOME', 'http://www.example.com');

4. Now set up your blog install, with the same settings as on the old domain (if you have a caching plugin installed, delete your cache files).
5. Install the Search and Replace plugin, and do a search and replace for your old URL's and change them with your new URL's.
6. Once you're done, and you've checked everything works twice:
7. Remove the robots.txt file.
8. On your old blog, add the following line to your apache .htaccess or vhost-config:

   Redirect 301 /blog/ http://www.newdomain.com/

9. Remove the old blog code and database.
10. Change your FeedBurner account, if you have one, to pick up the correct feed.

That should be it. If you have any suggestions or spot things that I've missed, please note them in the comments!

Move your WordPress blog to a new domain in 10 steps! is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

As Sebastian explained in an excellent post on SEOmoz, there are two different kinds of directives: crawler directives and indexer directives.

Crawler directives
The robots.txt file only contains the so called Crawler directives, telling search engines, identified by their User-agent:, where they are not allowed to go by using Disallow: and where they can (and should) go by using Allow:, and by pointing them at a Sitemap:.

As Sebastian pointed out and explains thoroughly in another brilliant post, pages that search engines aren't allowed to spider, can still show up in the search results, when they have enough links pointing at them. This basically means that if you want to really hide something from the search engines and thus from people using search, robots.txt just isn't good enough.

Indexer directives
Indexer directives are directives that are, even with the birth of the X-Robots-Tag, set on a per page or even per element basis. Up until July 2007, there were two: the microformat rel="nofollow", which means that that link should not pass authority / PageRank, and the Meta Robots tag.

With the Meta Robots tag, you can really prevent search engines from showing the pages you block in the search results. You can reach the same with the relatively new X-Robots-Tag HTTP header. If you don't know what an HTTP header is, I'd suggest reading the Wikipedia page on it, but in short: look at it as the envelope around your content. This HTTP header is better than the meta robots tag for a couple of reasons, one of them is that you can send those headers for other documents too. So, let's get into some examples.

Example uses of the X-Robots-Tag

If you want to prevent search engines from showing files you've generated with PHP, add the following in the header file:

header("X-Robots-Tag: noindex", true);

This would not prevent search engines from following the links on those pages, if you want to do that, do the following:

header("X-Robots-Tag: noindex, nofollow", true);

But doing it in PHP is probably not the easiest use for this kind of thing. I myself greatly prefer setting headers in Apache, when possible. Consider, for instance, preventing search engines from caching / showing a preview for all .doc files on your domain, you would only have to do the following:

<FilesMatch "\.doc$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or, if you'd want to do this for both .doc and .pdf files:

<FilesMatch "\.(doc|pdf)$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or another case, your robots.txt file itself is showing up in the search results. Adding this to your Apache config or your .htaccess file would solve that:

<FilesMatch "robots\.txt">
Header set X-Robots-Tag "noindex"
</FilesMatch>

I had a slight uncomfortable feeling when writing this down, so I e-mailed Matt Cutts asking the following: "<snip> would that mean that you will still fetch it for robots.txt purposes, but won't show it in the index?". I'm waiting for him to answer, and will add his response here once I have it.

Tools
I've quickly created a bookmarklet which shows all the headers for a page (works in Moz browsers only I think, and a Greasemonkey script which pops up when a page is using an X-Robots-Tag header.

Conclusion
As you can see, if you combine the examples above with the stuff you can learn from for instance AskApache's .htaccess tutorial, the X-Robots-Tag HTTP header becomes a very powerful tool. Use it wisely and with caution, as you won't be the first to block your entire site by accident, but it's a great addition to your toolset if you know how to use it.

Playing with the X-Robots-Tag HTTP header is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.example.com$ [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]

Adding the www instead of removing it

And, as requested in the comments, the code to add www to your domain name:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^example.com$
RewriteRule (.*) http://www.example.com$1 [R=301]

How to remove www from your URL with mod_rewrite is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Keeping track of what people are searching for
I had a few things to make sure while building this search function. First of all, I wanted to have the same info I get when I look at the statistics page for the Search Meter WordPress plugin. Secondly, I wanted to show more then just the post or page title. The last, but also very important demand I have is that it's fast. Well the first one was quite easy, I decided to put the data into the table for the search meter plugin and just use that interface, instead of writing that myself. The function was basically adapted from the code of the plugin:

function save_search($searchstring, $numresults) {
global $table_prefix;
// Save into the DB. Usually this will be a new query, so try to insert first
$query = "INSERT INTO {$table_prefix}searchmeter (terms,date,count,last_hits) VALUES ('$searchstring',CURDATE(),1,$numresults)";
$success = mysql_query($query);
if (!$success) {
$query = "UPDATE '{$table_prefix}searchmeter' SET count = count + 1, last_hits = $numresults WHERE terms = '$searchstring' AND date = CURDATE()";
$success = mysql_query($query);
}
return true;
}

Constructing the Yahoo! API query
The second thing wasn't too hard to do either, as Yahoo! returns a snippet with the searchresult for the query. The REST query is constructed as follows:

$query = "site:yoast.com+".urlencode($searchstring);
$request = 'http://search.yahooapis.com/WebSearchService/V1/webSearch?appid='.$yahooappid.'&query='.$query.'&output=php';

The part output=php makes the API return serialized PHP, which, after calling unserialize, turns into a nice clean array, which you can then loop through to display the results.

The only thing left was adding the search box to my layout for the non-blog pages, and fixing the blog to look the same. In the theme I had to change the action URL of the searchform.php file to /search/, and there it was: my new site wide search function!

So far, it's all good. I've got a few things left, like building in paging so you can see more than 10 results. I've got one thing that bothers me though: it's a bit slow. Of course I could solve this by spidering my own site with something like ht://Dig, but I think there ought to be a better solution... Perhaps you guys and girls know of one?

Implementing a sitewide search function is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

PHP 301 for Apache:

header("HTTP/1.1 301 Moved Permanently");
header("Location: http://www.example.com/newpage/");
exit;

PHP 301 for IIS:

header("Status: 301 Moved Permanently");
header("Location: http://www.example.com/newpage/");
exit;

PHP 301 redirects for Apache and IIS is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

A year or so back, I was using underscores in my URL's on this site for spaces, and I decided to switch them to dashes, since several people, including Matt Cutts, had blogged that this was "the way to go". Now at that time I just redirected all old URL's to new ones, which created like 30 lines in my Apache config, something I didn't really like...

This morning I took the time to find a more elegant solution, and the easiest one I could come up with is the following:

RewriteCond %{REQUEST_URI} ^(.*)_(.*)/$
RewriteRule (.*)_(.*)/ http://yoast.com$1-$2/ [R=301]

This will rewrite a single underscore to a single dash, using a 301 redirect. So if you have directories with four underscores, it will go through this rewrite four times before reaching the final URL. I had some directories in which this was the case, and made sure it only had to redirect a maximum of two times by adding the following lines above it:

RewriteCond %{REQUEST_URI} ^(.*)_(.*)_(.*)/$
RewriteRule (.*)_(.*)_(.*)/ http://yoast.com$1-$2-$3/ [R=301]

You could add even more lines like these if need be, I'd make sure that no more than two redirects are needed to rewrite the underscores into dashes.

The "/" on the end of the RewriteCond is there to make sure this is a directory, and not a file, since I've got quite a few images with underscores in the names and didn't want to bother renaming them.

Convert underscores to dashes with Apache mod_rewrite is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

RewriteEngine On
RewriteCond %{REQUEST_URI} ^(.*)//$
RewriteRule . http://yoast.com%1/ [R=301]

Trailing double forward slashes in URL’s in the SERPs… is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

The solution can be found in one single RedirectMatch line. This line should be placed above the WordPress code, or in the Apache config. The line is the following:

RedirectMatch 301 /([0-9]+)/([0-9]+)/([0-9]+)/(.*)$ 
http://www.example.com/$4

This matches every URL which starts with a date, and permanently redirects it to the new URL for this post. Ofcourse you could be running your blog in a subdirectory, like I am, in that case, the line would be:

RedirectMatch 301 /subdir/([0-9]+)/([0-9]+)/([0-9]+)/(.*)$
http://www.example.com/subdir/$4

This way, all your incoming links will still work, and thus count for SearchEngines and you have the benefits of the new URL structure.

Changing your permalink structure is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!