WordPress Hosting is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Let's go through some terms here:

Indexed / Indexing
The process of downloading a site or a page's content to the server of the search engine, thereby adding it to it's "index".

Ranking / Listing / Showing
Showing a site in the search result pages (aka SERPs).

So, while the most common process goes from Indexing to Listing, a site doesn't have to be indexed to be listed. If a link points at a page, domain or wherever, that link will be followed. If the robots.txt on that domain prevents the search engine from indexing that page, it'll still show the URL in the results if it can gather from other variables that it might be worth looking at.

If my explanation above doesn't make sense, have a look at Matt Cutt's video explanation:

Click here to view the embedded video.

So, if you want to effectively hide pages from the search engines, and this might seem contradictory, you need them to index those pages. Why? Because when they index those pages, you can tell them not to List them. There's two ways of doing that: by using robots meta tags, like this (and I've got an article on robots meta tags that's more extensive):

<meta name="robots" content="noindex,nofollow"/>

The issue with a tag like that is that you have to add it to each and every page. That's why the search engines came up with the X-Robots-Tag HTTP header. This allows you to specify an HTTP header called X-Robots-Tag, and set the value as you would the meta robots tags value. The cool thing about this is that you can do it for an entire site. So, if your site is running on Apache, and mod_headers is enabled (it usually is), you could add the following single line to your .htaccess file:

Header set X-Robots-Tag "noindex, nofollow"

And it'd have the effect that that entire site can be indexed, but will never be shown in the search results. So, get rid of that robots.txt file with Disallow: / in it, and use the X-Robots-Tag instead!

Preventing your site from being indexed, the right way is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

How to get the most out your Magento install?

Magento is notorious for it speed and performance. But isn't necessary to have a killer web server to get very reasonable performance, it is all about the configuration of your server and the code of your template. However, after the 1.3 release there are some performance improvements, like Frontend Flat Catalog, and people measure reduction of loading times up to 40%. Another recent development is the Magento Compiler this module compiles all Magento files. From tests this module gave between 25% to 50% better performance. This module is still in beta and should not be used in a production environment for now.

The question is, what else can you do to speed up your Magento install?

Optimize your template for speed

More information can be found at the Magento SEO article.

Enable compression of your files

Mod_deflate allows output from your server to be compressed. To enable it for Magento edit your .htaccess, around line 74, in Magento root directory.

<ifmodule mod_deflate.c>
############################################
## enable apache served files compression
## http://developer.yahoo.com/performance/rules.html#gzip
# Insert filter
SetOutputFilter DEFLATE
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</ifmodule>

Apache Module mod_expires

mod_expires controls the setting of the Expires HTTP header and the max-age directive of the Cache-Control HTTP header in server responses. To enable Expires HTTP header for Magento you had to change the code underat your .htaccess to.

<ifmodule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 1 month"
</ifmodule>

Use PHP Accelerator

Install a PHP opcode cacher such as APC or XCache. This seems to deliver rather large improvement in the responsive of the Magento install. Some people note a reduce of loading time up to 70%.

Tune your MySQL configuration

You can modify the configuration of the MySQL server to take advantage of the server's RAM. The efficiency of this step are different, it seems to depends mainly on the number of products. When you have only 100 products, difference will be hardly to notice (below 100ms). But for shops with over 50.000 products there is a huge reduction of the loading time. To give an idea about the settings, for shops with only a limit number of products a query_cache_limit of 1MB will be enough. For larger stores the optimal query cache value can be 64MB. To get the optimal value you really need some testing. A good start point for your Query Cache configuration can be found here.

Speed up your Cache files

Magento makes extensive use of file-based storage for caching and session storage. The slowest component in a server is the hard drive, so if you use a memory-based file system such as tmpfs, you can save all those extra disk IO cycles by storing these temporary files in memory instead of storing them on your slow hard drive.

Let´s do this with tmpfs

Let's say your Magento install is at /var/www/domain.com/ so your cache and session directories are /var/www/domain.com/var/cache/ and /var/www/domain.com/var/session/ For cache we will allocate a max of 256MB RAM and for sessions 64MB RAM. It is important to give access to everyone (e.g your webserver Apache)

mount -t tmpfs -o size=256M,mode=0744 tmpfs /var/www/domain.com/var/cache/
mount -t tmpfs -o size=64M,mode=0744 tmpfs /var/www/domain.com/var/session/

Now it might be a good idea to restore these volumes each time your server boots. Just add the following lines to your /etc/fstab:

tmpfs /var/www/domain.com/var/cache/ tmpfs size=256,mode=0744 0 0
tmpfs /var/www/domain.com/var/session/ tmpfs size=64,mode=0744 0 0

Please note, your tmpfs is temporary in the sense that nothing will be created on your hard drive. If you reboot, everything in tmpfs will be lost.

Save the sessions in your database

Another option for the sessions is to save them in your database. Magento supports this very well and it is one small step to make this work. Just edit app/etc/local.xmland set

<session_save><![CDATA[files]]></session_save>

to

<session_save><![CDATA[db]]></session_save>

it will then save all sessions in the database which is a much quicker access than the filesystem. When you use a cluster enviroment you had to use this option.

Enable Apache KeepAlives

Enable Apache KeepAlives, this allows persistent connections. These long-lived HTTP sessions allow multiple requests to be send over the same TCP connection, this can result in an almost 50% speedup in latency times for HTML documents with lots of images. An example setting can be:

KeepAlive On
KeepAliveTimeout 2

Need more Magento Performance?

Ok, you are a very successful online retailer and it is time to cluster, no problem. This means that one single-server doesn't have enough power to keep your customers happy and so you need a clustered environment with two or more servers. A first step can be to start using a Content Delivery Network (CDN).

The most easy to start is using an separate media server, e.g. media.domain.com. Magento has native support for this. You can configure 'Base Media URL' secure and unsecure under web configurations. This allow you to serve media form one server without the needs to synchronize any directory. Another option is to start using a Content Delivery Network (CDN) for delivering the static files like images. MaxCDN has some options for Magento that are quite nice.

Another option is to use more than one database servers this tactic is called "database replication". One database will be the "master" and all others will be called "slaves". The master is the only database that accepts any sort of write-base queries. These write queries are then replicated to any of the slave servers in real-time. A post at the Magento blog will get you on the right track.

To summarize, 5 quick wins

• move the the rules in the .htaccess files from the directories to virtualhost configuration directives
• install Xcache or APC PHP cache accelerator
• clean up your template and layout
• make sure Apache KeepAlives is enabled
• use a memory-based file system for Magento's var/cache/ and var/session/ directory

Conclusion

This article give a introduction how to improve your Magento site performance, none of these tips are revolutionary. Because every website's scenario is different, when you really want to get the most out of your Magento install you need to hire a professional. The only way to discover the optimal server configuration is testing, a really great tool to use is Browsermob, load testing with real web browsers so you will able to put even the Magento checkout process under load testing.

Now, all of the above you can’t do on your average host, you need a VPS that provides you full control of your server environment. Check out WestHost’s sister company, VPS.net, for a VPS environment that would give you full control.

Magento performance hosting is a post by Joachim Houtman on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

As Sebastian explained in an excellent post on SEOmoz, there are two different kinds of directives: crawler directives and indexer directives.

Crawler directives
The robots.txt file only contains the so called Crawler directives, telling search engines, identified by their User-agent:, where they are not allowed to go by using Disallow: and where they can (and should) go by using Allow:, and by pointing them at a Sitemap:.

As Sebastian pointed out and explains thoroughly in another brilliant post, pages that search engines aren't allowed to spider, can still show up in the search results, when they have enough links pointing at them. This basically means that if you want to really hide something from the search engines and thus from people using search, robots.txt just isn't good enough.

Indexer directives
Indexer directives are directives that are, even with the birth of the X-Robots-Tag, set on a per page or even per element basis. Up until July 2007, there were two: the microformat rel="nofollow", which means that that link should not pass authority / PageRank, and the Meta Robots tag.

With the Meta Robots tag, you can really prevent search engines from showing the pages you block in the search results. You can reach the same with the relatively new X-Robots-Tag HTTP header. If you don't know what an HTTP header is, I'd suggest reading the Wikipedia page on it, but in short: look at it as the envelope around your content. This HTTP header is better than the meta robots tag for a couple of reasons, one of them is that you can send those headers for other documents too. So, let's get into some examples.

Example uses of the X-Robots-Tag

If you want to prevent search engines from showing files you've generated with PHP, add the following in the header file:

header("X-Robots-Tag: noindex", true);

This would not prevent search engines from following the links on those pages, if you want to do that, do the following:

header("X-Robots-Tag: noindex, nofollow", true);

But doing it in PHP is probably not the easiest use for this kind of thing. I myself greatly prefer setting headers in Apache, when possible. Consider, for instance, preventing search engines from caching / showing a preview for all .doc files on your domain, you would only have to do the following:

<FilesMatch "\.doc$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or, if you'd want to do this for both .doc and .pdf files:

<FilesMatch "\.(doc|pdf)$">
Header set X-Robots-Tag "index, noarchive, nosnippet"
</Files>

Or another case, your robots.txt file itself is showing up in the search results. Adding this to your Apache config or your .htaccess file would solve that:

<FilesMatch "robots\.txt">
Header set X-Robots-Tag "noindex"
</FilesMatch>

I had a slight uncomfortable feeling when writing this down, so I e-mailed Matt Cutts asking the following: "<snip> would that mean that you will still fetch it for robots.txt purposes, but won't show it in the index?". I'm waiting for him to answer, and will add his response here once I have it.

Tools
I've quickly created a bookmarklet which shows all the headers for a page (works in Moz browsers only I think, and a Greasemonkey script which pops up when a page is using an X-Robots-Tag header.

Conclusion
As you can see, if you combine the examples above with the stuff you can learn from for instance AskApache's .htaccess tutorial, the X-Robots-Tag HTTP header becomes a very powerful tool. Use it wisely and with caution, as you won't be the first to block your entire site by accident, but it's a great addition to your toolset if you know how to use it.

Playing with the X-Robots-Tag HTTP header is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.example.com$ [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]

Adding the www instead of removing it

And, as requested in the comments, the code to add www to your domain name:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^example.com$
RewriteRule (.*) http://www.example.com$1 [R=301]

How to remove www from your URL with mod_rewrite is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

PHP 301 for Apache:

header("HTTP/1.1 301 Moved Permanently");
header("Location: http://www.example.com/newpage/");
exit;

PHP 301 for IIS:

header("Status: 301 Moved Permanently");
header("Location: http://www.example.com/newpage/");
exit;

PHP 301 redirects for Apache and IIS is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

A year or so back, I was using underscores in my URL's on this site for spaces, and I decided to switch them to dashes, since several people, including Matt Cutts, had blogged that this was "the way to go". Now at that time I just redirected all old URL's to new ones, which created like 30 lines in my Apache config, something I didn't really like...

This morning I took the time to find a more elegant solution, and the easiest one I could come up with is the following:

RewriteCond %{REQUEST_URI} ^(.*)_(.*)/$
RewriteRule (.*)_(.*)/ http://yoast.com$1-$2/ [R=301]

This will rewrite a single underscore to a single dash, using a 301 redirect. So if you have directories with four underscores, it will go through this rewrite four times before reaching the final URL. I had some directories in which this was the case, and made sure it only had to redirect a maximum of two times by adding the following lines above it:

RewriteCond %{REQUEST_URI} ^(.*)_(.*)_(.*)/$
RewriteRule (.*)_(.*)_(.*)/ http://yoast.com$1-$2-$3/ [R=301]

You could add even more lines like these if need be, I'd make sure that no more than two redirects are needed to rewrite the underscores into dashes.

The "/" on the end of the RewriteCond is there to make sure this is a directory, and not a file, since I've got quite a few images with underscores in the names and didn't want to bother renaming them.

Convert underscores to dashes with Apache mod_rewrite is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

RewriteEngine On
RewriteCond %{REQUEST_URI} ^(.*)//$
RewriteRule . http://yoast.com%1/ [R=301]

Trailing double forward slashes in URL’s in the SERPs… is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

The solution can be found in one single RedirectMatch line. This line should be placed above the WordPress code, or in the Apache config. The line is the following:

RedirectMatch 301 /([0-9]+)/([0-9]+)/([0-9]+)/(.*)$ 

http://www.example.com/$4

This matches every URL which starts with a date, and permanently redirects it to the new URL for this post. Ofcourse you could be running your blog in a subdirectory, like I am, in that case, the line would be:

RedirectMatch 301 /subdir/([0-9]+)/([0-9]+)/([0-9]+)/(.*)$

http://www.example.com/subdir/$4

This way, all your incoming links will still work, and thus count for SearchEngines and you have the benefits of the new URL structure.

Changing your WordPress date permalink structure is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Apache.Startkabel.nl is a post by Joost de Valk on Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!