How to get the most out your Magento install?

Magento is notorious for it speed and performance. But isn't necessary to have a killer web server to get very reasonable performance, it is all about the configuration of your server and the code of your template. However, after the 1.3 release there are some performance improvements, like Frontend Flat Catalog, and people measure reduction of loading times up to 40%. Another recent development is the Magento Compiler this module compiles all Magento files. From tests this module gave between 25% to 50% better performance. This module is still in beta and should not be used in a production environment for now.

The question is, what else can you do to speed up your Magento install?

Optimize your template for speed

More information can be found at the Magento SEO article.

Enable compression of your files

Mod_deflate allows output from your server to be compressed. To enable it for Magento edit your .htaccess, around line 74, in Magento root directory.

<ifmodule mod_deflate.c>
############################################
## enable apache served files compression
## http://developer.yahoo.com/performance/rules.html#gzip
# Insert filter
SetOutputFilter DEFLATE
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</ifmodule>

Apache Module mod_expires

mod_expires controls the setting of the Expires HTTP header and the max-age directive of the Cache-Control HTTP header in server responses. To enable Expires HTTP header for Magento you had to change the code under at your .htaccess to.

<ifmodule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 1 month"
</ifmodule>

Use PHP Accelerator

Install a PHP opcode cacher such as APC or XCache. This seems to deliver rather large improvement in the responsive of the Magento install. Some people note a reduce of loading time up to 70%.

Tune your MySQL configuration

You can modify the configuration of the MySQL server to take advantage of the server's RAM. The efficiency of this step are different, it seems to depends mainly on the number of products. When you have only 100 products, difference will be hardly to notice (below 100ms). But for shops with over 50.000 products there is a huge reduction of the loading time. To give an idea about the settings, for shops with only a limit number of products a query_cache_limit of 1MB will be enough. For larger stores the optimal query cache value can be 64MB. To get the optimal value you really need some testing. A good start point for your Query Cache configuration can be found here.

Speed up your Cache files

Magento makes extensive use of file-based storage for caching and session storage. The slowest component in a server is the hard drive, so if you use a memory-based file system such as tmpfs, you can save all those extra disk IO cycles by storing these temporary files in memory instead of storing them on your slow hard drive.

Let´s do this with tmpfs

Let's say your Magento install is at /var/www/domain.com/ so your cache and session directories are /var/www/domain.com/var/cache/ and /var/www/domain.com/var/session/ For cache we will allocate a max of 256MB RAM and for sessions 64MB RAM. It is important to give access to everyone (e.g your webserver Apache)

mount -t tmpfs -o size=256M,mode=0744 tmpfs /var/www/domain.com/var/cache/
mount -t tmpfs -o size=64M,mode=0744 tmpfs /var/www/domain.com/var/session/

Now it might be a good idea to restore these volumes each time your server boots. Just add the following lines to your /etc/fstab:

tmpfs /var/www/domain.com/var/cache/ tmpfs size=256,mode=0744 0 0
tmpfs /var/www/domain.com/var/session/ tmpfs size=64,mode=0744 0 0

Please note, your tmpfs is temporary in the sense that nothing will be created on your hard drive. If you reboot, everything in tmpfs will be lost.

Save the sessions in your database

Another option for the sessions is to save them in your database. Magento supports this very well and it is one small step to make this work. Just edit app/etc/local.xml and set

<session_save><![CDATA[files]]></session_save>

to

<session_save><![CDATA[db]]></session_save>

it will then save all sessions in the database which is a much quicker access than the filesystem. When you use a cluster enviroment you had to use this option.

Enable Apache KeepAlives

Enable Apache KeepAlives, this allows persistent connections. These long-lived HTTP sessions allow multiple requests to be send over the same TCP connection, this can result in an almost 50% speedup in latency times for HTML documents with lots of images. An example setting can be:

KeepAlive On
KeepAliveTimeout 2

Need more Magento Performance?

Ok, you are a very successful online retailer and it is time to cluster, no problem. This means that one single-server doesn't have enough power to keep your customers happy and so you need a clustered environment with two or more servers. A first step can be to start using a Content Delivery Network (CDN). In the low end a Amazon CloudFront/S3 as CDN can be very interesting option.

The most easy to start is using an separate media server, e.g. media.domain.com. Magento has native support for this. You can configure 'Base Media URL' secure and unsecure under web configurations. This allow you to serve media form one server without the needs to synchronize any directory. Another option is to start using a Content Delivery Network (CDN) for delivering the static files like images. In the low end a Amazon CloudFront/S3 as CDN can be a very interesting option.

Another option is to use more than one database servers this tactic is called "database replication". One database will be the "master" and all others will be called "slaves". The master is the only database that accepts any sort of write-base queries. These write queries are then replicated to any of the slave servers in real-time. A post at the Magento blog will get you on the right track.

To summarize, 5 quick wins

• move the the rules in the .htaccess files from the directories to virtualhost configuration directives
• install Xcache or APC PHP cache accelerator
• clean up your template and layout
• make sure Apache KeepAlives is enabled
• use a memory-based file system for Magento's var/cache/ and var/session/ directory

Conclusion

This article give a introduction how to improve your Magento site performance, none of these tips are revolutionary. Because every website's scenario is different, when you really want to get the most out of your Magento install you need to hire a professional. The only way to discover the optimal server configuration is testing, a really great tool to use is Browsermob, load testing with real web browsers so you will able to put even the Magento checkout process under load testing.

Now, all of the above you can’t do on your average host, you need a VPS that provides you full control of your server environment. Check out WestHost’s sister company, VPS.net, for a VPS environment that would give you full control.

Magento performance hosting is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

It turned out to be incredibly easy, as long as you have PHP 5.2 or higher, that is.

This is the code:

require_once(ABSPATH . 'wp-includes/class-snoopy.php');
$snoopy = new Snoopy;
$snoopy->fetch("http://twitter.com/statuses/user_timeline/jdevalk.json?count=1");
$twitterdata = json_decode($snoopy->results,true);
echo "<p>\"".$twitterdata[0]["text"]."\"</p>";

As you can see, I use the Snoopy library to fetch the data, as that comes with WordPress by default. Than I decode the JSON results by using the json_decode function. This is the reason you need PHP 5.2 or up, as this was only included with this version of PHP.

Next, we output it. Of course, this is a quick & dirty implementation. If I got dugg now, and 1,000 people a minute came looking at that page, it should have some sort of caching in there. For now though, this is fine as it is!

Update: As Kim noticed in the comments, I've added some code to automatically link any @username mentions to those usernames. Replace the last echo line above with this:

$pattern = '/\@([a-zA-Z]+)/';
$replace = '<a href="http://twitter.com/'.strtolower('\1').'">@\1</a>';
$output = preg_replace($pattern,$replace,$twitterdata[0]["text"]);
echo "<p>\"".$output."\"</p>";

You could even decide to add rel="nofollow" the $replace if you don't want those links to give juice.

Update 2: To make it even more complex, here is the entire code I now use, which excludes replies and caches the Twitter API requests so you won't overload the API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

require_once(ABSPATH . 'wp-includes/class-snoopy.php');
$tweet   = get_option("lasttweet");
$url  = "http://twitter.com/statuses/user_timeline/jdevalk.json?count=20";
if ($tweet['lastcheck'] < ( mktime() - 60 ) ) {
  $snoopy = new Snoopy;
  $result = $snoopy->fetch($url);
  if ($result) {
    $twitterdata   = json_decode($snoopy->results,true);
    $i = 0;
    while ($twitterdata[$i]['in_reply_to_user_id'] != '') {
      $i++;
    }
    $pattern  = '/\@([a-zA-Z]+)/';
    $replace  = '<a href="http://twitter.com/'.strtolower('\1').'">@\1</a>';
    $output   = preg_replace($pattern,$replace,$twitterdata[$i]["text"]);  
 
    $tweet['lastcheck'] = mktime();
    $tweet['data']    = $output;
    $tweet['rawdata']  = $twitterdata;
    $tweet['followers'] = $twitterdata[0]['user']['followers_count'];
    update_option('lasttweet',$tweet);
  } else {
    echo "Twitter API not responding.";
  }
} else {
  $output = $tweet['data'];
}
echo "<p>\"".$output."\"</p>";

As you can see it also saves the amount of followers you have into $tweet['followers'], which I then use to display this:

<p>
  <a href="http://twitter.com/jdevalk">
    <?=$tweet['followers']?> followers on Twitter, and you?
  </a>
</p>

Update 3: Another fix to automatically make all links clickable, below this line:

$output = preg_replace($pattern,$replace,$twitterdata[$i]["text"]);

Add:

$output = make_clickable($output);

This will use the WordPress internal make_clickable function to make sure that all URL's are clickable.

Enjoy!

Easily display your last Tweet is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

[code language="php"]
function FrontpageDiggs($url) {
$result['source'] = "http://digg.com/rss_search?search=".$url."&area=promoted&type=url&section=all";
$output = file_get_contents($result['source']);
preg_match_all("/.*().*/",$output,$matches);
$result['result'] = count($matches[1]);
return $result;
}
[/code]

Now if you echo $result['result'], you'll get the number of posts for $url that made the frontpage.

Get the number of popular digg posts for a URL is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

wp_redirect(get_permalink($url),301);

Easy isn't it? Now go off and USE IT. I'm seeing way too much people who forget to add the ",301", and stick with the default, which is a 302....

WordPress 301 redirect is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Now I did, and I found the cacheAPC example to be very, very easy. It relies on the Alternative PHP Cache, an opcode cache PECL extension for PHP. I wrote two functions, which I then put in to all my pieces of code which I've published that use a lot of calls to the different API's. The first is curlopen, the function I use to open connections:

function curlopen($request) {
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $request);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	curl_setopt($ch, CURLOPT_TIMEOUT, 100);
	curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
	$results = curl_exec ($ch);
	curl_close($ch);
	return $results;
}

The second one is the actual cache function, look at how easy it is:

function request_cache($url, $ttl) {
	if(!$doc = apc_fetch($url)) {
		$doc = curlopen($url);
		if ($doc === false) return false;
		apc_store($url,$doc, $ttl);
	}
	return $doc;
}

It basically does three things:

1. It looks if the requested resource is already in the cache, and if it is, it fetches that;
2. If it's not, it opens it through curlopen and stores it in the cache;
3. It returns the requested data;

As you can see the request_cache function takes two parameters: the request url and the TTL, which, in seconds, determines how long that resource should be cached. Now if you request a PageRank for a URL, it's fairly safe to set this to 24 hours, and you can see how much requests this saves!

PHP-APC: Speed up your web applications! is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Keeping track of what people are searching for
I had a few things to make sure while building this search function. First of all, I wanted to have the same info I get when I look at the statistics page for the Search Meter WordPress plugin. Secondly, I wanted to show more then just the post or page title. The last, but also very important demand I have is that it's fast. Well the first one was quite easy, I decided to put the data into the table for the search meter plugin and just use that interface, instead of writing that myself. The function was basically adapted from the code of the plugin:

function save_search($searchstring, $numresults) {
global $table_prefix;
// Save into the DB. Usually this will be a new query, so try to insert first
$query = "INSERT INTO {$table_prefix}searchmeter (terms,date,count,last_hits) VALUES ('$searchstring',CURDATE(),1,$numresults)";
$success = mysql_query($query);
if (!$success) {
$query = "UPDATE '{$table_prefix}searchmeter' SET count = count + 1, last_hits = $numresults WHERE terms = '$searchstring' AND date = CURDATE()";
$success = mysql_query($query);
}
return true;
}

Constructing the Yahoo! API query
The second thing wasn't too hard to do either, as Yahoo! returns a snippet with the searchresult for the query. The REST query is constructed as follows:

$query = "site:yoast.com+".urlencode($searchstring);
$request = 'http://search.yahooapis.com/WebSearchService/V1/webSearch?appid='.$yahooappid.'&query='.$query.'&output=php';

The part output=php makes the API return serialized PHP, which, after calling unserialize, turns into a nice clean array, which you can then loop through to display the results.

The only thing left was adding the search box to my layout for the non-blog pages, and fixing the blog to look the same. In the theme I had to change the action URL of the searchform.php file to /search/, and there it was: my new site wide search function!

So far, it's all good. I've got a few things left, like building in paging so you can see more than 10 results. I've got one thing that bothers me though: it's a bit slow. Of course I could solve this by spidering my own site with something like ht://Dig, but I think there ought to be a better solution... Perhaps you guys and girls know of one?

Implementing a sitewide search function is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

PHP 301 for Apache:

header("HTTP/1.1 301 Moved Permanently");
header("Location: http://www.example.com/newpage/");
exit;

PHP 301 for IIS:

header("Status: 301 Moved Permanently");
header("Location: http://www.example.com/newpage/");
exit;

PHP 301 redirects for Apache and IIS is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

php_value session.use_trans_sid 0
php_value session.use_only_cookies 1

I also wanted Google to get a clean URL when it decided to spider one of the old URL's again, and didn't have access to mod_rewrite, so I redirected them with some PHP. The solution is quite simple:

if (isset($_GET['PHPSESSID'])) {
	$requesturi = preg_replace('/?PHPSESSID=[^&]+/',"",$_SERVER['REQUEST_URI']);
	$requesturi = preg_replace('/&PHPSESSID=[^&]+/',"",$requesturi);
	header("HTTP/1.1 301 Moved Permanently");
	header("Location: http://".$_SERVER['HTTP_HOST'].$requesturi);
	exit;
}

PHPSESSID in your URL? Learn to 301 redirect them with PHP is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

[tags]del.icio.us, delicious, API, YDN, Zend, PHP[/tags]

Playing with the del.icio.us Web Badge JSON API using PHP is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

Wikipedia describes an API as: 'An application programming interface (API) is the interface that a computer system, library or application provides in order to allow requests for services to be made of it by other computer programs, and/or to allow data to be exchanged between them.'

Now these API’s are super useful, I use them in several of my scripts and have great fun with them. I used the Technorati API, for instance, in the first version of the Technorati rank and link count Greasemonkey script I wrote. This was fast, and worked quite well for a while, so I was very happy. After a few hours of browsing though, it stopped working. The error I got was 'You have used up your daily allotment of Technorati API queries.'

Now I can’t understand why they’d want to limit use of the API like that. Why not? Well, now that I have decided to not use the API (after all how much is 500 queries when you distribute this script to 100+ people?), I’m left with no other choice but to scrape the content of their normal site. This costs them way and way more bandwidth, and it’s significantly slower. That’s not much of a service is it? Remember: I do want to use their services! I want to know the Technorati rank for each page I’m visiting, that’s a good sign for them, isn’t it?

So, please Technorati: get rid of that maximum number of queries. Oh and Google and Yahoo!, if you’re listening? Do the same, please!

[tags]technorati, google, yahoo, api, technorati api, google api, yahoo api, application programming interface[/tags]

Web 2.0 and API’s: the case for Unlimited queries is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

An example:

Want such a thing for yourself? Go check it out: FeedBurner graph.

[tags]analytics, feedburner, graphics, feedburner api[/tags]

FeedBurner API fun is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

I created a simple implementation on top of it, using PHP's DOMDocument extension to walk through the result, as using NUSoap or something like that felt like overkill, and it would be hard since the request isn't a SOAP request. It's quite simple, but it's also slow...

So, my wishlist for the next version of this API:

• a way to do a SOAP request instead of a HTTP request;
• faster, please, please W3C, make it faster;
• a way to specify what info i want, it would be cool to be able to just know if a site is valid or not.

If they fix that, I'm a happy camper :)

W3C Validator API is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

The solution is easy as well, rename class soapclient and function soapclient to nusoapclient in nusoap.php, and make sure you use new nusoapclient instead of new soapclient in your scripts.

PHP5 and NUSOAP is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!

First, we have to get the cached page from the Google cache through the API:

function GoogleCache($url) {
$soapclient = new soapclient('http://api.google.com/search/beta2');
$soapoptions = 'urn:GoogleSearch';
$params = array(
'key' => [INSERT YOUR GOOGLE API KEY HERE],
'url' => $url
);
$ret = $soapclient->call('doGetCachedPage', $params, $soapoptions);
$err = $soapclient->getError();
return $ret;
}

Now we have to make a function which uses this one, and gets the last visit date out of it:

function GoogleLastSpidered($url) {
$cache = GoogleCache($url);
preg_match("/.*as retrieved on (.*).
.*/",$cache,$matches);
return $matches[1];
}

If you want, you could now feed the output of this function to strtotime() to do stuff with the date, or you could just echo it.

[tags]seo, google api, google cache[/tags]

Getting Google's last visit date from the Google API is a post from Joost de Valk's Yoast - Tweaking Websites.A good WordPress blog needs good hosting, you don't want your blog to be slow, or, even worse, down, do you? Check out my thoughts on WordPress hosting!