Should you test that?

November 20th, 2013 – 6 Comments

Why testing is not always the right way to start optimizing your conversion.

Learning more and more about Conversion Optimization, it appeared to me as though testing (A/B or multivariate) is the only way to go.  Most of the sites of agencies claiming to help optimize your conversion, state that you should separately test every freaking little change you make on your page. Of course, I am exaggerating a little, but my point remains that conversion rate optimization implies a lot of testing. And that is a lot of work….

Scientific Progress

The scientist in me revolted! In science, we make progress (in an ideal world, I know… but still) to work and improve upon each other’s work. We try to make scientific progress! To do things that no man has done before! Can’t we apply that principle to the conversion-business? Does every site owner really have to invent the conversion-wheel for himself? Do we really need to test everything?

In this post I am not going to argue that conversion testing is superfluous. In some situations testing is inevitable and will lead to large improvements in your sales. But: let’s look at two situations in which testing is – at least in my opinion – not the (first) way to go!

1. Your site (of part of it) is just too crappy

In some cases the website just is not ready for any testing. We recently altered our entire checkout-page because there was just too much wrong with it. Our checkout page was crappy. There were too many thing wrong. Testing all these things separately would have taken ages. We decided to improve our checkout page on numerous aspects. Afterwards we tested whether the total package of changes resulted in a higher conversion-rate (and it did). But the initial changes were not tested separately. We did the initial changes on the base of knowledge from the scientific literature and from previous tests we ran.

Reading about conversion on the Internet will give you plenty of hints to improve your website without first testing every little detail. You should definitely read Wheel of Persuasion and also Thijs’ previous posts on Yoast.

Conversion rate optimization should start with a critical view at your own site or checkout page. Could you improve largely upon it just by looking at the common knowledge about conversion?  Then first make these large improvements. After that, you can start testing and start fine-tuning. You can alter small things and test how you can further maximize your conversion.

 2.    You have very little conversions

Testing only makes sense if you actually have conversions.  You will need a fair amount of visitors and conversions to do an A/B-test properly. On sites with small amounts of visitors, the time period of a test will be rather lengthy. Otherwise, these tests will never come to any significant results. This will have consequences for the reliability of the results of your A/B-tests. The Z-statistic used in the A/B-tests is just not that reliable used in tests with a very low conversion rate. The same goes for tests with very small amounts of data. For more detailed information about the Z-statistic you can read my previous post. I think you should at least have 30 conversions a week to do proper testing. Note: that is my opinion, not a statistical law!

When you just started your site, or your site just does not have that many visitors, testing small changes in your website design for conversion improvements just does not make a lot of sense. In that case, you should also benefit from the existing body of knowledge of all these excellent conversion rate experts.


While optimizing the conversion on your website, you can use multiple tools. Testing can definitely be a good way to go, especially while fine-tuning your conversion.  However, you could also improve your conversion rate by applying knowledge and experience of other conversion rate experts.

New: Conversion Reviews on Yoast

And we have news for you! At Yoast we are currently fine-tuning (we’re are actually beta-testing already) a new product: the Conversion Review. This review will give you practical guidance in the changes you should make to improve the conversion of your website. Furthermore, the review will give you tips to set up your own A/B-tests to fine-tune and optimize the conversion even further. These Conversion Reviews will be sold at as of January 2014.

6 Responses to Should you test that?

  1. Alan Reeves
    By Alan Reeves on 20 November, 2013

    My problem is #2; not enough traffic and conversions. I’ve found that heat maps help. With a heat map, I can see not only where people are clicking but where they are spending most of their time on the page. So, if my call to action is at the bottom of the page, an A/B test will let me know if it is converting but a heat map helps me know how many people are even made it down to the button.

  2. Gorakhnath Sirsikar
    By Gorakhnath Sirsikar on 20 November, 2013

    “..applying knowledge and experience of other(s)..” is what one should take away from this. It is so similar to “we cannot all make the same mistakes over and over, learn from others’ mistakes & that’s how evolution happens”, was something my father used to tell us repeatedly growing up. Good to see your research concurs the idea behind it, for Interfaces too :)

  3. David
    By David on 22 November, 2013

    Great article. I often find that marketers and growthhackers push startups to start TESTING and measuring METRICS too early. Lars Logfren has a great post about this. It all depends on what stage you are in:

    I also advise companies with “too few traffic and conversions” to do a bunch of usability tests before stressing out. Use tools like or going into coffee shops and look at people use your website. There might be some very stupid basic reasons why people aren’t converting. You can fix those fast.

  4. Jeff Burritt
    By Jeff Burritt on 27 November, 2013

    Most small business owners I know often draw wrong conclusions from their traffic and conversions. Testing is only part of the equation. Without a properly defined target customer, quality traffic source, funnel system, call-to-action, and follow-up, they’re just guessing. So testing is just one part, albeit important part, of a larger cohesive strategy.

  5. Dillard
    By Dillard on 30 November, 2013

    Hi, Yeah – thanks for your insightful input! People should take heed of that, even experts can’t right all the time. Follow best practise for sure, but test, test and test some more! You’ll always unearth a few gems and ‘gotchas’ along the way.

  6. Reginald
    By Reginald on 8 December, 2013

    Hi Marieke,

    Well written and really agree with the conversion part. For me, if you have low traffic, it is very hard to plan out or test something.

    Thanks for sharing this.

Check out our must read articles about Analytics