Multivariate testing: what it is and when to use it

We’ve written quite some posts on Conversion Rate Optimization and A/B testing on yoast.com in the last few years. However, we’ve never really touched the subject of multivariate testing. In this post I’ll explain what multivariate testing is, its pros and cons and when you should (and should not) be using it.

What is multivariate testing?

Explaining what multivariate testing is, seems quite easy. In fact, when you look up multivariate testing on Wikipedia, it says the following:

“In internet marketing, multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous A/B tests performed on one page at the same time.”

However, I find this to be oversimplifying things a bit. With A/B tests you’re testing different versions of a complete page. This is not what you do with a multivariate test (MVT). With an MVT you’re testing elements within a page.

So lets say you have 2 versions of a headline, 2 versions of an image and 2 versions of a block of text. With a multivariate test, you will test every possibly combination to figure out which combination yields the best conversion rate. And this is where things get tricky. Because in the setup I mentioned above, there are already 8 combinations (2 x 2 x 2 = 8):

1. Headline A + Image A + Textblock A
2. Headline A + Image A + Textblock B
3. Headline A + Image B + Textblock A
4. Headline A + Image B + Textblock B
5. Headline B + Image A + Textblock A
6. Headline B + Image A + Textblock B
7. Headline B + Image B + Textblock A
8. Headline B + Image B + Textblock B

Advantages

The first advantage of multivariate testing is that you can see the effect small changes on your site have. Of course, small changes can also be tested with an A/B test, but that’s suboptimal as you can only test one small change at a time. Generally, A/B tests are used for the big changes and multivariate testing is used for optimizing smaller elements.

Also, by using a multivariate setup, you’re able to not only test the effect of changing one element, but you’re also able to test the combined effect (interaction effect) of several elements. Will changing an element still have the same effect if you change another element on that same page? These are questions you wouldn’t be able to answer with an A/B test but can answer with multivariate tests. With multivariate tests, you can pretty accurately see what the effect of each element is in which setup or situation.

Limitations

The biggest drawback of multivariate testing is that you need an even bigger amount of traffic and conversions than you do for an A/B test. I always say you should have at least 100 conversions on each variation. So if your multivariate test has 3 different versions of 3 different elements, you’ll need at least 2700 conversions. To illustrate: 3 x 3 x 3 = 27 combinations with each at least 100 conversions.

And that’s a lot for most websites, especially if we’re talking sales on a specific product. This means that the page needs to have quite some traffic on it, since most conversion rates aren’t that high.

Lastly, the multivariate test setup looks at a lot more variables than an A/B test and how these variables interact. This means there’s a bigger chance that mistakes or errors can occur in the reporting. So you should check your multivariate test results even more than your A/B test results.

When to use multivariate testing

As I’ve already mentioned above, multivariate tests are used to test smaller elements on a page. You’re testing small variations of the same element, instead of overhauling the complete page as you would in an A/B test. So first and foremost, multivariate testing isn’t supposed to be used as a starting point.

Obviously, before starting to test at all, there are a few things you have to do. First of all, check whether there’s enough traffic and conversions on the page you want to run a test on. Since an MVT setup increases your amount of variations really quickly, it’s important to have an abundance of traffic and conversions. You usually know whether your traffic and conversion rate are sufficient because you’ve probably A/B tested the page in question.

What’s next?

The next step is to find out what kind of changes you think are needed for that page. To do this you first need to be clear on what the goal of the page is. What do you want people to do on your page? Write down what you think could be the cause of visitors not completing this goal. You can determine this by being critical yourself, doing user testing (or even a survey) and looking at your analytics.

Now you know what your page’s goal is and what could be causing a lower conversion rate on this goal. The next step is finding out what you could possibly change to make more people fulfill that goal. You have to think of different versions of the factors that could be hurting your conversion rate.

When you’ve found your page’s goal and causes for it being suboptimally completed, you should formulate hypotheses. Write down what you’ll be changing, what effect you expect this to have and maybe even why. For instance:

• If we shorten the checkout form, more people will complete their purchase.

To my mind, hypotheses are even more important for multivariate testing, because it’s so easy to add more variations. Formulating hypotheses will prevent you from randomly adding new variations. Make sure you have a hypothesis for every single variation you’re creating.

Multivariate or A/B?

If you find your hypotheses are about changes that drastically alter the page’s layout or look, you’re better off running an A/B test. As said, A/B tests give you the possibility to determine whether one (version of a) page performs better than the other.

If however, you find that your hypotheses are about small changes (f.i. text on your call-to-actions) then multivariate testing could be a good choice. But you have to be sure your page meets the criteria for multivariate testing that I mentioned above.

What do you think?

Multivariate testing isn’t as easy as it might seem. It’s something you should only get into if you know what you’re doing. So I’m wondering: have you ever tried multivariate testing? And did you follow my steps? Or do you have other steps you think should be followed? Let me know in the comments!