Multivariate Test Results with Optimizely

The last couple of weeks I’ve been experimenting with Optimizely, a testing tool founded by two former Google product managers, trying to optimize the conversion rate of this blog with the help of a multivariate test. Multivariate tests, also called multivariable tests, provide insight into which content or creative variation produces the best improvement in the defined goals of a website.

Just to be clear: this article is not an “how to perfom a multivariate test with Optimizely” article, but is about the results of some of my test sessions with Optimizely. By the way, this is not a paid review.

How to prepare your multivariate test?

With good testing comes a good preparation. Know why and what you would like to improve and design successful and meaningful test scenarios to achieve your goal you’ve set out.

3 vital elements you should know before you start fiddling with testing tools:

• what are the top tasks, conversion goals or challenges of the page?
• what will you test to improve CTR?
• what are your test variables?

1. What are your top tasks

Determine your visitors’ top tasks by clearly identifying the goals of your website.

Some examples:

• Improving sign-up rate,
• Reducing bounce rate,
• Increase newsletter subscriptions,
• Increase downloads,
• Optimizing sales

If your website is serving multiple purposes, such as getting more subscribers and increasing your visitors’ engagement, the best strategy is to tackle one challenge at a time.

2. What should you test to improve CTR?

Create a list of hypotheses for the low conversion rate and narrow it down to the ones which you think have the highest impact on the conversion goal. Use web analytics data and usability testing to determine what influences conversion rate.

Some examples:

• Maybe your sign-up form is too long,
• Maybe your CTA button isn’t noticeable,
• Maybe the product page isn’t clear enough.

3. Do you have meaningful test variations?

Come up with some good alternatives for your test subject but be careful not to test too many of them. The more variations you have, the more traffic you’ll need because you are dividing your traffic to more variations. In fact, the traffic requirements for a test will grow exponentially with each variation. On average it takes 100 conversions per variation to gather some statistical significance.

More variations = More traffic = More time

If you are selling products, you could test the way you visualise them on a products page. Change the message of the CTA button and let it stand out, add special price promotions, add star rating, increase the size of the product thumbnails, …

Visual Website Optimizer made a simple calculator to estimate how much traffic your test will require to have significant results.

My multivariate test results

I’ve decided to share these results to show you all how easy it can be to optimize your CTR. It doesn’t always require expensive tools and extensive knowlegde of coding.

1. Increase ‘circulation traffic’

The main goal of this blog is to inform people on the latest trends and research about Usability, User Experience and Web Design. I want my visitors to consume news as much as possible and read articles one after the other. Improving ‘circulation traffic’, better known as the metric “pages per visit”, is essential to accomplish this.

I made a list of hypotheses of which three came out as most likely threats to this goal:

• the ‘Calls to action’ on the homepage to the articles aren’t noticeable enough (‘continue reading’),
• the list of popular articles in the sidebar isn’t visible enough,
• the list of resources in the sidebar isn’t visible enough.

 

For the first test I designed some variations for the ‘continue reading’ link on the homepage. As you can see in the image underneath, the blue underlined version outperformed the original, the orange CTA button approach and the orange text link.

A possible explaination would be that throughout the website all links integrated into paragraphs are blue and underlined so people are more familiar with it. Jakob Nielsen mentioned this back in 2004 in his “Guidelines for Visualizing Links“.

 

The test concerning the list of ‘Popular Articles’ section had the same goal as the previous one: increasing the amount of clicks to the articles. The results are showing that ‘Popular Articles’ outperformed the original version with 110.6%, which means I doubled the CTR.

 

A similar test was focused on the list of ‘Latest Resources’. Again, the goal was to increase the amount of clicks. I didn’t add any resources during the testing period to keep the test clean. ‘Resources on UX’ and ‘Community News on UX & Usability’ performed way better than the original and ‘Latest Resources on Usability & UX’.

 

2. Increase the ‘social awareness’ and news subscription methods

Not long ago I designed some circular icons in the sidebar linking to my RSS feed, Twitter account, Facebook page and Newsletter Subscription. Although not many people clicked on these icons according to Google Analytics (and therefore not the best test subject), I was interested to see if the icons would be clicked more frequently if I changed the transparency on hover effect.

 

3. Increase clicks on affiliate links

A while ago I added Amazon affiliate links into some of my articles, mainly to pay for the hosting and to organize contests where I would need to buy the prices. I have two items leading to the product page of Amazon: the list of books in the sidebar and an article titled “A must have book collection on UX“.

I came up with four variations for the title in the sidebar above the books. Results are pretty similar to the ‘Popular Articles’ test: the word “Poplar” works very well. However, in combination with the word “most” we see a decrease in clicks. I’m not really sure why “My book collection” and “Favorite UX books” didn’t work. Can I assume that these two variables are too personal and visitors prefer to know what the general crowd is reading?

 

Up to the last test. I introduced two labels called “Most popular” and “Popular” into the article about UX Books. These labels are given to the books which are bought most by our visitors.

Adding these two labels definitely pays off.

What’s next?

The multivariate test is finished and you’ve gathered the test results. Now what? Make the change by putting the winning variation online, design good follow up tests and successful test scenarios, assess the results and start all over again. Testing really is a never-ending story. I always integrate the original of the very first testsession into the next as an additional testversion.

What have you been testing lately? Share us your results!