Comments on: And the winner was…

By: David Hamill

David Hamill — Wed, 03 Dec 2008 07:29:46 +0000

Zachary I spoke to Alasdair from Maxymiser on your first point. Apprently his client only allowed a little bit of the results to be used in publicity so it was just clickthroughs. My own concern was that version 4 was the only one that wouldn't introduce a hidden cost later on in the process.

I can only guess that it version 3 was the best in terms of conversion because this is the page they have on their live site now.

By: Harry Brignull

Harry Brignull — Thu, 20 Nov 2008 17:42:05 +0000

Zachary, you are asking all the right questions. This wasn't my own study - I am just reblogging work done by Maxymiser. There are more details about the study in previous post. Maxymiser assure me that the study reached P<0.05 i.e. 95% confidence level in this study. The conversion measured was clickthroughs into the checkout process.

By: Zachary Abbell

Zachary Abbell — Wed, 19 Nov 2008 21:12:01 +0000

Did you measure conversion/sales rate and/or click thru rate? If only conversion rate, how can you tell that any individual version was really more successful in purchasing(as the checkout funnel is a multiple step process which isn’t displayed in the test)? I’d be interested to find out more about how these numbers were determined.

Additionally, what statistical significance did you reach? 90%? 95%? 98%? 99.9%?

By: David Hamill

David Hamill — Mon, 17 Nov 2008 16:59:56 +0000

Oh, there was me leaping to your defence and you’d gone and done it already :0)

By: David Hamill

David Hamill — Mon, 17 Nov 2008 16:57:51 +0000

I disagree Pedro. Maxymiser works on lives sites, not on development ones. It’s also not a recruited sample. It’s real customers buying real products.

All versions run concurrently, so seasonal variations are not an influence either as far as I’m aware.

By: Alasdair Bailey

Alasdair Bailey — Mon, 17 Nov 2008 16:55:00 +0000

Hi guys,

To address some of the points that James raises at the top of the thread, the Maxymiser system monitors statistical chance to beat all percentages to ensure that statistical validity is reached before we draw conclusions.

We also identify new vs returning visitors so that a returning visitor always sees the same variant – important for usability and the stats.

We can further test changes on multiple pages, something many of our competitors struggle with, making multi page form testing possible.

Delivery cost is a tricky one, you’re right on that point. I believe in this particular test that the client was interested in discovering the point at which this should be presented in order to minimise exit rates. Multivariate testing can be used progressively alongside funnel analysis to push more traffic through the funnel one page at a time.

Feel free to contact me privately or follow up here with any questions, I will monitor this page for a few weeks.

Kind regards,
Alasdair (Maxymiser)

By: David Hamill

David Hamill — Mon, 17 Nov 2008 16:45:15 +0000

I was considering this myself, so I had a look at the pages in the process.

You need to provide lots of information before you see the delivery costs.

I’d be interested in seeing how a page worked just like version 3 but with an intelligent/dynamic and simple provision of delivery costs up front.

There are two factors I’m really interested in.

Is it the complexity of the delivery costs that causes the problem. Or is it the fact that they are introduced so early in the process?

If the delay of delivery costs were to cause an uplift in conversion then that really would be interesting.

I know, I should get out more.

By: Pedro

Pedro — Mon, 17 Nov 2008 16:38:35 +0000

The 1st comment is correct. What is the margin of error?

By: Harry Brignull

Harry Brignull — Mon, 17 Nov 2008 16:19:13 +0000

If you want to see some larger screen grabs, click here.

By: Ed

Ed — Mon, 17 Nov 2008 14:30:56 +0000

:-)

It is facinating to see the results. Can anyone explain the difference in performance between Version 5 and Version 4?

By: David Hamill

David Hamill — Mon, 17 Nov 2008 12:35:11 +0000

That’s really interesting. I couldn’t have been more wrong. There’s not a huge difference between the best and the worst. An extra button and some blurb about delivery. Perhaps the delivery blurb was a bit off-putting?

By: Harry Brignull

Harry Brignull — Mon, 17 Nov 2008 11:41:03 +0000

Yes, I didn’t really start talking about the drawbacks of the method – but like anything else it has its strengths and weaknesses. If you have lots of combinations you want to test, this creates a combinatorial explosion and can take months to get enough data, even from a fairly high volume site.

Also, it’s clearly a useless method if you don’t have a live site with a healthy user-base.

By: James

James — Mon, 17 Nov 2008 11:18:37 +0000

and the Margin of error was?

A big caveat that you don’t mention is that this sort of testing only works on high volume sites. To get a significant result each variant would need many exits. This depends on the uplift of each variant. This is to make sure that you are not measuring noise instead of real behaviour. The test also needs to be done over a short period of time to make sure that your results are not influenced by seasonal factors.

The other negative is that user can be faced with a changing user interface. If I log in from home and the checkout button has moved around you may lose the sale.

This method of testing is great for working out very small changes on very large volume sites.