In 30 fast slides (and some notes), I walk through: what is A/B testing, what you can test, what to measure, how you can test and an example of why you should test everything.
Slide 4 -
A/B testing allows you to quantify user behavior and optimize user experience. It is rooted in
<a href="http://en.wikipedia.org/wiki/Statistical_hypothesis_testing", alt="Wikipedia" rel="nofollow" target="blank">statistical hypothesis testing.
Slide 5 -
Some of the front-end, UI tools you can use (back-end is a different conversation):
- <a href="http://optimizely.com?ref=stephen.fm", alt="Optimizely" rel="nofollow" target="blank">Optimizely
- <a href="https://vwo.com/?ref=stephen.fm", alt="Visual Website Optimizer" rel="nofollow" target="blank">Visual Website Optimizer
- <a href="https://developers.google.com/analytics/devguides/collection/gajs/experiments", alt="Google Experiments API" rel="nofollow" target="blank">Google Experiments
- <a src="http://unbounce.com/", alt="Unbounce" rel="nofollow" target="blank">Unbounce
Slide 7 -
An example of a test for CatCo's meme website, which shows half of site's visitors the normal layout and the other half a red background. The ugly red background apparently converts more.
Slide 8 - A few of the metrics you can use to measure performance. Some are easier to track than others.
Slide 10 -
This is a Basecamp (formerly 37 Signals) landing page test. These are two radically different designs, which are great to help optimize towards a global maximum. Signal v. Noise, has dozens of great posts like <a href="https://signalvnoise.com/posts/2977-behind-the-scenes-highrise-marketing-site-ab-testing-part-1", alt="Highrise A/B Testing" rel="nofollow" target="blank"> this one on how they A/B test their products.
Slide 12 -
Another example of a great test to run.
Slide 14 -
Headers and call-to-actions are easy to test and are likely the highest ROI opportunities starting out.
Slide 19 -
If you go to any major tech website, like Facebook or Google, you've been included in an A/B test. All the big guys do it and have been testing for years. Amazon first started A/B testing way back in 2004. Google preforms thousands of A/B tests on their search algorithms.
Slide 21 -
The majority of your tests won't be statistically significant or won't produce the outcome you want. This is simply the nature of the statistical beast.
Mathematically, you need a fairly large sample size to determine significance. Unless you have a conversion rate that is out of this world, you'll need thousands, tens of thousands of visitors to get a statistically significant sample. Check out
<a src:"http://www.evanmiller.org/ab-testing/sample-size.html", alt="Evan Miller's A/B Testing Tools" rel="nofollow" target="blank"> Evan Miller's A/B Testing Tools to get a sense of the sample sizes you need.
Slide 30 -
The results from the multi-variate landing page test that the 2008 Obama Campaign ran. Quite amazing when you think about it.
- <a href="http://amzn.to/297wpjL", alt="Amazon - A/B Testing" rel="nofollow" target="blank"> A/B Testing by Dan Siroker & Pete Koomen
- <a href="http://kylerush.net/?ref=stephen.fm", alt="Kyle Rush"rel="nofollow" target="blank"> Kyle Rush's Blog
- <a href="http://www.wired.com/2012/04/ff_abtesting/all/", alt="The A/B Test: Inside the Technology That’s Changing the Rules of Business "rel="nofollow" target="blank">Wired - The A/B Test: Inside the Technology That’s Changing the Rules of Business</a?