A Crash Course on A/B Testing

Stephen Puiszis -

Presentation Notes

Slide 4 -
A/B testing allows you to quantify user behavior and optimize user experience. It is rooted in
<a href="http://en.wikipedia.org/wiki/Statistical_hypothesis_testing", alt="Wikipedia" rel="nofollow" target="blank">statistical hypothesis testing.

Slide 5 -
Some of the front-end, UI tools you can use (back-end is a different conversation):

  • <a href="http://optimizely.com?ref=stephen.fm", alt="Optimizely" rel="nofollow" target="blank">Optimizely
  • <a href="https://vwo.com/?ref=stephen.fm", alt="Visual Website Optimizer" rel="nofollow" target="blank">Visual Website Optimizer
  • <a href="https://developers.google.com/analytics/devguides/collection/gajs/experiments", alt="Google Experiments API" rel="nofollow" target="blank">Google Experiments
  • <a src="http://unbounce.com/", alt="Unbounce" rel="nofollow" target="blank">Unbounce

Slide 6 - An example of including the Optimizely into a Rails application. Test variations are created using JavaScript or jQuery, depending on the application's requirements.

Slide 7 -
An example of a test for CatCo's meme website, which shows half of site's visitors the normal layout and the other half a red background. The ugly red background apparently converts more.

Slide 8 - A few of the metrics you can use to measure performance. Some are easier to track than others.

Slide 10 -
This is a Basecamp (formerly 37 Signals) landing page test. These are two radically different designs, which are great to help optimize towards a global maximum. Signal v. Noise, has dozens of great posts like <a href="https://signalvnoise.com/posts/2977-behind-the-scenes-highrise-marketing-site-ab-testing-part-1", alt="Highrise A/B Testing" rel="nofollow" target="blank"> this one on how they A/B test their products.

Slide 12 -
Another example of a great test to run.

Slide 14 -
Headers and call-to-actions are easy to test and are likely the highest ROI opportunities starting out.

Slide 19 -
If you go to any major tech website, like Facebook or Google, you've been included in an A/B test. All the big guys do it and have been testing for years. Amazon first started A/B testing way back in 2004. Google preforms thousands of A/B tests on their search algorithms.

Slide 21 -
The majority of your tests won't be statistically significant or won't produce the outcome you want. This is simply the nature of the statistical beast.

Slide 22
%p
Mathematically, you need a fairly large sample size to determine significance. Unless you have a conversion rate that is out of this world, you'll need thousands, tens of thousands of visitors to get a statistically significant sample. Check out
<a src:"http://www.evanmiller.org/ab-testing/sample-size.html", alt="Evan Miller's A/B Testing Tools" rel="nofollow" target="blank"> Evan Miller's A/B Testing Tools to get a sense of the sample sizes you need.

Slide 30 -
The results from the multi-variate landing page test that the 2008 Obama Campaign ran. Quite amazing when you think about it.

Sources:

  • <a href="http://amzn.to/297wpjL", alt="Amazon - A/B Testing" rel="nofollow" target="blank"> A/B Testing by Dan Siroker & Pete Koomen
  • <a href="http://kylerush.net/?ref=stephen.fm", alt="Kyle Rush"rel="nofollow" target="blank"> Kyle Rush's Blog
  • <a href="http://www.wired.com/2012/04/ff_abtesting/all/", alt="The A/B Test: Inside the Technology That’s Changing the Rules of Business "rel="nofollow" target="blank">Wired - The A/B Test: Inside the Technology That’s Changing the Rules of Business</a?