Beating A/B testing with 20 lines of code

flickr/Zach Klein

Finding the best interface choice and enticing banner ad relies on A/B testing by users. But machine learning can replace the focus groups with 20 lines of code.

That's the idea presented by Steve Hanov using the example of which slot machine in a casino you suspect will pay the best. For websites, that may become a Buy Now!! button color. Rather than have thousands of potential customers test it, Hanov suggests using code and the epsilon-greedy method, the simplest strategy to the most popular choice, derived from the casino example.

Using an example from one of his own sites, Hanov reports a green button outperformed orange and white buttons. Why don't people use this method? Because most don't understand machine learning algorithms, and mainstream tools don't support it. And if you need a refresher on A/B testing, try, "The Ultimate Guide to A/B Testing" relying on alternate pages based on user choices.

I like it

beating a random strategy (or the one that relies on random for a portion of its execution) is always a tough proposition. This is a really nice both pragmatic and practical approach - love it.

Tim on

I actually noticed that Google was doing this with my Adwords account a couple of weeks ago.

steve8918 on

Definitely going to use this. Perhaps this method should be called A/Z testing or A/n testing.

solecoder on

Next steps

Decision optimization is the next step past A/B testing.

Kevin on

I think this comes down to the debate between the marketer and the designer. And it will always be a great debate.

Jen Walter on

I've seen minor tweaks to a form raise conversions by 20%. The person running a particular test may not even know that a particular change was significant, or even that it happened.

btilly on

Other options

One hypothetical case it could end up serving the worse design more often is if you had times of the day where your users behave very differently due to time zones, i.e. Europe vs North America.

medell on

I disagree with the notion that you should never let artistic vision trump the results of a test.

Andrew Ingram on

However, to claim "The strategy that has been shown to win out time after time in practical problems is the epsilon-greedy method" is false.

noelwelsh on

Will this help resolve any arguments about building your website?

For the latest IT news, analysis and how-tos, follow ITworld on Twitter, Facebook, and Google+.

Now read this:

Developer declares 'I am done with the Freemium Business Model'

Khan Academy offers JavaScript as their first computer language

Study says Facebook profile can predict job performance

This story, "Beating A/B testing with 20 lines of code" was originally published by ITworld.

Copyright © 2012 IDG Communications, Inc.

Shop Tech Products at Amazon