The Scientific Method of Marketing: A/B Testing

Ever wonder how the 2012 Obama Campaign raised $690 million over the internet? Such a shockingly large amount of money was mostly raised through email fundraising. This makes me wonder about the recent fundraising email I sent out to family and friends asking to donate to my relay for life team…should I have done more research and testing first?

A/B testing was utilized by the Obama campaign in order to determine the most efficient way to fundraiser through email. The email was tested in the following areas:

  • Subject Line
  • Amount of money to ask for
  • The message
  • The format

A/B testing removes the guesswork. Instead making decisions off of “we think” we know, A/B testing insures one is making education “we know” decisions. A/B testing  is an easy way to test changes to your website or emails against your current design and structure. The test shows you which changes were most effective. The Obama campaign found that using unattractive giant size font for links and highlighted yellow sections in emails had the best success rate. Even profanity was found to be successful, “Hell Yeah I Like Obamacare”.

What I found fascination about the fundraising email is the constant need to change and adjustment. The Science Behind Those Obama Campaign E-Mails, points out, “There was no such thing as the perfect e-mail; every breakthrough had a shelf life. “Eventually the novelty wore off, and we had to go back and retest”.

Amelia Showalter, the former director of analytics for the the 2012 Obama campaign was featured in ELLE magazine as the “most eligible hires in techdom”. Schowalter used A/B testing to determine how to write and format a visually effective email. Her research went as far as to determine if people still enjoyed receiving emails from the president addressed  you by name. Check out Amelia’s TedTalk, Testing it will free your mind.

Amelia leaves you with 5 tips to free your mind:

  1. Start somewhere. Don’t force yourself to be perfect.
  2. Fail upwards. Testing can prove your instincts wrongs.
  3. Test all the time. Make it a priority. It will force you to be more innovate.
  4. Surround yourself with diverse people.
  5. Trust in testing.

Instead of a marketing team solely listening to Highest Paid Person’s Opinion (HiPPO), A/B testing relies on scientific decision making.

Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO, explains some limitations of A/B testing. A/B testing utilize statistics significance to understand which variant is better. However A/B testing  does not always indicate why the variant was better. Its important to remember when changing the navigation of a website, it takes viewers time to learn the new navigation system. In order to collect useful data, the new navigation system should be tested for a few weeks.
Presidential Campaigns are not the only ones using A/B testing. HawkHost, is a high quality viral hosting and reselling service. HawkHost used A/B testing in order to improve their website with their goal being a sale. HawkHost first tested their homepage heading and subheading as their variant.  The heading and subheadings message, color and font combination was tested. Even which letters where put in bold font or underlined was tested. The changes yielded a increased in conversion rate by 3 times the original.
Remember, don’t rely on the HiPPO to make decisions for your business or campaign. ALWAYS TEST. You never know what results will surprise you.
The Scientific Method of Marketing: A/B Testing

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s