test me, i dare you

Remember how excited you were about testing sitting around the conference table?  Yea I know, now it’s real work but a little elbow grease will yield good, actionable results so stick it out.

At this point you have a good feel for what your assumptions are…will the blue CTA win out?  Will the text-only email perform better than an image-heavy version? Take bets and make some guesses.  The only way to find out is to let it go then sit back and wait.  Here comes the least popular thing you can tell a marketer…one test doesn’t give you all the answers.  You want to drive results right?  This is all about conversions and what does best so you have to know whether or not you have significant results.  No, it’s not as simple as saying email A got an 8% open rate and email B got a 9% open rate so BOOM email B worked!  It’s a little more than that.  Some marketers are lucky enough to work with statisticians on staff, but that’s a rarity.  So how do you decide something is significant?  Math of course (scary, I know).    In textbook terms you need to run significance testing.  Good news, there are calculators online for that!  Bad news, you still have to know things to be able to use them effectively.  So before I point you to the calculator here’s what you need to know:

Low Confidence Level is defined as being in the range of 90-95% while High Confidence Levels are defined as 95-99%+. If you plan to conduct multiple tests or have low traffic, go with the Low level. If you have high traffic or “significant business value flowing” through the page, go with the High Confidence level.How do I know this you ask?  Because I love to read and I spent some time reading a great book on Landing Page Optimization.  It’s a well written book for those who want lots of detail on Landing Page optimization, definitely recommend it, however for your “Cliff Notes” version of testing all you need to know is whether you are going with Low or High confidence.   The place to use this new found knowledge?  Well this handy calculator of course!
So what happens if you are testing email and run the results through the calculator, and GASP! your results aren’t significant??  Run your test again!  No, I don’t mean email the same people with the same stuff all over again…not unless you want to make your audience angry.  What I mean is develop a new email that tests the same hypothesis and send it to the same folks again. Don’t be afraid to retest your assumptions and even known results over time.  Your audience changes and so do the things that cause them to react, so don’t be afraid to revisit old tests to see how things change.
What if you aren’t testing email, what if you’re building a fabulous landing page and need to do some testing there, how do you determine significance if you don’t have fancy schmancy software to help you out? Well you can read all about it here on this great post by Hubspot where they even have a handy spreadsheet to use in calculations.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s