Split Testing Emails

1. Add a Split element on your Campaign Map
2. To split the traffic unequally, click the Split element and configure the Settings
3. Add an email message to the ADD VERSION A path.
4. Add an email message to the ADD VERSION B path.
   
   
   

After publishing your Campaign Map, Performance Mode will show you the statistics for each path down the Split element.

After you have compiled statistics on both the Version A and Version B paths of the split you can adjust the percentage sent down each path. Return to the Campaign Map, enter Edit Mode, and click on the Split element. In the Settings section, slide the slider to "100%" under the winnner (Version A or Version B).

Can I split test more than two versions of the email?

No, the Campaign Builder only allows A/B testing in the Split filter element.

Is there any way around this? I want to split test 500 different versions of 12 different emails.

Unless you really know what you are doing we don't recommend spending too much time on complicated split testing schemes. Your time is better spent improving your offer to your customers. However, you can split test other Campaigns. Each of the Campaigns could have an A/B split test of an email, effectively giving you a A/B/C/D split test. We don't recommend this approach as the statistics for each split test are shown in Performance Mode on the different Campaign Maps.

What about multivariate testing? I've heard its better than regular split testing.

ONTRAPORT does not support multivariate testing. A split test will show you which email you compose is more effective, and you can easily interpret the results. A multivariate test will have multiple options in each test, and requires a huge population of contacts to garner meaningful results.

What is a significant difference between the two versions? I only have one percentage point difference in "Email Opens"

You can search for articles on "statistical significance in A/B testing" and find out that it depends on the number of contacts who have been through the split ("sample size"), the nature of the difference between the two versions (is it really significant to use the Oxford comma in one version?), and the confidence level of the test. There are on-line calculators you can use to determine if the test results are significant or not.

I tested different colored backgrounds and didn't notice a change in my "open" rates. What gives?

Test "things that matter" like subject lines if you are testing the number of "opens". While a background color may affect link clicks or other stats after the email is opened, it won't affect the number of opens.

I put two identical emails on each side of the split test, and one performed 10% better with 100 contacts! It's broken.

That happens with small sample sizes; the smaller the sample size the less important the percentage of difference is between the A and B versions. As more go down each path you will see the difference narrow.