A/B Email Testing

Use A/B email testing to test multiple email variations and automatically launch the best-performing one. For example, you can test:

• Two different subject lines
• Two different calls to action
• Combinations of these (2 subject lines x 2 calls to action = 4 variations to test)

Up to five email messages can be added to an A/B email test.

Set up an A/B Test

1. Navigate to Outbound > A/B Test Messages and choose New A/B Test.
2. Enter a name and description.
3. Choose the list to target with these emails.
4. Choose your sample size.
   • # of contact records (must be 10+)
       or
   • % of your total list (must be 33% or less)
5. Select whether to add notes in your CRM for test messages.
6. Choose messages to test.
7. Pick a scheduled time for the initial A/B test.
8. Pick a time for the final run (after the test runs, a specific time, or no schedule).
9. Click Save at the top, then Schedule Test to begin testing.
   
   

Final run

Act-On will select the winning message based on the one with the highest number of clicks (not opens). In the event of a tie, the message with the highest click ratio will be chosen.

The winning message will go out to the remaining contacts on your list who did not receive the testing emails. The recipients of the test emails will appear in the 'Not Sent/Suppressed' section of the email report, with the reason shown as 'Suppression List.' This simply indicates that they already received the test versions of the message.

Manage Your Tests

Drafts

Saving a new A/B email test creates a draft – you can access it later via Messages > A/B Testing > Drafts.

Winning Messages

After all test runs have been sent, you can choose your own winning message and override Act-On's choice. Open the test, hover over your desired message, and click the check box that appears. Make sure to save the test for this change to take effect.

Distribution List Issues

If your test list includes a large number of suppressions, it is possible for the final send to go out to fewer contacts than the test messages did.

To mitigate these issues, use good list maintenance practices, remove hard bounces & opt-outs from your list, and/or segment your list to reduce the number of suppressions. If these measures don't prevent the problem, you can choose a smaller sample size for your test.

Active Tests

In the Active section, you can see the status of tests and access the dashboard for any active test.

Messages sent in a test run can also be found under Outbound > Triggered Messages > Other.

Canceling a test

If you cancel an active A/B email test in the Active section:

• If the test has not run, the canceled A/B email test will return to the Drafts page. You can edit the test and launch it again if desired
• If the test has run, you will be prompted to confirm the cancellation. After you confirm, the final run will be canceled and the record will move to the Completed tab. This cannot be undone
  • If you do not wish to cancel the final run, but only remove its scheduling, edit the A/B email test via the dashboard. Change Final Run to Not Scheduled and then click Save
    
    

Completed Tests

The Completed section shows all completed tests (or those canceled after the test run was sent). Click a completed test in this list to see the test's report dashboard.

Final-run sent email messages are also shown under the Messages > Sent menu. When viewing the message report for the final send, contacts who received one of the test versions will show as suppressed for "Suppression List".

FAQ

Why did email A win when email B looks to have had more activity?

Act-On shows all of the activity for the emails. If email B appears to have had more activity, then this was after the test deadline, for example, if the test ended at 11:00 and various clicks on email B were all at 11:30, 11:45, etc. So if email A had more clicks at the 11:00 test deadline, then Email A would win.