A/B Email Testing

  • Updated

Test multiple email variations and automatically launch the best-performing one with A/B email testing. For example, you can test:

  • Two different subject lines
  • Two different calls to action
  • Combinations of these (2 subject lines x 2 calls to action = 4 variations to test)

Up to five email messages can be added to an A/B email test.

Set Up An A/B Test

  1. Go to Outbound > A/B Test Messages and click New A/B Test.
  2. Enter a Test Name and Description.
  3. Under Distribution List(s) > Send To choose the lists/segments to target with these emails. If required, add Suppress Against segments to exclude contacts in those segments.
  4. If you have a connected CRM you can select the check box to Attach sent notes to contacts or leads from my CRM lists.
  5. Under Messages To Test, choose the emails to test. You can select both regular and legacy emails here.
  6. Under Test Run, choose your Sample Size.
    • # of contact records (must be 10+)
        or
    • % of your total list (must be 33% or less)
  7. Pick a scheduled date & time for the initial A/B test.
  8. Under Final Run, pick a time for the final run (after the test runs, a specific time, or no schedule).
  9. At the top, click Save, then Schedule Test to begin testing.

Final Run

Act-On will select the winning message based on the one with the highest number of clicks (not opens). In the event of a tie, the message with the highest click ratio will be chosen.

The winning message will go out to the remaining contacts on your list who did not receive the testing emails. The recipients of the test emails will appear in the 'Not Sent/Suppressed' section of the email report, with the reason shown as 'Suppression List.' This simply indicates that they already received the test versions of the message.

Manage Your Tests

Drafts

Saving a new A/B email test creates a draft – you can access it later via Messages > A/B Testing > Drafts.

Winning Messages

After all test runs have been sent, you can choose your own winning message & override Act-On's choice. Open the test, hover over your desired message, and click the check box that appears. Make sure to save the test for this change to take effect.

Distribution List Issues

If your test list includes a large number of suppressions, the final send may go out to fewer contacts than the test messages did.

To mitigate these issues, use good list maintenance practices, remove hard bounces & opt-outs from your list, and/or segment your list to reduce the number of suppressions. If these measures don't prevent the problem, you can choose a smaller sample size for your test.

Active Tests

In the Active section, you can see the status of tests and access the dashboard for any active test.

Messages sent in a test run can also be found under Outbound Triggered Messages Other.

Canceling a Test

If you cancel an active A/B email test in the Active section:

  • If the test has not run, the canceled A/B email test will return to the Drafts page. You can edit the test and launch it again if desired
  • If the test has run, you will be prompted to confirm the cancellation. After you confirm, the final run will be canceled and the record will move to the Completed tab. This cannot be undone
    • If you do not wish to cancel the final run, but only remove its scheduling, edit the A/B email test via the dashboard. Change Final Run to Not Scheduled and then click Save

Completed Tests

The Completed section shows all completed tests (or those canceled after the test run was sent). Click a completed test in this list to see the test's report dashboard.

Final-run sent email messages are also shown under the Messages > Sent menu. When viewing the message report for the final send, contacts who received one of the test versions will show as suppressed for "Suppression List".

FAQ

Why did email A win when email B looks to have had more activity?

Act-On shows all of the activity for the emails. If email B appears to have had more activity, then this was after the test deadline, for example, if the test ended at 11:00 and various clicks on email B were all at 11:30, 11:45, etc. So if email A had more clicks at the 11:00 test deadline, then Email A would win.

Was this article helpful?

Have more questions? Submit a request