Doing an A/B Test
A/B Testing is used in order to test different versions of your e-mail in real-time and to easily and quickly find out which version is the best to send to your profiles. In this way, you can test content variability within the same mail, ranging from simply having two different subjects to having two completely different versions of the message. This will allow you to draw conclusions on what worked best and for which reason.
After creating your A/B campaign, Actito will send each version of the e-mail to a sample of profiles in your database. You can have up to 4 different versions. Please note that the target population sample will always be the same for each version.
After a test period of the duration of your choice, Actito will calculate which e-mail worked best according to either the opening or the click rate. The best performing e-mail will then be sent to the rest of the target (people who have not been reached yet).
Creating your A/B Campaign
A/B testing is a campaign mode in itself, like Marketing or Scenarized e-mails. In order to create an A/B campaign, is therefore necessary to make your selection right at the creation of your message.
In "Manage e-mail campaigns" (Catalog > E-mail campaigns > Manage e-mail campaigns), click on "Create a new campaign".
Next, select the "A/B TESTING" mode.
In practice, after the test period, an A/B campaign will behave like a Marketing campaign. It is therefore used to contact a large number of profiles.
If you want to make A/B tests on Scenarized e-mails, it is possible to use the "A/B split" block in the Scenario editor.
Step 1: General Data
As for every campaign, the first step will consist in giving a name to your campaign, selecting the target Database, the entity and the language of the campaign.
In A/B campaigns, you only have the possibility to select one definition language. Indeed, A/B testing is used to determine which version of your e-mail works best, but the behavior of your profiles might vary greatly according to their culture and their language. A more informal message might work best in a language but not in another, and idioms that cannot be translated might prevent you from having exactly the same message in two languages, in which case the selection criteria will be distorted.
If you want to contact your whole database, it is possible to select a default language at step 2 in order to contact all your profiles in the same language.
AB test parameters
Next, you will be able to select:
- The number of versions of your e-mail: you can test up to 4 four different versions.
- The parts of the e-mail on which you want to test variations. You can combine a different version of:
- The sender
- The subject
- The pre-header
- The content
Message headers
If you decided to test variation on the sender, the subject and/or the pre-header, you will be invited to provide each version in the 'Message headers' section.
Step 2: Target Group
As for any campaign, you will have to select the Subscription and the target population of your e-mail. You should target all the profiles that you want to contact: the test population will be selected randomly and the repartition criteria will be set at a later stage.
If you want to contact profiles no matter what their mother language is, you can un-check the 'Language' box to contact everyone in the language chosen at step 1.
Step 3: Message
You can create your message from a template, by importing an HTML or by coding your own HTML.
If you import HTML files to do an AB test on the content, the ZIP file must include as many HTML files as versions, with the version being specified in the suffix of the file name, under the format "_a", "_b", etc.
If you chose to do A/B testing on the content at step 1, you will have to define a specific content for each version.
You will be able to jump from a version to another by using the drop-down list in the top right corner of the message creation screen.
In the template editor, the version that you are editing will be displayed in the top left corner.
What if you only have minimal variation between your versions? In the template editor, you can copy the content of another version by clicking in the 'Options' in the top left corner and choosing 'Import content from another version'. After that, you are free to apply the variations on which you want to do A/B testing.
Step 4: Check and Test
After creating your message, you will be able to test it by selecting a sample of profiles from your testing group. Then you will have to provide one or more e-mail addresses to which the test e-mails will be sent.
The "A/B" column enables you to select which version you want to generate for each test profile. If you leave the default value "A/B", each version of the e-mail will be generated for this test profile (including version C and D if they exist).
Alternatively, you can choose to only select one version from the drop-down list for each profile.
Step 5: Sending
At step 5, you will get a summary of the general data, the message headers and the target group of your campaign.
In addition, you will be able to set the AB testing parameters.
AB testing parameters
- Sample size for versions: a sliding bar will allow you to choose the repartition between the test version and the winner version. If you have more than 2 versions (A/B/C/D testing), each version will automatically have the same sample size.
- Optimization criteria: you can choose the criteria used to determine the winner version, either the opening rate or the clicks rate.
- Period: determine the test period at the end of which the winner version will be decided. It can be set either in hours or in days.
After selecting the AB testing parameters, you can choose to send the campaign right now or to schedule it for later.
It is possible to set up an AB test with 100% of the target as sample, for example if you want to identify the best version in order to optimize your next campaigns.
To do so, you will still need to choose an optimization criteria and a test period. They will have no impact on sending, as the e-mail will be sent immediately sent to the whole target, but they will be used to identify the winner version in the campaign report.
Analyzing your A/B campaign
Once you're a/B campaign has been sent, it will be available in the "In progress" tab during the duration of the test period that you defined at step 5 of the campaign creation.
During the test period, it is not possible to "Correct" this campaign. However, it is possible to "Finish" it if you do not want to send the winner version of the e-mail.
If you want to check the repartition of the sample population or view each version of your e-mail, you can click on the "View definition" button.
View reporting during the test period
You can view the campaign report at any time in order to check the evolution of the opening or click rate of each version in real time.
Do to so, click on the "View report" button.
The "Summary" tab, which is found in every campaign report, will provide the global stats of your campaign. To view the repartition of you're A/B campaign, go to the "A/B testing" tab, which is only present for this specific type of campaigns.
This lets you see the number of e-mails sent for each version. The target population sample will always be the same for each version, but slight variations may appear, when addresses are filtered for instance.
Next, you can compare the delivery, opening and click rate of the different versions of your e-mail. It can be done at any time during the test period.
Viewing the optimized reporting
When the test period is finished, the campaign will appear in the "Sent" tab and you will be able to view the complete campaign report.
In the "A/B testing" tab, the header of the "Optimized" column will now display the winner version.
The campaign stats found in the "Optimized" column only relate to the optimized version, which means the e-mails sent after the test period, without including the related test sample.
At the bottom of the page, the "Winner" tab specifies the winner version and reminds the criteria used to choose it.
Once the winner version is sent, your A/B campaign will appear in your last campaigns on your Dashboard, among the other mass campaigns.