All Collections
Campaigns
Advanced Campaigns
How to carry out an A/B test?
How to carry out an A/B test?

Test different elements of your email, measure their performance, create your future email campaigns based on what's working and what’s not.

Weronika Wróblewska avatar
Written by Weronika Wróblewska
Updated over a week ago

A/B Tests are a part of an A/B testing add-on, click here to learn how to get it on Marketplace »


IN THIS ARTICLE

How it works

What is A/B testing for?

Carry out the A/B test by adding two or more versions of your email.

  • The original version is marked as “Version A”.

  • To create a new email version you can either copy the existing version or add a new blank version by clicking the green + symbol.

Create and test a maximum of 5 email versions per one step in your campaign.

  • That means that besides the original “Version A” you can add versions B, C, D, and E to your opening email (Email 1) and another 5 versions to each of the follow-up messages (Email 2, Email 3, etc.)

  • Please note that you can have a maximum of 12 emails in one Path (1 opening email and 11 follow-ups) which gives you the possibility of testing 60 versions of emails in one campaign.

  • For Agency and Sales Assistant users, the maximum in one Path is 16 steps.

Test one thing at the time.

  • If you want to increase your open rate, focus on multiple variations of your subject line.

  • If your goal is to engage your prospects and receive more replies, experiment with the content of your email, for example, by including different Call To Action (CTA) in each email version.

Send one email version to at least 100 prospects.

  • If you’re planning to test 4 versions of email, it’s best to import at least 400 prospects who share similar characteristics.


Why should you A/B test your email campaign?

  • Improve email deliverability by experimenting with varied content.

  • Learn which of your cold email practices bring better results.

  • Study your prospects’ behavior to craft personalized messages that fit your target group.

  • Turn your cold prospects into valuable leads by having engaging conversations.

  • Create lasting business relationships.


What can you A/B test?

  • Subject line to measure your open rates.

  • Email content to increase reply rates.

Please note

Delivery time, timezone, and signature stay intact for the whole step. These settings cannot be changed separately for each email version.


How to carry out an A/B test?

Before you start

  • Take care of your domain and email warm-up.

  • Prepare an up-to-date list of prospects.

  • Divide your target group by characteristics.

  • Plan your test.

  • Prepare various email versions.

Step #1 Create different email versions

Each campaign starts with version A of an opening email - this is your original message. Add one or more email versions to start testing your campaign.

Adding versions

  • Click the green "+" symbol to add an empty version of your email.

  • Click the ‘Options’ icon and select ‘Copy’ to duplicate your email version.

Gif with copying the email version

Deleting versions

  1. Hover onto the version that you want to remove.

  2. Click the 'Options’ icon.

  3. Select ‘Delete’ from the dropdown menu.

  4. Click 'Yes, I'm sure' to confirm.

Gif with deleting the email version

Please note

  • You need to have at least one email version in your campaign.

  • You can’t delete any version as one of them has already been sent.


Step #2 Add follow-ups

Add one or more follow-ups to increase the chance of opening your email and test them too.


Sending follow-ups

Sending follow-ups in the same thread

Tooltip about sending a message in the same thread

Leave the subject empty if you want to send your emails in the same thread. Your follow-ups will be sent with the previous subject line, attached to the other emails in the sequence. It will start with "Re:" in the subject line. For example, "Re: Hi!"

Sending follow-ups as a separate message

Entering the subject line to make separate email

Type in a new subject to send follow-ups separately. Each follow-up will be sent as a separate message with the new subject line and without the body of the previous message attached as a citation.

However, if you wish to attach the previous message, just click the quotation mark icon. Your follow-up will be sent with the new subject and the content of the previous message will be included in the thread.


The process is fairly the same as when you split test the opening email. Add a blank new version by clicking ‘+’ (‘Add version’) or copy the existing version. One version of a follow-up will be randomly assigned to a prospect in each step.

How does it work? Let’s say you have 1 opening email and 2 follow-ups in the campaign sequence. That gives 3 steps:

  • Email 1 (opening email) without A/B testing.

  • Email 2 (follow-up) with A/B testing.

  • Email 3 (follow-up) also with A/B testing.

New versions will be drawn for a prospect right before sending a message. That means that your prospect can receive, for instance, version B of Email 2 and version A of Email 3.


Step #3 Preview the campaign

Check what each version of your email and follow-ups will look like in your prospect’s inbox. Scroll down to the bottom of the page and click the PREVIEW button.

  • In the preview, you’ll see all versions of your email for each step in the campaign.

  • Only one version will be sent to each prospect.

  • The versions can be different for each step. For example, Prospect #1 can receive version A of the opening email and version D of the follow-up message. Prospect #2 will be sent version C of Email 1 and version A of Email 2, and so on.

Gif with going through all steps of campaign in Preview

Customize the opening email

After email customization, we add a symbol of the version that you edited, for example, ‘A’ and tag ‘CUSTOMIZED’. See the example below.

Information about "customized email" next to the Opening email

Step #4 Send test campaign

Test the sending and see what your email will look like in your prospect’s inbox.

  • Click ‘SEND TEST CAMPAIGN’ to test and send several email versions to your email address randomly.

  • To avoid cluttering your inbox we won’t be sending all of your versions at once.

"Send test campaign" button

If everything looks good, click ‘SEND’ to start your A/B test.


Step #5 Compare the results

Your campaign will start running as soon as you hit the ‘SEND’ button. The emails will be sent according to the Delivery time and timezone that you set up for this campaign.

To check your results, open your stats dashboard.

Statistics

  • Steps with different versions will be marked with the ‘A/B’ tag.

  • Your best results will be marked by a special winner icon.

  • You can turn off the email versions of lower performance.

Gif with reviewing emails in A/B testing

‘Best result’

  • Your best results will be marked with a little badge - a special winner icon.

  • ‘Best result’ badge will be assigned to the best open rate (‘OPENED’), click rate (‘CLICKED’), and reply rate (‘RESPONDED’).

  • ‘Best result’ badge for Interest level is available only for the prospects marked as 'Interested'.

A/B testing results

Turning off/on an email version

  • You can turn off an email version with worse results so it won’t be sent to prospects anymore.

  • Open your stats dashboard and click the toggle switch to turn off/on an email version.

Switching off one version of email
  • Inactive are marked in grey color.

  • Active versions are marked in green.

  • Keep in mind that at least one version of the sending email has to be active.

Information "You need to have at least one version" in A/B testing
  • If you decide to turn off email versions where you customized an email, those customized emails will still be sent.

Confirmation screen when turning off one version of email

If-campaigns

  • Switch between Path YES and Path NO to compare your email performance in campaigns with conditions.

Last Activity

  • Open the PROSPECTS tab in your campaign and check your prospect’s activity.

  • See when and which version of your email was SENT, OPENED, CLICKED, or RESPONDED to.

Last activity of a prospect in campaign

Exporting results

  • Go to your campaign stats dashboard → click ‘Actions’ → select ‘Export as .csv’.

  • We add column ‘VERSION’ regardless of A/B testing. If you didn’t use A/B testing in your campaign, the column will be left empty.

Simple campaigns

Campaigns without conditions. Column PATH was intentionally left empty.

Exported data from campaign with A/B Testing

If-campaigns

Column PATH filled in with the data Path YES or Path NO.

Exported data with Path info


Learn more on our Blog


Did this answer your question?