A/B Testing: FAQ

Why should you test your campaign, how to quickly check if the campaign was A/B tested, how to turn off the sending email version.

Weronika Wróblewska avatar
Written by Weronika Wróblewska
Updated over a week ago

Why should I use A/B testing? How am I going to benefit from this test?

A/B testing shows you which variation of your email proves to be the most fruitful and helpful. While experimenting with the subject line or email copy, you learn which of your practices brings better results. In turn, you discover how to tweak your campaigns to turn your cold contacts into business partners. Also, your chances to avoid being caught by anti-spam filters are higher when using multiple versions of your email. The more variations of email and the more randomized sending, the better for your email deliverability.

What can I test?

Anything that you have in the content of your email. You can experiment with multiple variations of the subject line and the email body. Delivery time that you set up for each step (each email in the campaign sequence) and your signature will stay the same.

Is there a way to quickly check if I used A/B testing in my campaign?

Yes, look for an ‘A/B’ tag on the list of your campaigns. Split test campaigns are marked with the ‘A/B’ tag.

How many versions of my email can I test?

You can add and test up to 5 versions: A, B, C, D, and E. If you’re not happy with the version you created, you can delete it and then add a new one.

5 versions of emails: from A to E

How can I make the subject line feel more personal?

To personalize your subject line you can use one of the snippets available in the email editor, for example, {{FIRST NAME}} or {{COMPANY}}. There are plenty of options.

Subject line of email

Before you hit the ‘SEND’ button, though, make sure you’re not missing any important information. Preview your campaign and send a test campaign to yourself to see what your emails will look like in your prospect’s inbox. Read more about snippets »

Can I also A/B test follow-up emails?

Yes, you can carry out the A/B test for all steps in the campaign. The process is fairly the same as when you split test the opening email. Add a blank new version by clicking the grey + symbol (‘Add version’) or copy the existing version. One version of a follow-up will be randomly assigned to a prospect in each step.

How does it work? Let’s say you have 1 opening email and 2 follow-ups in the campaign sequence. That gives 3 steps:

  • Email 1 (opening email) without A/B testing.

  • Email 2 (follow-up) with A/B testing.

  • Email 3 (follow-up) also with A/B testing.

A new version will be drawn for a prospect right before sending it. That means that your prospect can receive, for instance, version B of Email 2 and version A of Email 3.

Can I perform A/B testing on a campaign with conditions?

Yes, there are no restrictions. Please mind that although we don’t block performing A/B tests in If-campaigns, the statistics may not be accurate due to a small number of prospects in each Path.

Is there a minimum number of prospects needed for the A/B test?

No, there isn’t. Woodpecker won’t block the sending even if you have only 2 or 3 prospects in the campaign. Nevertheless, we recommend sending one email version to at least 100 prospects. So if you’re planning to test 4 versions of email, it’s best to import at least 400 prospects.

How are the prospects assigned to one version?

Prospects are selected in random order from the pool of email addresses added to the campaign. First, the prospect will be drawn for version A of Email 1, second, for version B, thirdly, for version C, and so on. Next, in the same process, we will randomly select prospects for the next step in a campaign.

How many prospects are assigned to one version? Do you assign prospects evenly?

Prospects are selected randomly yet distributed evenly. This means that each version should be sent to roughly the same number of prospects.

What about the statistics? Will they show that I used A/B testing in the campaign?

Yes, open your campaign and have a look at your stats dashboard. There you’ll see all versions you have tested. Your best results will be marked with a little badge – a special winner icon. For example, your open rate may be the best in version A (check the column ‘OPENED’) while the highest response rate may bring version C (‘RESPONDED).

‘Best result’ badge for Interest level is available only for the prospects marked as 'Interested'.

If you’re not happy with the performance of some of the email versions, you can turn them off so they won’t be sent to prospects anymore.

Can I turn off the sending email version?

Yes, you can turn off an email version with worse results so it won’t be sent to prospects anymore. Please note that at least one version of the sending email has to be active.

Open your stats dashboard and click the toggle switch to turn them off/on.

Turning off one version of email
  • Versions that you turned off are marked in grey color.

  • Active versions are marked in green color.

  • If you decide to turn off sending versions with customized emails, those emails will still be sent.

Can I delete a version?

Deleting versions is possible as long as no emails have been scheduled to be sent. What does it mean and how does it work? You can see scheduled emails as QUEUED emails in the statistics.

Please note that the option to delete a version will be disabled if at least one version has been sent from a particular step. You’ll see a message “You can’t delete any version as one of them has already been sent”.

Can I customize an email while doing A/B testing?

When previewing your campaign you’ll see the option to customize Email 1. If you edit one of the versions, that version will be assigned to a prospect. Other versions of Email 1 won’t be shown in the preview. See the example below.

Can I edit my email versions after the sending has started?

Yes, for example, when you notice that the performance and conversion rate of one of the sending email versions is low.

Can I add a new version when my campaign is running?

Yes, you can add another version at any time.

It seems that I cannot add any version. Why?

A/B testing is possible only with the purchased A/B testing add-on. Click here to learn how to get it »

What happens to the versions when I copy the whole campaign?

Making a copy Woodpecker will also replicate all the added versions except the ones which you customized in preview. Learn more about customizing email in the preview »

How do conditions work when I carry out the A/B in some versions?

In case you added an open-based condition – all of the version needs to have open tracking allowed. We’ll show you a short reminder in case you forget about switching the open tracking on: ‘Open tracking needs to be on because of the condition you’ve set for the next follow-up”.

The same goes for click-based conditions. We’ll notify you that you need to ‘Add a tracking link to all versions of the previous email’. Otherwise, sending will be impossible.

Feel free to go ahead and check our blog post covering A/B Testing in Email: an FAQ to Clear All Your Doubts »

Did this answer your question?