Campaign Subject Line A/B Test

Where to find: Marketing → Email Campaigns → the A/B test tab in the campaign create/edit screen Who it's for: Anyone who wants to answer "which headline gets opened more?" by measuring rather than guessing In short: You write two subject lines; the campaign first goes to a small test audience with both subjects, the one with the higher open rate wins, and the remaining audience is sent automatically with the winning subject.

What is it for?

The single biggest influence on an email campaign's open rate is the subject line. Does "20% off starts now" get opened more, or "A special price for the items in your basket"? Instead of guessing, you try it.

The flow works like this:

A small share of the audience  →  half gets Subject A, half gets Subject B
        ↓  (wait time)
Open rates are compared → the winning subject is picked
        ↓
The ENTIRE remaining audience is sent with the winning subject

How to set it up

1. Marketing → Email Campaigns → New Campaign (or edit an existing draft).
2. Where the subject field is, there are two tabs: Single subject and A/B test. Switch to A/B test.
3. Fill in the fields:

4. Below the form, the Send plan box explains the plan with your own numbers: "N recipients per variant; the remaining M get the winning subject after X hours." Under it, the rule: "The winner is chosen by open rate; on a tie or with insufficient data subject A is used."

What happens once the campaign starts sending?

A Subject A/B test card appears on the campaign detail, comparing the two variants side by side:

The winning variant carries a Winner badge.

Phases

Depending on the state the card also shows a summary sentence — for example "Both subjects went out to 10% of the audience. The winner is picked at …" or "Winning subject is … The remaining audience is being sent with it."

In the recipient table

The recipient list has a Variant column: it shows whether each recipient got A or B, and marks the rest as Remaining. While the remaining audience waits, their status shows as Queued (A/B).

Campaigns with an A/B test also carry a small A/B badge in the campaign list.

How many people do you need?

The test splits the audience three ways: group A, group B and the remaining audience. If your audience is very small, the difference between the two groups is just noise — so with very small audiences the system skips the test and sends with Subject A ("No test run").

Quota and cost

An A/B test sends no extra emails: the test audience is part of the audience you were going to send to anyway. The total number of sends stays the same and counts towards your plan's monthly email quota as usual.

Step by step

1. Email Campaigns → New Campaign; prepare the content and audience as usual.
2. In the subject section switch to the A/B test tab.
3. Subject A: "20% off this weekend only" · Subject B: "The items in your basket are on sale".
4. Leave Test audience share at 10% and Wait time at 4 hours.
5. Check the numbers in the Send plan box and start the campaign.
6. A few hours later, see the winner on the Subject A/B test card in the campaign detail.

Frequently asked questions

Can I pick the winner myself? No, the winner is chosen automatically by open rate. On a tie or with insufficient data, Subject A is used.

Is a longer wait time more accurate? Generally yes — people take time to open an email. 4 hours is a reasonable start; for morning sends 6-8 hours is worth trying.

Can the content (body) be tested too? No, today only the subject line is tested. The content is identical in both variants.

Can I change the A/B settings while the campaign is sending? No. Settle the settings before sending starts.

It says "No test run". Your audience was too small for a test; the campaign was sent with Subject A. This is not an error.

What does "Queued (A/B)" mean in the recipient table? That person is in the remaining audience; they will be sent once the winning subject is known.