Most A/B tests on email lists are too small to see the difference they're looking for. The sample size planner in an experiment's Audience step tells you, before anything is sent, how many contacts the test needs, or the smallest difference your audience can reliably show.
A test that's too small usually ends "inconclusive", even when one version really is better. That isn't a failure of the test. It's a sign the question was asked of too few people. Planning first tells you whether to run the test as it is, test a bolder change, or wait for a bigger audience.
The worked example below uses Retro Revival, a fictional clothing shop, with made-up numbers. Its whole reward test is a use case of its own.
What the planner needs from you
Open an experiment and go to Audience & sample. The Sample size planner sits under the audience counts. It plans for the primary metric you picked in step 1, with four inputs:
- The control's expected rate. For example, the share of delivered recipients who buy. The planner fills it in from your own history.
- Smallest rise worth detecting. The smallest difference over the control that would change what you do. It's written in points for a rate: 1 point takes 3.7% to 4.7%.
- Power. The chance the test finds a rise of that size when it's real. 80% is the usual choice; 85%, 90% and 95% need more contacts.
- Expected delivery rate. Bounced emails can't buy, so the planner adds contacts to cover them.
The significance level comes from step 1, and the split between arms from the Arms & content step.
Start from your own history
You rarely know your baseline off by heart, and a guess is often far off. The planner reads it from your account, and says where each number came from:
- Past experiments on this list, or else on your other lists. These cover every metric, purchases and revenue included.
- Your last sends to this list, for a click or response primary metric when no experiment has measured it yet.
- Your sends in the last 90 days, for the delivery rate and for unsubscribe and complaint guardrails.
With no history for a metric, it asks you to enter the rate you expect. You can also overwrite any number. It's then marked Your estimate, for planning only, and Use history puts the measured one back. A changed baseline or delivery rate is only used for planning. The rise to detect and the power are saved with the experiment.
Read the recommendation
Once you enter the smallest rise worth detecting, the planner shows the Recommended sample: the contacts the whole test needs, and how many in each arm. If your targeted audience is big enough, it says so. One button then sets the sample to that size: Use that many contacts, or Draw a random sample of them if you weren't sampling yet.
If the audience is too small, it says so plainly: This audience is too small to detect a rise of less than a given number of points. That figure is the honest size of your test. A real difference smaller than it could easily go unseen. Plan for that figure updates the plan to it.
Contacts per arm
| Smallest detectable difference | Contacts per arm |
|---|---|
| 1.0 pts | 6,444 |
| 1.5 pts | 3,027 |
| 2.0 pts | 1,793 |
| 2.5 pts | 1,206 |
| 3.0 pts | 877 |
The curve shows the trade-off. Halving the difference you want to detect multiplies the contacts each arm needs by almost four. Retro Revival's purchase rate was 3.7%, at 98% delivery and 80% power:
| Smallest rise to detect | Contacts per arm |
|---|---|
| 1 point | 6,444 |
| 1.3 points | 3,943 |
| 1.5 points | 3,027 |
| 2 points | 1,793 |
| 3 points | 877 |
Retro Revival hoped free shipping would lift purchases by 1.3 points, which needs 3,943 contacts per arm. It had 4,000 in all, so 2,000 per arm. With that sample, the planner reports a smallest reliably detectable rise of about 1.9 points. The team ran the test anyway, knowing a smaller real difference could go unseen.
When the test needs more contacts
A few choices raise the number, and the planner says which one is doing it:
- More arms. Each arm is compared with the control at a stricter level, so all comparisons together keep your significance level. Three arms need more contacts per arm than two.
- An uneven split. A small holdout or a large control makes each comparison cost more than an even split.
- Guardrail margins. A guardrail with a margin can need more contacts than the primary metric. The planner then sizes the test by the guardrail.
- Sequential analysis (Business plan and up). Looking at the result as the window runs needs more contacts than a single read at the end. The planner says how many more.
A smaller test slice: Test & Roll
Sometimes you don't need proof, only the better choice for the rest of your list. Send the result to the rest with the policy Send the leader (profit policy) sends whichever arm looks best after a short test to everyone else.
For that policy, the planner can suggest a Profit-optimal test slice (Test & Roll). It weighs how much results varied across your past experiments against what the rest of the list stands to gain. It needs at least two past experiments, and two arms with no holdout. The slice is usually much smaller than the recommended sample. It's a profit choice: a slice that size is usually too small to find a significant winner.
The plan is frozen at the lock
The rise to detect and the power are saved with the experiment and frozen when you lock it, with the rest of the pre-registration. The results page later reports the smallest difference the finished test could reliably have detected. If the test ends inconclusive, that number tells you whether a real difference could have gone unseen.
What about a quick subject-line test?
A plain A/B send from the Send page has no planner, and nothing is fixed in advance. For a subject line that matters, such as a renewal reminder, run it as an experiment and plan it on click rate rather than opens. Opens are inflated by Apple Mail Privacy Protection, so they make a poor primary metric.
Arm A (control)
Subject
The October restock is here
Arm B
Subject
Back in stock: the 3 cameras you asked for
Clicks are much more common than purchases, so a subject-line test usually needs far fewer contacts than a reward test. Your own click history from recent sends to the list fills in the baseline.
Getting started
Create an experiment with Run as experiment from an email's Send page. Pick the primary metric in Hypothesis & metrics, then open the Sample size planner in Audience & sample. Check where the baseline came from, enter the smallest rise worth acting on, and size the sample before you lock it. Experiments (A/B tests) walks through every step.