Experiments (A/B tests)

An experiment tests one change to an email against a control. You write down the question, the one number that answers it and who takes part, and MailInApp fixes all of that before anything is sent. Then it sends every version at the same time, measures what happens, and tells you how strong the evidence is.

The figures on this page use Retro Revival, a fictional clothing shop, with made-up numbers. Retro Revival tests whether free shipping or 10% off sells more, with the reward hidden under a scratch-off card.

What an experiment is

An experiment belongs to one email and answers one question. It has:

• A hypothesis: what you expect to happen and why, written before you see any results.
• One primary metric: the number that decides the outcome, such as purchase conversion. Everything else is shown to explain it, never to decide it.
• Two or more arms: versions of the email. One of them is the control, and every other arm is compared with it.
• A random sample of your contacts, drawn once and split between the arms. Nobody gets two arms.
• An observation window: how long clicks and orders count after the send. The result is read when it ends.

You start one from Experiments in the sidebar with New experiment, or with Run as experiment from the A/B box on an email's Send page, its row menu on the Emails list, or the campaign calendar.

When a quick A/B send is enough

A plain A/B send splits one send between up to five variants and shows their rates side by side. It's quick, and it's fine for a rough check of two subject lines. But nothing is fixed in advance: not the metric, not the sample size, not when to stop.

Use an experiment when the answer matters. It adds what a quick send leaves out:

• a sample and a metric settled before the send, so nobody picks the reading that looks best afterwards;
• a statistical test, with intervals and a plain-language verdict;
• checks that the send itself was fair, such as every arm going out at the same time;
• store rewards per arm, and orders matched back to the arm that earned them;
• a recorded decision, kept with the experiment for good.

An experiment also isn't limited to 200 recipients like a direct send: it sends through its own job, in batches, to its whole sample.

Plans

Experiments are included from the Lite plan up. What each plan allows:

A holdout counts as one of the arms.

The six steps

An experiment is set up in six steps. You can go back to any of them until you lock it.

Hypothesis & metrics

Write the hypothesis first. Then pick the primary metric, the one number that decides the experiment. Each metric is a share or an amount over its own base:

Delivered means sent minus hard bounces. A contact counts once, however many times they click or buy.

Secondary metrics are shown next to the primary one to explain it. They don't decide the outcome. Guardrails are metrics a winner mustn't make worse, such as the unsubscribe rate or the complaint rate. An arm that beats the control but is clearly worse on a guardrail isn't a winner.

Two metrics have limits of their own:

• Open rate is for context only. Apple Mail Privacy Protection opens emails automatically, so it can't be the primary metric or a guardrail.
• Observed reveal rate and Post-reveal click rate need reveals recorded from email apps, and MailInApp doesn't record those yet. Leave them out for now. Post-reveal clicks also depend on who revealed, so that metric isn't a randomized comparison.

The significance level is how strong the evidence must be before a difference counts. 5% (two-sided) is the usual choice. Analysis is Fixed horizon (read once, when the window ends) or, on the Business plan and up, Sequential (checked as the window runs, with stricter thresholds). Why the window is fixed explains the difference.

Under Statistics, a few more choices are settled now, so they can't be picked later to suit the result. They cover adjusting for what was known before the send, the currency for revenue, a cap on very large orders, and optional margins. Reading experiment results explains each one.

Arms & content

Each arm is one version of the email. Give each a label and a share of the sample, and mark one as the control. Split evenly sets equal shares.

An arm can change the subject line, the sender, and the content. For content there are two ways:

• This email, with this arm's changes. Open the email in the studio, pick the arm in the toolbar's Arm menu, and edit the blocks it should change. Everything else stays shared by every arm. This keeps the arms identical except for what you're testing.
• A different email. The arm sends another email entirely. Its results still land on this experiment.

A few settings can't differ between arms, because they decide how answers or payments are checked: a quiz's correct answers, for example, or a product's price. The studio shows which fields an arm can change.

A test doesn't have to touch the content at all. Here the two arms share one email and differ only in their subject line:

On the Pro plan and up, one arm can be a holdout: contacts who receive nothing. Comparing the others with it shows how many people would have bought without the email, so it needs a purchase or revenue primary metric.

Audience & sample

Choose a contacts list, then who to target with filter rules. Start from a saved segment copies a segment's filter into the experiment. Editing it there doesn't change the segment.

The Audience counts card counts four stages with the same rules the send applies:

1. In the list: every contact on the list.
2. Eligible: contacts who can be emailed at all. The Left out list says why the others can't: unsubscribed or bounced, a verified invalid address, no marketing consent, an unconfirmed subscription, opted out of this email's topic, or already in another experiment.
3. Targeted: eligible contacts who match your filter.
4. In the experiment: the sample, split between the arms.

Next to the filter, Audience insights breaks the targeted contacts down by a list field, by purchase history or by engagement score. Every breakdown has a No value bucket for contacts missing that field. Check it before you sample: if many contacts have no value, balancing on that field won't do much.

Then pick how to sample:

• Everyone targeted: every targeted contact takes part, split at random between the arms.
• Random sample: a random sample of the targeted contacts, split at random.
• Balanced random sample (Pro and up): the sample is drawn separately within each value of a field you pick, so every arm gets the same mix, for example of customers and prospects.
• The whole segment: every contact the rules select. This is rule-based selection, not a random sample, so the results describe that segment only. Arms are still assigned at random.

Leave out contacts who are already in another experiment keeps experiments from overlapping. It covers experiments that are locked, still in their observation window, or whose offer can still be redeemed, so an earlier test's offer can't leak into this one.

The Sample size planner works out how many contacts the test needs before anything is sent. It starts from your own history: past experiments on the list, or your recent sends. You enter the smallest rise worth detecting and the power you want, and it recommends a sample. If the audience is too small, it says the smallest rise it can reliably detect instead.

Retro Revival hoped purchases would rise from 3.7% to 5.0%. The planner asks for 3,943 contacts per arm to detect that 1.3-point rise. With 4,000 contacts in all, the test can reliably detect a rise of about 1.9 points. A smaller real difference could easily go unseen, so plan for the size you can afford before you launch, not after. Find out how many contacts your A/B test needs walks through the planner step by step.

Reward & attribution

Give each arm its own store offer, or none: a percentage off, a fixed amount off, or free shipping. Every arm gets a reward, or none does, and every arm's codes stay valid for the same number of days, so the arms are compared fairly.

Each recipient gets their own single-use code, tied to them. Show it in the email with the {{discount_code}} and {{discount_url}} merge tags, for example in a scratch-off card's prize and link. The lock waits until a block shows the reward.

A few things to set up first:

• The codes are minted on the Shopify or WooCommerce store connected to the experiment's contacts list. Without one, the step warns that no codes can be minted.
• On WooCommerce, free-shipping codes need a Free Shipping method that requires a valid free shipping coupon. The step warns when your store has none.
• For free shipping, enter your average shipping cost per order. It's counted as promotional cost for each free-shipping order.

Order attribution matches each store order to an arm. The first rule that matches wins, in this order:

1. Discount code: the order used the arm's code. The strongest match, even days later.
2. Tracked link: the buyer arrived through the email's tagged link.
3. In-email checkout: the buyer paid through a product block's checkout.
4. Email address (probable): the buyer's address matches a recipient who clicked during the window. These orders are reported as probable.

The first three rules are on by default. Email address (probable) is off until you pick it, and it's the only rule that can match a holdout's orders, since the holdout gets no email. Different rewards per arm covers the codes and the rules in detail, and Test which reward sells more with a scratch-off A/B test runs a full reward test.

If the email has a scratch-off card, Scratch-off call to action sets where its shop button sits: Inside the revealed prize or Below the card. Measuring a scratch-off explains what each placement lets you measure.

Schedule & window

Choose As soon as I launch or At a set time. Every arm goes out together, interleaved over the same minutes. Send-time optimization is off for experiments, so a difference between arms can't come from when each was sent.

The observation window is how long clicks and orders count after sending, from 1 to 60 days. The result is read once, when it ends. Pick a length that covers how long people usually take to act, and that the offer stays valid for.

Send the result to the rest tests on a sample first, then sends one arm to everyone else the audience targets. It needs a random sample (step 3), so there's a rest to send to. Choose a policy:

• Send a significant winner: an arm that wins on the primary metric by the decision time goes to the rest. Otherwise the rest gets the control.
• Send the leader (profit policy): the rest gets whichever arm looks best by the decision time, significant or not. That pays off when arms are close, but it isn't evidence that one is better.

Decide after counts from when the sample's last email goes out. If an arm has fewer than 30 events by then, the sample is read once more at Wait at most. If it's still short, the rest gets the control. This only chooses what the rest receives: the experiment's own decision still waits for the whole window.

Review & lock

The last step sums up the plan and lists anything to fix before locking. Lock pre-registration then freezes the hypothesis, metrics, audience and arms, and draws the sample: which contacts take part, and which arm each one gets, are final.

Until you launch, you can still rename the experiment, change the send time, and edit each arm's label, subject line, sender and content. Launch reserves the whole send from your monthly emails, then schedules it. The holdout takes nothing from your quota. After launch, only the name can change.

At launch, MailInApp also builds each arm's email the way it will be sent. It refuses to launch if one arm's email shows another arm's scratch-off reward, even in the fallback that email apps without the scratch-off show. Testing two rewards in an experiment covers what the check catches. The preflight checklist checks the email as you saved it, so preview each arm with the studio's Arm menu too.

While it runs

After launch, the experiment moves through its statuses on its own:

• Scheduled until the send time, then Sending while every arm goes out.
• Observing for the observation window. An interim analysis runs every few hours. Its results are marked interim: don't decide on them.
• Analyzing once the window ends. The final analysis comes in about an hour later.
• Decided or Inconclusive once you record your decision, then Closed when you close it.

While it runs, only the name can change. Cancel experiment stops it at any point before the decision, and can't be undone. Other sends of the same email leave out the contacts this experiment holds, so nobody gets a second version through another route.

You can delete an experiment while it's a draft, once it's closed, or after a cancel. Its results and assignments are deleted with it.

Limits and what isn't measured

• Reveals aren't recorded from inboxes yet. A scratch-off or spin-to-win reveal happens inside the email app, and MailInApp doesn't record it there yet. Measure clicks on the prize link and purchases instead.
• Opens are context only. Apple Mail Privacy Protection and image proxies open emails automatically, so open rates run high and vary by email app.
• Each arm is checked at launch. The Send page's preflight checks the email as saved, not each arm's version.
• Not in the mobile app yet. The MailInApp mobile app doesn't show experiments.

Use it from the API

Your own code and automation tools can read experiments with an API key, and webhooks can tell them when one is decided or its results are ready. They can read, but not lock, launch or decide. See Read experiment results.

Related

• Reading experiment results
• Test which reward sells more with a scratch-off A/B test
• Find out how many contacts your A/B test needs
• Send the winning version to the rest of your audience
• Contacts & sending from MailInApp
• Scratch-off block
• Behavioral segments