Reading experiment results

An experiment ends with a decision you record yourself. The results page gives you what you need for it: the difference between the arms, how sure you can be of it, and whether the send itself was fair. This page walks through it panel by panel, then explains what the numbers mean.

The examples use Retro Revival, a fictional clothing shop, with made-up numbers. Its control, arm A, offered 10% off. Arm B offered free shipping. Each arm's email was delivered to 1,960 recipients, and purchase conversion was the primary metric.

Find the results

Each experiment's results live in its email's Insights, under Experiments. From the Experiments list, a row's menu has Results in Insights. The first results appear after the first analysis, which runs once the send has finished. While the window is open, the page says the results are interim: don't decide before it ends.

The header shows the status, the window, the evidence so far and the actions: Explain, Export, Record decision, Close experiment and Deploy winner, each when it applies.

The results page, panel by panel

Pre-registration

What you fixed at the lock: the hypothesis, the metrics and their roles, the significance level, the arms, the sample and the statistics choices. It's there so anyone reading the result can check it answers the question that was asked.

Results

The primary metric card shows each arm minus the control, with its interval, its Holm-adjusted p-value and a verdict: Significant, Not significant, or Significant: the control did better. Under it, one sentence says what the result means in plain words. It ends with the smallest detectable difference: how big a difference this many recipients could reliably detect.

For Retro Revival, the sentence reads that free shipping did 1.33 points better than 10% off (5.0% vs 3.7%). The true improvement is likely between 0.05 and 2.62 points. A gap this large would be unusual if the two performed the same.

Secondary metrics and guardrails sit in the table below. Secondary metrics are exploratory: each is marked Stands out or Doesn't stand out, never "significant". A guardrail reads Worse than the control or Not shown to be worse. Only a guardrail that's clearly worse blocks a winner.

Against the holdout

With a holdout arm, a second card compares each mailed arm with the contacts who received nothing, per assigned contact. It shows what the email added over not sending at all. It's descriptive: the decision still compares the arms with the control.

Interaction stages

Separate bars, one pair per stage: Unique clickers and Purchasers, each a share of its own base. They aren't steps of a funnel. People can buy without clicking a tracked link, since the discount code also ties an order to its arm.

The trend over the window

The primary metric over the window, one point per daily analysis. It's for watching only. A lead that comes and goes along the way isn't evidence: the test is read once, when the window ends.

Data quality

Read this panel before the result. Its Statistical checks say whether the draw, the send and the data held up:

• Assignments: every arm holds the contacts the draw gave it.
• Sample ratio: each arm's emails went out in the proportion it was drawn for.
• Bounces: bounce rates are similar across arms. If they aren't, metrics are counted per sent email instead of per delivered one.
• Balance and Mix: the arms started alike on what was known before the send, such as engagement and past orders.
• Other experiments: when contacts were in another experiment at the same time, whether the two interact on the primary metric.

The Sample ratio check matters most. It fails when chance would produce a gap that large between the drawn and the sent shares less than once in 2,000 tries. That points to a sending problem, so a failed check marks the result invalid.

Below the checks: Send timing (how far apart the arms went out), Sample and overlap, Delivered (assigned, sent, bounced, delivered per arm), Reward codes, Orders by attribution rule, and Revenue and promotions.

Rest of the audience

If the experiment sends an arm to the rest of the audience, this card shows each look at the test sample, which arm the rest got and why, and how many were sent. It only chose what the rest of the audience got. The experiment's own decision is still recorded on the whole window.

Decision

Once you record a decision, it shows here for good, with your rationale and the evidence the analysis gave it.

What the numbers mean

Each comparison has three parts:

• The difference: the arm's rate minus the control's, in percentage points. Free shipping's 5.0% minus 10% off's 3.7% is +1.33 points.
• The interval: the range the true difference likely falls in. Here it's 0.05 to 2.62 points. It's clear of zero, so a real improvement is likely, though it could be small.
• The p-value: how often a gap this large would show up if the arms really performed the same. Here p = 0.041, below the 0.05 significance level, so the difference is significant.

Unique clickers tell a different story. Free shipping had 15.0% against 13.5%, but p = 0.185 and the interval crosses zero. That's directional, not significant: the click gap could be chance.

The results sentence takes one of these forms:

• Winner: an arm beat the control on the primary metric, and no guardrail blocks it.
• Inconclusive: not enough evidence to choose. This is a normal, useful result. It tells you the difference, if any, is smaller than this test could see.
• No difference: every arm is provably close to the control, within the no-difference margin you set before the lock. Without a margin, a flat result is inconclusive.
• Can't be trusted: a data-quality check failed. Check the panel before deciding.
• Too early to call: the window is still open.

Inconclusive results are common on small lists. Retro Revival's test could only reliably detect a difference of about 1.9 points, so its 1.33-point result was a fortunate one. Had free shipping won 85 purchasers instead of 98 (4.3% against 3.7%), p would be 0.29: inconclusive. The right next step then is a bigger sample or a longer window, sized with the sample size planner, not a second look at the same numbers.

Why the window is fixed

Every daily look at a running test is another chance for noise to look like a winner. Stop at the first look that clears the usual threshold, and false winners become far more likely. That's why a Fixed horizon experiment is read once, at the end of the window you set.

Retro Revival's z statistic ended at 2.039, above the fixed threshold of 1.96. Read once, the result is significant. But it first passed 1.96 on day 5, so someone peeking daily would have stopped there, on weaker evidence.

On the Business plan and up, Sequential analysis lets you look as the window runs. Each look has a stricter boundary, so the overall chance of a false winner stays at your significance level. There's no look until about half the expected data is in, nor in the first 48 hours after the send finishes, and at most one look a day. On Retro Revival's numbers, no look crosses its boundary: the last one is 2.056, just above 2.039. If an arm does cross early, you can record the decision before the window ends.

Several arms and several metrics

More comparisons mean more chances for one to look good by luck. MailInApp corrects for this:

• The primary metric uses Holm's correction across the arms compared with the control. Its intervals are shown wider to match: with three arms besides the control, each is shown at 98.3% instead of 95%.
• Secondary metrics use Benjamini–Hochberg. They're exploratory, so they can stand out but never decide the experiment.
• Guardrails are read at a looser level on purpose, since missing a real harm costs more than a false alarm.

Adjusted comparisons

If you kept Adjust for what we knew before the send on in the Statistics choices, the results also show an adjusted comparison. It accounts for each contact's engagement score and purchase history at the draw, which usually narrows the intervals. The deciding comparison is the adjusted one when it ran. The unadjusted one is always shown too.

The choice is made before the lock, like the rest of the plan, so nobody can switch it on after seeing which reading looks better. Adjustment is skipped, for every arm at once, when an arm is too small or there's nothing to adjust for.

Revenue metrics

Revenue per delivered email divides each arm's attributed store revenue by its delivered recipients. Each contact's revenue is capped at a high percentile of buyers, chosen before the lock, so one very large order can't decide the test. Its interval comes from a resampling method when revenue is too lopsided for the usual formula.

Revenue and promotions shows the money per arm. Retro Revival's, in US dollars:

Revenue excludes taxes and shipping, and discounts are already reflected. This is not a profit calculation.

• Promotional cost is the orders' discounts, plus your shipping subsidy estimate for each free-shipping order.
• Code redemption is codes redeemed on an attributed order, divided by codes issued to the arm's recipients.
• Average order value depends on who bought, so it can't decide the experiment. Here free shipping's orders were smaller, but more people bought.
• Refunds made in the window on Shopify and WooCommerce are subtracted from revenue. A Shopify store connected before experiment rewards existed needs reconnecting first. Refunds of in-email checkout payments aren't subtracted.
• Other currencies: orders in a currency other than the experiment's are left out of revenue and counted separately.

Revenue per delivered was higher with free shipping, but p = 0.13: not significant on its own. Purchase conversion, the primary metric, decides.

Orders are matched to arms by the attribution rules you picked, in order: discount code, tracked link, in-email checkout, then email address. The first rule that matches decides, and the rest aren't checked. The Orders card counts each rule's matches, and email-address matches are labelled probable.

Different rewards per arm explains each rule and how the codes are minted.

Recording a decision

When the window has ended and the final analysis is in, Record decision opens a dialog. Pick Winner (and the winning arm), No difference or Inconclusive, and write your rationale. The evidence recorded with it comes from the analysis, not from the form:

• Picking an arm the analysis didn't name stays marked on the record for good: "The analysis didn't name this arm the winner."
• Recording a winner on a result that failed a data-quality check, or before any analysis, needs an override note.
• A recorded decision can't be changed.

Then:

• Close experiment makes the record final. Its configuration, results and decision can't change afterwards. If the experiment was decided early, close it once the final analysis of the whole window is in.
• Deploy winner creates a new email from the winning arm, with its content and offer, its subject and sender, and the experiment's contact list. Everyone the experiment assigned is left out of every send of that email. Nothing is sent until you send it from its Send page. Send the winning version to the rest of your audience follows one through.
• Export downloads the report as a PDF (with your company name and brand colour on the Business plan and up), as CSV tables or as JSON. It holds numbers only: no names or addresses.
• Explain asks Mia to describe the result in plain words, using only the report's numbers and labels. Mia never calls a difference significant unless the analysis did. Each run uses one Mia AI request.

Recording the decision answers the experiment's question. Sending the winner to more people is a separate step, after it.

Related

• Experiments (A/B tests)
• Test which reward sells more with a scratch-off A/B test
• Send the winning version to the rest of your audience
• Webhooks & events
• Responses & webhooks