Test Which Reward Sells More with a Scratch-Off A/B Test

Free shipping or 10% off: which reward turns more subscribers into buyers? This guide answers it the way an experiment does. Hide each reward under the same scratch-off card, send both versions at once, and let purchases decide.

Retro Revival is a fictional clothing brand, and every figure here is made up for illustration. Its test in numbers: 50,874 subscribers · 2 reward variants · 7-day observation window.

Meet Retro Revival

Retro Revival designs vintage-inspired clothing from recycled and organic materials, made with independent craftspeople. Its new collection, Grunge Revival: The 90s Edit, revisits 1990s alternative fashion.

To launch it, the marketing team wants to offer subscribers a reward. Rather than guess which one works, it sets up a controlled experiment and decides on purchases.

Same email, one difference

Every recipient gets the same email: subject line, collection copy, product carousel, scratch-off card and shop button. Only the reward revealed under the card differs. That way, a difference in purchases can be put down to the reward and not to the design.

The two versions are arms of one experiment. An experiment compares each arm with a control, and only another arm can be named the winner. So the control is the offer you'd send anyway. Here that's 10% off, arm A. Free shipping is arm B.

Both arms share one email. In the studio, pick an arm in the toolbar's Arm menu and change the card's prize for that arm only. Try both arms below: scratch the card, see it revealed, or see the fallback that email apps without the scratch-off show.

The card's shop button sits Inside the revealed prize. Where the scratch works, the button only appears once the card is scratched, so a click comes after the reward was seen. The email has no second button below the card, because that one could be clicked without a reveal. Where the link sits compares the two placements.

Pre-register before you send

On the Hypothesis & metrics step, the team writes its plan down before anything is sent:

• Hypothesis: free shipping will convert more subscribers into buyers than 10% off.
• Primary metric: Purchase conversion, the share of delivered recipients who bought. This one number decides the experiment.
• Secondary metrics: Unique click rate and Revenue per delivered email. They explain the result but don't decide it.
• Guardrail: Unsubscribe rate. A winner can't be significantly worse on it.
• Significance level: 5% (two-sided), with Fixed horizon analysis over a 7-day observation window.

The team leaves out Observed reveal rate and Post-reveal click rate. MailInApp doesn't record reveals from inboxes yet. Post-reveal clicks also depend on who chose to reveal, so that metric isn't a randomized comparison.

Lock pre-registration, on the Review & lock step, freezes the hypothesis, metrics, audience and arms, and draws the sample. Nobody can switch to a better-looking metric once results come in.

Reach the right subscribers

The 90s Edit is for subscribers who like rock and the 1990s. On the Audience & sample step, the team picks its 50,874-contact list and adds two filter rules, Rock–Alternative AND 1990s. The Audience counts card shows each stage:

• Eligible comes before targeting. 4,554 contacts can't be emailed at all, and the Left out list says why: unsubscribed, no marketing consent, an invalid address and so on. That leaves 46,320.
• Targeted applies the filter to eligible contacts only. 11,240 like Rock–Alternative and 14,870 like the 1990s, but 6,420 like both.
• In the experiment is the sample: 4,000, split 2,000 per arm.

Before sampling, the team checks Audience insights by the customer_type field. Of the 6,420 targeted contacts, 2,311 are customers and 3,981 are prospects. 128 have no value: the No value bucket, small enough to leave as is.

The audience depends on the list behind it. Set up a subscriber database you can test on covers the fields and consent, and Combine audience filters with AND and OR the filter itself.

Sample and assign fairly

The team picks Balanced random sample on customer_type. The sample is drawn separately within customers, prospects and contacts with no value, so both arms get the same mix: 720 customers, 1,240 prospects and 40 with no value each. A plain random sample is fair on average, but any one draw can drift.

Each contact is in one arm only, and nobody gets both rewards. Leave out contacts who are already in another experiment keeps an earlier test's offer from leaking into this one.

Is 2,000 per arm enough? The Sample size planner says no, for the rise the team hoped for. Lifting purchases from 3.7% to 5.0%, a 1.3-point rise, needs 3,943 contacts per arm.

With 4,000 in all, the test can reliably detect a rise of about 1.9 points. The team goes ahead anyway, knowing a smaller real difference could go unseen. Find out how many contacts your A/B test needs walks through the planner.

Rewards and launch

On the Reward & attribution step, arm A gets Percentage off at 10% and arm B gets Free shipping. Both arms' codes are valid for 7 days, so neither offer has longer to work. Each recipient gets their own single-use code, tied to them. The codes are minted at send time on the Shopify or WooCommerce store connected to the experiment's contacts list.

The scratch-off's prize link is the {{discount_url}} merge tag, which takes each recipient to the store with their own code. The previews on this page link to the shop directly, since a preview can't mint a code.

For free shipping, the team enters an Average shipping cost per order of $5.50. It's counted as promotional cost on each free-shipping order. On WooCommerce, free-shipping codes also need a Free Shipping method that accepts a coupon, and the step warns when there's none.

The same step sets Order attribution: the rules that match a store order to an arm, fixed at the lock. The team keeps the default three, Discount code, Tracked link and In-email checkout. It leaves Email address (probable) off, so every order it counts is a firm match.

Launch builds each arm's email the way it will be sent. It refuses to launch if one arm's email shows the other arm's reward anywhere, fallback included. Then both arms go out at the same time, interleaved over the same minutes. 40 emails per arm bounce, so 1,960 per arm are delivered. Every rate below is a share of those delivered emails.

What happens after the scratch

Scratching the card happens inside the recipient's email app. MailInApp doesn't record reveals from inboxes yet, so this guide can't show how many people scratched, or who scratched and then didn't click. What reaches MailInApp is a click on the prize link.

Because the button sits inside the revealed prize, a click in an email app where the scratch works comes after a reveal. In apps that show the fallback, the prize and its link are visible from the start.

Unique clickers came out at 294 for free shipping against 265 for 10% off: 15.0% against 13.5% of delivered emails. That's a 1.5-point gap, but p = 0.185. The click difference is directional, not significant: it could be chance.

Clickers and purchasers are separate measures, each over the 1,960 delivered emails. They aren't steps of a funnel. Someone can buy without a tracked click, since the discount code also ties an order to its arm.

Purchases decide

When the window ends, the results page reads the primary metric. 98 free-shipping recipients bought, against 72 with 10% off: 5.0% against 3.7%.

The results sentence says free shipping did 1.33 points better than 10% off. The true improvement is likely between 0.05 and 2.62 points, and p = 0.041. That's below the 5% significance level, so the difference is significant. In relative terms, free shipping sold to 36.1% more recipients.

The interval is wide, and its low end is close to zero. Free shipping very likely sells more here, but by how much is uncertain. Remember the planner: this test could only reliably see about 1.9 points, so a 1.33-point result was a fortunate one.

The Unsubscribe rate guardrail held: 0.20% for free shipping against 0.26% for 10% off, so it doesn't block the winner.

Revenue and promotions shows the money per arm, in US dollars:

Revenue excludes taxes and shipping, and discounts are already reflected. This is not a profit calculation.

Free-shipping orders were smaller, but more people bought. Revenue per delivered email was higher with free shipping, but p = 0.13: not significant on its own. It's a secondary metric, so it explains the result without deciding it.

Promotional cost is what each reward gave away. For 10% off, that's the discount on each order. For free shipping, it's the $5.50 shipping estimate for each free-shipping order. Neither column is profit: the cost of the products sold isn't in either one.

Each order reaches its arm through the attribution rules, checked in a fixed order. The first one that matches decides, and the rest aren't checked. Only orders placed during the 7-day window count.

Every Retro Revival buyer used their code, which is why code redemption matches the purchasers: 72 and 98 codes of 2,000 per arm. So the first rule decided every order. The Orders card on the results page shows how many orders each rule matched. Revenue metrics covers refunds and other currencies.

Record the decision and close

The experiment ends with a decision the team records itself. Record decision opens a dialog: Winner, arm B, free shipping. The rationale notes the significant purchase difference, the not-significant clicks and revenue, and the wide interval. The evidence stored with it comes from the analysis: significant.

Close experiment then makes the record final. Export downloads the report as a PDF, CSV tables or JSON. It holds the pre-registration, the numbers per arm, the checks and the decision, with no names or addresses.

The experiment ends here. Recording free shipping as the winner answers the question. It doesn't send anything. Offering free shipping to the rest of the audience is a separate action with its own audience and timing: Send the winning version to the rest of your audience.

Three takeaways

1. Change one thing. Both arms got the same email, so the gap in purchases comes from the reward. One email with an override per arm keeps it that way.
2. Clicks point, purchases decide. Free shipping drew more clicks, but not significantly. Only the primary metric, fixed before the send, picked the winner.
3. A decision isn't a send. The team recorded free shipping as the winner, with the evidence. Sending it to more people, with its shipping cost in mind, comes after.

What if it had been inconclusive?

Say 85 free-shipping recipients had bought instead of 98. That's 4.3% against 3.7%, with p = 0.29. The results page would read Inconclusive: not enough evidence to choose.

That's a normal, useful result. It says any difference is smaller than this test could see. The team would record Inconclusive with its rationale, and keep 10% off as its usual offer. To learn more, it could plan a bigger sample or a longer window with the planner, and run a new experiment. It wouldn't call free shipping the winner because its rate was higher.

Getting started

Open the email in the studio and add a scratch-off card with its Prize link set to {{discount_url}}. Then start an experiment with Run as experiment, give each arm its reward, and lock the plan before you send. Experiments (A/B tests) covers every step, and Reading experiment results how to read what comes back.