Before you draw a sample, look at who your filter targets. The Audience insights panel breaks the audience down by any list field, by purchase history or by engagement, with a bucket for missing values. A minute here catches messy values and gaps that would otherwise skew the test.
This is part 2 of a three-part series. Part 1 built the filter. This part reads the audience it targets, and part 3 saves audiences for reuse.
The examples follow Retro Revival, a fictional vintage clothing shop, with made-up numbers.
Where the panel is
Open an experiment's Audience & sample step. Audience insights sits beside the filter. It shows who the filter targets right now, after eligibility, so it describes the same contacts as the Targeted count. Change the filter and the panel follows.
Pick what to look at under Break down by:
- List fields: any column of the contact list, such as
customer_typeorgenre. - From activity: Purchase history, from the purchases MailInApp and your connected store recorded, and Engagement score, in bands from "Under 1" to "20 or more".
Each value shows how many targeted contacts have it and their share. The 12 most common values are listed and the rest are counted together. Contacts with no value get their own row, No value, or No activity recorded for engagement. On a very large list, the panel may break down a sample of it and says so.
Retro Revival's mix
Retro Revival's filter targets 6,420 subscribers. Broken down by customer_type:
| customer_type | Contacts | Share |
|---|---|---|
| prospect | 3,981 | 62% |
| customer | 2,311 | 36% |
| No value | 128 | 2% |
Customers are about a third of the audience. Customers and prospects buy at very different rates, so if one arm happened to get more customers, it could win because of who was in it rather than because of its reward. That's what a Balanced random sample prevents: it draws separately within each value, so both arms get the same mix.
Targeted contacts
- Customer
- Prospect
Each dot stands for 80 contacts.
| Arm | Customer | Prospect | No customer type |
|---|---|---|---|
| Arm A · 10% off | 720 | 1,240 | 40 |
| Arm B · Free shipping | 720 | 1,240 | 40 |
| Arm | Customer | Prospect |
|---|---|---|
| Arm A · 10% off | 11 | 14 |
| Arm B · Free shipping | 7 | 18 |
What to check before you sample
Values that should be one. Filters ignore capitals, but the breakdown and balanced sampling don't. "Customer" and "customer" show as two rows and become two groups. If you see near-duplicates, fix the values in the list first.
The size of the No value row. Retro Revival's 128 contacts without a customer_type are 2% of the audience. A balanced sample treats them as a group of their own and splits them evenly, which is fine at that size. If a third of the audience has no value, fill the column before you balance on it, or balance on another field.
Groups that are too small. A value held by a handful of contacts adds little to balancing and can't tell you anything on its own. Merge rare values into a broader one, such as an "other" value, if the field has many.
Who's actually active. The Engagement score view shows how much of the audience has interacted with you lately. A large No activity recorded share means lower rates than your history suggests, so check the planner's baseline before you trust its sample size.
What it isn't. The panel describes the audience. It doesn't compare anything, and its shares don't predict results.
Then choose the sample
Under Sample, pick Balanced random sample and set Balance by to the field you checked. Balanced sampling is included from the Pro plan up. On other plans, a Random sample still assigns arms at random, and with a few thousand contacts the mix usually comes out close.
Retro Revival balanced on customer_type and drew 4,000 contacts: 1,440 customers, 2,480 prospects and 80 with no value, split evenly between the two arms.
Getting started
In a draft experiment's Audience & sample step, open Audience insights and break the audience down by the field you plan to balance on. Clean up duplicate values and large gaps in the list, then choose Balanced random sample. To reuse this audience in other sends and tests, read part 3.