Should a new trial user get one short email that points at the product, or a guided tour over several days? A journey experiment answers it with your own signups. Each new contact is assigned one path at the split and keeps it, and the goal step counts who got started. When the result is in, you decide, and you send everyone down the winning path yourself.
This guide follows Darkroom Cloud, a fictional photo-storage app, and every figure here is made up. It's the kind of test that fits a journey better than a single send: the two versions differ in more than one email, and new contacts arrive every day rather than all at once.
Meet Darkroom Cloud
Darkroom Cloud stores and organizes photographers' libraries. Its trial lasts 14 days, and the team knows one thing about who stays: people who upload a first album early tend to keep using it. About 140 people start a trial each day.
Every signup lands in a Trial signups contact list through the API, and a journey starts with the Contact joins a list trigger on that list. The team wanted to know which welcome gets more new users to upload a first album within a week.
Two paths after one split
The journey's first step is a split with two paths:
- Path 1, Quick start (the control): one email with a single button, "Upload your first album", then a wait of 6 days and a goal step.
- Path 2, Guided tour: an email that asks what you shoot most with a Poll block and opens the tour from there, a wait of 2 days, a second email with tips for that kind of photography, a wait of 4 days and a goal step.
Path 1 · Quick start (control)
Subject
Your first album in two minutes
Path 2 · Guided tour
Subject
Let's set up Darkroom Cloud together
The goal step's rule is "first_album is yes". The app sets that contact field through the API the moment someone uploads. A goal step checks its rule when a contact reaches it, so both paths end with one, 6 days after the split. That's inside each contact's 7-day window, so the two paths are measured at the same point.
Choose "Run an experiment"
On the split, Routing offers three modes. Fixed shares keeps the weights you set and measures nothing. Maximize a result (optimization) moves more contacts to whichever path converts better as results come in. Run an experiment (pre-registered test) keeps the shares fixed and gives you a result you can trust.
- Path 1
- Path 2
Day 0
Day 2
Day 7
Day 21
The shares you set, for as long as the journey runs. Nothing is compared.
| When | Path 1 | Path 2 |
|---|---|---|
| Day 0 | 50% | 50% |
| Day 2 | 50% | 50% |
| Day 7 | 50% | 50% |
| Day 21 | 50% | 50% |
| When | Path 1 | Path 2 |
|---|---|---|
| Day 0 | 50% | 50% |
| Day 2 | 19% | 81% |
| Day 7 | 12% | 88% |
| Day 21 | 8% | 92% |
| When | Path 1 | Path 2 |
|---|---|---|
| Enrollment, 20 days | 50% | 50% |
| After the decision | 0% | 100% |
Maximizing sounds tempting, since it would have sent most new users down the better path within days. But it never gives a fair comparison or names a winner, so the team would never know how much better the guided tour is, or whether it's better at all. They wanted that answer before changing their onboarding for good, so they chose the experiment.
Pre-register the question
Set up the experiment opens four steps. In Hypothesis & metrics, the team wrote: "A guided tour gets more new users to upload a first album within a week than a quick-start email, because new users need a walkthrough." Then they fixed the metrics:
- Goal rate as the primary metric: the share of contacts who reached the split and whose journey ended at a goal step within their window. It needs a goal step after the split, which both paths have.
- Unique click rate as a secondary metric, to explain the result without deciding it.
- Unsubscribe rate as a guardrail, so a winner that drives people away is caught.
In Arms & content, the arms are the split's paths in order. The team named them "Quick start" and "Guided tour", kept path 1 as the control and left the shares at 50% each.
Work out how long to enroll
In Schedule & window, Each contact's window is set to 168 hours: every contact's results count for 7 days from the moment they reach the split.
How long to enroll read the journey's last 28 days. 3,920 contacts reached the split in that time, 140 a day, and 30% of those whose window had ended reached a goal. The team asked for the Smallest rise worth detecting of 5 percentage points, at 80% power. The planner then showed:
- Contacts that must reach the split: 2,754, or 1,377 per path.
- Recommended enrollment: 20 days. At 140 contacts a day, each path reaches its sample in 19.7 days. Then each contact's 7-day window runs, so final results come about 27 days after the launch.
Use 20 days set the enrollment. Review & lock froze it all, and the team launched.
While it enrolls
From the launch, every contact who reaches the split in the next 20 days is assigned a path and keeps it, even if they sign up again later. After those 20 days, the split goes back to its weights. Each contact counts only once their own window has ended. Until then the results page lists them as Still inside their window.
Halfway through, the team wanted to reword the guided tour's second email. The editor showed a notice on that step: it was on a running experiment's paths, and any change would be logged. Changes are allowed, but each one is listed under Changes during the experiment with its date, because contacts before and after it got different versions. The team decided to wait until the experiment ended.
The journey also can't be deleted while its experiment observes. That protects the contacts already assigned and the results they'll produce.
Read the result
The final analysis came one window after enrollment ended. 2,800 contacts had reached the split: 1,412 on the quick start and 1,388 on the guided tour. The Sample ratio check passed (p = 0.65): each path got the share of the contacts it should have.
The results counted everyone who reached the split, whatever happened next, which is intention to treat. A contact whose tour email bounced still counts on the guided tour. That keeps the comparison fair.
The primary metric:
| Path | Reached the split | Reached a goal | Goal rate |
|---|---|---|---|
| Path 1, Quick start (control) | 1,412 | 424 | 30.0% |
| Path 2, Guided tour | 1,388 | 487 | 35.1% |
The guided tour's goal rate was 5.1 points higher, likely between 1.6 and 8.5 points (p = 0.004). The result reads Significant. With this many contacts, the test could reliably detect a difference of about 5.0 points, which is the rise the team planned for.
The other numbers explain it without deciding it. The guided tour's unique click rate was 60% against 56%, but it's a secondary metric, read as exploratory. The unsubscribe guardrail was 1.5% against 1.3%, Not shown to be worse, so nothing blocked the winner.
Record the decision, then send everyone down path 2
The experiment ends with a decision the team records. Record decision asked for the outcome, the winning arm and a rationale. They picked Winner, the guided tour, and wrote why. The evidence recorded with it, "significant", comes from the analysis, not from the form.
Recording the decision changes nothing in the journey. A new button did appear, on the results page and on the split: Send everyone down path 2. Its dialog said what it does: from now on, every contact who reaches the split goes down path 2. It also repeated the evidence, "The analysis named this arm the winner: the difference is significant." The results don't change, since everyone already assigned stays in their arm.
The team clicked it. The canvas now reads "Everyone → path 2 (experiment winner)", with 100% and 0% on the two paths. Undo would put the split back to its weights. Nothing would have happened without that click.
What if it hadn't been significant?
Say the guided tour had come out only slightly ahead, not significant. The team could record Inconclusive and keep the split as it is, or run a longer experiment. That's a normal result: it says a difference this small can't be told apart on this many signups.
They could also record a winner anyway. The record then says, for good, that the analysis didn't name that arm. Send everyone down path N still works, but its dialog warns: "Not significant: the analysis didn't name this arm the winner, so this is your call."
Getting started
Put a split at the start of your journey with one path per version, and end each path with a goal step inside the window you plan. Choose Run an experiment (pre-registered test), then Set up the experiment, and let How long to enroll set the enrollment. Experiments inside journeys covers each step, and the split step compares the three routing modes.