What a checkout A/B test is
In an A/B test, part of your buyers see your current checkout (A) and the rest see a changed version (B). Both run at the same time, on the same traffic, so the only real difference between the groups is the change itself. After enough checkouts you compare how many people paid in each group.
Running both at once is the point. Comparing this week with last week mixes your change up with everything else that moved: ad spend, season, a sale, a creative that took off.
What's worth testing in a checkout
Start with changes big enough to plausibly move buyers. Tiny tweaks need huge amounts of traffic before you can tell them apart.
- Trust: trust badges, a guarantee line, delivery and return info near the pay button.
- Payment icons: which methods you show up front, and in what order.
- Fields: fewer or optional fields, like company name or a second address line.
- Layout and colours: one column or two, button colour and wording.
- Offers: order bump on or off, a different bump product or price, the offer after purchase.
How A/B testing works in the Settl checkout
You create a test in your store's dashboard. Variant B starts as an exact copy of your current checkout design and offers; you change what you want to test in the same checkout designer you already use. Then you choose the share of buyers that sees B, from 1% to 99%, and start the test.
Every new checkout is assigned to A or B once and keeps it, so a buyer who leaves and comes back sees the same version. Payments, orders and tracking work exactly the same in both variants. A store runs one test at a time.
When you're done, you stop the test and either keep A, or make B your live checkout for everyone in one click. Earlier tests stay in your history with the variant you picked.
How to read the result
For each variant you see checkouts, orders, conversion, revenue, revenue per checkout and average order value. Revenue from an offer after purchase counts towards the variant the buyer was in.
Settl compares the two conversion rates with a standard two-proportion test. Until each variant has at least 100 checkouts and there are at least 10 orders in total, it tells you to keep going. After that it either names a winner with 95% confidence, or says there's no clear winner yet because the difference could still be chance.
Look at revenue per checkout as well as conversion. An order bump can lower conversion slightly and still bring in more money per checkout, and that may be the better deal for you.
Common A/B testing mistakes
- Stopping as soon as B looks ahead. Early leads often shrink; wait for the verdict.
- Testing five changes at once. If B wins you won't know which change did it. Group changes by one idea.
- Running a test over a big sale or a launch only. Buyers then behave differently than usual; run it over at least a full week.
- Changing variant B halfway through. It mixes two tests into one result. If you must, start a new test.
When A/B testing isn't worth it yet
If your store has only a handful of checkouts a week, a test can take months to reach a verdict. In that case, fix the obvious things first, such as showing the payment methods your buyers expect and removing fields you don't need, and start testing when traffic grows.