How to read a test result with confidence

In this lesson

1. 1On a running experiment open the Result tab and find baseline (Control) and variant.
2. 2Read the p-value and the confidence interval; if the interval includes zero, say out loud that you have no result yet.

Setup checklist

The 11 steps to test and personalize your site, from your first A/B test to results you can trust. Tick them off as you go; your ticks stay in this browser.

0 of 11 ticked

1. 1.Go to Experiences and click Create experience.
2. 2.Answer How will the change be applied? with Visual editor, then choose Page experiment.
3. 3.Create a second experience of type Personalization and set Audience to Specific users with a Segment.
4. 4.Compare it with the Page experiment and note there is no split between variants.
5. 5.Build the personalization in the visual editor on a page you can reload, then reload as a matching visitor.
6. 6.Open Catalog and look at both tabs, Products and Sources.
7. 7.Open Feeds, click Create feed, choose Merchandising feed, name it and select an algorithm, then save.
8. 8.On a running experiment open the Result tab and find baseline (Control) and variant.
9. 9.Read the p-value and the confidence interval; if the interval includes zero, say out loud that you have no result yet.
10. 10.Create an experience with How will the change be applied? set to SDK, choose Server experiment and give it a name.
11. 11.In your backend identify the user (userId, or sourceId plus profileId), POST to the Choose API before rendering, and apply choices[].body matched by name.

Your tech stack and tools

The core stack (everything you actually need)

• Intempt console (Experiences, Catalog) (This is where you build tests, personalizations and recommendations)
• intempt-js web SDK (This is what applies your changes on your live site)
• Experiences (This is where every test and personalization lives)
• Catalog and Feeds (This is your product data, ready to recommend anywhere)
• Choose API (This is how your server decides which version each visitor gets)

Highly recommended optional upgrades that level up your work

• Shopify (This is the fastest way to load your products if you sell on Shopify)
• Custom catalog source (This is how you load products from any other store)
• Node SDK (This is the easy way to call experiments from a Node backend)
• Statistical significance (This is how you know a winner is real, not luck)

Simple A/B tests are free, with your first 1K impressions a month included. Advanced testing is on the Organization plan.