Sequential testing
Intempt's Statistics Engine uses mSPRT sequential testing so you can monitor experiment results continuously without inflating false positive rates.
Overview
Intempt's Statistics Engine runs on a sequential testing method, so you can check experiment results as often as you want without corrupting them.
Traditional A/B testing practice says you should only read out an experiment's metrics once it has reached its target sample size. Checking results early and often, usually called "peeking," inflates the false positive rate well beyond the significance level you set for the test.
That happens because p-values fluctuate over the course of an experiment and can drop in and out of significance by chance alone, even when there's no real effect. If you pick whichever day the result happens to look significant, you're effectively cherry-picking a result that wouldn't hold up if you'd only looked once, at the end.
Sequential testing removes that tradeoff. It adjusts the significance threshold at every check so continuous monitoring no longer inflates the false positive rate, which means you can peek as often as you like and still trust the result.
How Intempt's sequential testing works
Intempt's Statistics Engine uses mSPRT (mixture sequential probability ratio test) as its sequential testing method. Combined with CUPED, mSPRT lets an experiment reach statistical significance faster while keeping the result valid at any point you check it, not just at a predetermined end date.
Sequential testing is one of the statistical controls on an experiment's Results tab, alongside Compare, relative-to, Confidence Interval (6 levels), CUPED, Benjamini-Hochberg, and Zoom. Toggling it on shows this tooltip in the product:
📘 Good to know
"Sequential testing allows early stopping when results are conclusive."
Once enabled, "Sequential Testing" also shows up as a read-only Enabled/Disabled status on the Statistical Methods card in the Time Series tab, and as a "Sequential Testing" tag on any variant row where it's turned on for that experiment. The toggle itself only lives on the Results tab; the other views just reflect its current state.
Sequential testing and multi-metric correction
Running sequential tests across several metrics and variations at once raises a second problem: the more comparisons you run, the higher the odds that at least one turns significant by chance. Intempt controls for this with the Benjamini-Hochberg (BH) procedure, a false-discovery-rate correction you toggle on or off alongside Sequential Testing on the Results tab.
📘 Good to know
BH is a single on/off toggle in Intempt, not a choice between multiple correction methods. Its tooltip in the product: "Benjamini–Hochberg procedure controls the false discovery rate when testing multiple metrics."
Turn both on together and a winning or losing variation has cleared two bars: it's stable enough to call early, and it's still significant after correcting for every other metric and variation in the same experiment.
Where to go next
Multivariate testing
Test more than two variants against a single control in one experience, then compare all of them side by side in Results.
Statistical significance
Statistical significance is a statistical method used to determine whether the difference between your variation and baseline is likely caused by the changes you made, or just random chance. In sim...