A/B test significance calculator
Enter the visitors and conversions of each version to see if your result is statistically significant, and what to do next.
—
—
—Confidence reached —
needed 95%
—
The bar shows where the real uplift most likely sits. It is significant when the whole bar is on one side of “No change”.
Planning your next test?
See how many days it needs to run.
A/B testing best practices
How to read an A/B test result
Before you act on a result, check these three things in order.
- 01
Direction
Did the metric move the way your hypothesis predicted? If it moved somewhere else, treat it as a new idea to test.
- 02
Impact
Is the lift big enough to matter for the business, compared with the effort to roll it out everywhere?
- 03
Confidence
Did it reach the confidence you set before the test? Use the calculator above, and only once the planned sample is in.
What to do with the result
| You see | Do this |
|---|---|
| Uplift, significant | Roll outShip the variant, then apply the learning to similar pages and campaigns. |
| Drop, significant | Keep controlThe change hurts. Keep the original and note why it failed. |
| Close to the line | ExtendConfidence is near the level you need. Keep it running to the planned sample if you can. |
| Far from the line | Move onNo real difference. Log it and test a bolder idea. |
Common mistakes
- Stopping the test early because the first days look good, when results usually move a lot before they settle
- Editing the versions or the audience while the test is running
- Running the test during a sale or holiday and treating the result as normal
- Changing several things at once, so you can't tell which one made the difference
- Rolling out a lift that is real but too small to be worth the work
Read the full A/B testing guide →
Questions about significance
What is the difference between a one-sided and a two-sided test?
A two-sided test checks if the variant is better or worse than the control. A one-sided test only checks if it is better, so it reaches significance with fewer visitors but can never confirm that a change hurts. Use the same setting you planned the test with.
What does statistically significant mean in an A/B test?
It means the difference between the versions is unlikely to be random chance. At 95% confidence, a difference this big would appear by luck less than 5% of the time if the change had no real effect.
What p-value should I use?
0.05, or 95% confidence, is the standard. For low-risk changes that are easy to undo, like copy or colours, 0.10 (90%) is a reasonable trade for faster answers. Pick it before the test starts, not after you see the data.
What should I do if my test is inconclusive?
If the confidence is close to the level you need, extend the test until the planned sample is reached. If it is far off, the change probably makes little difference. Log it and move on to a bolder idea.
Is a significant result always worth rolling out?
Not always. Statistical significance says the lift is real. Business significance asks if it is big enough to matter for the effort. A real +1% on a minor page may not be worth the build.
