An A/B test is statistically significant when the gap between the two conversion rates is too large to be chance at the confidence level you chose, usually 95%. Enter visitors and conversions for your control and your variant below. The calculator runs a two-proportion z-test and returns the p-value (one-sided and two-sided), the relative lift and a confidence interval for the difference. A second calculator tells you how many visitors a test needs before you start it.
How does the calculator decide if a result is significant?
It runs a two-proportion z-test, the standard test for comparing two conversion rates. Each visitor either converts or does not, so each version has a rate (conversions / visitors). The test asks how likely a gap this large would be if both versions really converted at the same rate.
| Step | Formula | Default example |
|---|---|---|
| Conversion rates | pA = conversions A / visitors A, same for B | 210 / 7,000 = 3.00%; 273 / 7,000 = 3.90% |
| Relative lift | (pB - pA) / pA | 0.90 / 3.00 = +30.0% |
| Pooled rate | (conversions A + B) / (visitors A + B) | 483 / 14,000 = 3.45% |
| Standard error (pooled) | √(p x (1 - p) x (1/nA + 1/nB)) | 0.308 points |
| z-score | (pB - pA) / standard error | 0.90 / 0.308 = 2.92 |
| p-value | Two-sided: 2 x (1 - Φ(|z|)); one-sided: 1 - Φ(z) | 0.0035 and 0.0018 |
| Confidence interval | (pB - pA) ± zcrit x √(pA(1 - pA)/nA + pB(1 - pB)/nB) | 95%: +0.30 to +1.50 points |
Φ is the standard normal distribution. The calculator computes it with a double-precision algorithm, so its p-values match statistics software to at least four decimal places. The z-test is a normal approximation and works well once each version has at least 10 conversions and 10 non-conversions; below that, the calculator warns you and you should wait for more data. Two-sided p-values from this test are the same as a chi-square test on the 2 x 2 table without a continuity correction.
How do you read the result?
- p-value. The chance of a gap at least this large if there were no real difference. At 95% confidence, a p-value below 0.05 is significant. It is not the chance that B is better.
- Relative lift. The size of the change, measured on the control rate. A significant 30% lift and a significant 3% lift are very different business results.
- Confidence interval. The range of differences the data supports. In the default example, the true gain could be as small as 0.30 points or as large as 1.50 points. If the interval crosses zero, the test has not shown a difference.
- One-sided or two-sided. Decide before the test. A one-sided p-value is half the two-sided one when B leads, which is why switching after the fact is a common way to turn a miss into a "win".
How many visitors does an A/B test need?
Plan the sample size before the test starts and do not stop early. Evan Miller's analysis of peeking showed that checking after every visitor and stopping at the first significant result can push the real false positive rate to 26.1% when the calculator says 5%. Enter your baseline rate and the smallest lift worth detecting:
The planner uses the standard normal-approximation formula for two proportions: n per version = (z1-α/2 √(2p̄(1 - p̄)) + z1-β √(pA(1 - pA) + pB(1 - pB)))2 / (pB - pA)2, where p̄ is the average of the two rates, z1-α/2 = 1.96 for 95% confidence and z1-β = 0.84 for 80% power. Other calculators use slightly different versions of this formula, so expect their answers to differ by a few percent.
Why are A/B tests hard on B2B websites?
Because traffic is low and the conversion you care about is rare. The default planner shows the problem: a page with 2,000 visitors a week and a 3% demo request rate needs about 14 weeks to detect a 20% lift. Halve the lift you want to find and the sample size roughly quadruples, to 53,211 visitors per version for a 10% lift. Three ways B2B teams work around it:
- Test bigger changes. A new offer, a shorter form or a different page structure can move conversion by 20% or more. A button color will not, and you will never have the traffic to prove a 3% gain.
- Test where the volume is. Ad creative has thousands of impressions a day. Comparing the click-through rate of two ads is the same z-test, with impressions as visitors and clicks as conversions; see our CTR calculator for the rates themselves.
- Pick a conversion higher in the funnel. Form starts or pricing page visits happen more often than closed deals. Confirm later that the winner also produces more sales-qualified pipeline, using the conversion rate at each stage; our conversion rate calculator works them out from your CRM counts.
Illustrative, not a benchmark. Brightquay, a fictional payroll software company, tests a shorter demo form on its pricing page, which gets 2,000 visitors a week at a 3% demo request rate. It only cares about a lift of 30% or more, so the planner asks for 6,455 visitors per version: seven weeks. After seven weeks each version has 7,000 visitors. The old form converted 210 (3.00%), the new one 273 (3.90%). That is the calculator's default: a 30% lift, two-sided p = 0.0035, significant at 95%. The confidence interval runs from +0.30 to +1.50 points, so the real gain could be a third of what the test shows. Brightquay ships the new form and keeps watching demo-to-opportunity rates for a month.
What else can make a significant result wrong?
- Uneven traffic split. If you set 50/50 and got 4,000 vs 3,400 visitors, the split itself is broken (a sample ratio mismatch). Fix the setup before you trust any p-value.
- Many metrics or many variants. At 95% confidence, one in 20 comparisons with no real difference comes out significant. Pick one primary metric before the test.
- Short tests. Run whole weeks so weekday and weekend visitors are both in the sample.
- The wrong conversion. A form that gets more fills from students and competitors lifts conversion rate and lowers pipeline. Check lead quality in the CRM.
Testing is one part of our conversion rate optimization service: we find where the funnel leaks, design tests that the traffic can actually support, and judge winners on pipeline. For the paid side, the same discipline applies to creative tests in our paid ads service, and our landing page glossary entry covers the page elements worth testing first.
Frequently asked
How do you calculate statistical significance for an A/B test?
Use a two-proportion z-test. Work out each conversion rate, then the pooled rate across both versions. z = (variant rate - control rate) / square root of (pooled rate x (1 - pooled rate) x (1/control visitors + 1/variant visitors)). Turn z into a p-value with the normal distribution. If the p-value is below 0.05, the result is significant at 95% confidence. The calculator above does all of this for you.
What does a p-value of 0.05 mean?
It means that if the two versions truly converted at the same rate, you would see a difference at least this large about 5% of the time by chance alone. It does not mean there is a 95% chance the variant is better, and it says nothing about how big the lift is. Read the confidence interval for that.
Should I use a one-sided or a two-sided test?
Use two-sided unless you decided before the test that you only care whether the variant is better and would treat "worse" the same as "no change". A one-sided test halves the p-value when the variant leads, so choosing it after you see the data makes weak results look significant. The calculator shows both p-values.
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you want to detect. At a 3% baseline, finding a 20% relative lift (3.0% to 3.6%) with 95% confidence and 80% power takes about 13,914 visitors per version. Use the sample size calculator on this page and decide the number before you start.
Can I stop an A/B test as soon as it is significant?
No. The p-value assumes you fixed the sample size in advance. Evan Miller showed that checking a test after every visitor and stopping at the first significant result can push the real false positive rate to 26.1% when you think it is 5%. Decide the sample size first and read the result once you reach it.
Is 90% confidence enough for an A/B test?
It can be for low-risk changes you can undo, such as ad copy or a button label, as long as you accept a 1 in 10 chance of a false positive when there is no real difference. For pricing pages, forms or anything that changes revenue, use 95% or higher.
