A/B Test Sample Size Calculator

Welcome to the most comprehensive a/b test sample size calculator on the web. Before you launch your next experiment, use this tool to determine the sample size needed to achieve statistically significant results. This calculator helps you avoid common pitfalls like ending tests too early or running them for too long. A proper a/b test sample size calculation is the foundation of reliable conversion rate optimization.

Sample Size vs. Minimum Detectable Effect

The following table breaks down the variables used in the a/b test sample size calculator.

Variable	Meaning	Unit	Typical Range
Baseline Conversion Rate (p1)	The conversion rate of your existing page (Control).	%	0.1% – 30%
Minimum Detectable Effect (MDE)	The smallest relative lift you aim to detect.	%	5% – 50%
Statistical Significance (α)	The risk of a false positive (Type I error).	%	90%, 95%, 99%
Statistical Power (1-β)	The probability of detecting a true positive.	%	80%, 90%, 95%
Sample Size (n)	Number of users needed for each variation.	Users	Varies widely

What is an A/B Test Sample Size Calculator?

An a/b test sample size calculator is an essential tool for marketers, developers, and product managers who engage in conversion rate optimization (CRO). Its primary function is to determine the number of users (the “sample size”) that need to participate in an A/B test to reliably detect a specific change in user behavior. Without a proper sample size, you risk drawing incorrect conclusions: either believing a change had an effect when it didn’t (a false positive) or missing a genuine improvement (a false negative). This calculator ensures your test results are statistically significant.

Anyone running controlled experiments on a website, app, or in a marketing campaign should use an a/b test sample size calculator. A common misconception is that you can simply run a test until one version “looks” better. This often leads to premature decisions based on random fluctuations. The a/b test sample size calculator provides a mathematical framework to avoid being misled by chance.

A/B Test Sample Size Formula and Mathematical Explanation

The core of an a/b test sample size calculator relies on a formula derived from hypothesis testing for proportions. The goal is to calculate the number of observations (n) per group needed to detect a difference between two proportions (p1, the baseline conversion rate, and p2, the variant conversion rate) with a given level of statistical power and significance.

The standard formula for a two-sided test is:

n = (Zα/2 + Zβ)2 * (p1(1-p1) + p2(1-p2)) / (p1 - p2)2

Here’s a step-by-step breakdown:

1. Z_α/2: This is the Z-score corresponding to the chosen statistical significance level (α). For a 95% significance level (α=0.05), this value is 1.96 for a two-tailed test.
2. Z_β: This is the Z-score for the statistical power (1-β). For 80% power (β=0.20), this value is 0.84.
3. p1: The baseline conversion rate of your control group.
4. p2: The expected conversion rate of your variant (p1 * (1 + MDE)).
5. (p1 – p2)²: The squared difference in conversion rates. This is the effect size; smaller differences require larger sample sizes to detect.

The a/b test sample size calculator automates this complex calculation, allowing you to focus on the strategic implications rather than the manual math.

Practical Examples (Real-World Use Cases)

Example 1: E-commerce Checkout Button Color

An e-commerce site wants to test if changing their “Complete Purchase” button from blue to green increases conversions. They use an a/b test sample size calculator to plan the experiment.

• Inputs:
  • Baseline Conversion Rate: 4.0%
  • Minimum Detectable Effect: 15% (they want to detect at least a 0.6% absolute lift)
  • Statistical Significance: 95%
  • Statistical Power: 80%
• Outputs:
  • Sample Size Per Variation: ~9,848 users
  • Total Sample Size: ~19,696 users
  • Variant Conversion Rate to Detect: 4.6%
• Interpretation: The team needs to drive approximately 9,848 users to the blue button version and 9,848 to the green button version. If after collecting this data the green button’s conversion rate is 4.6% or higher, they can be 95% confident the improvement is real and not due to random chance.

Example 2: SaaS Landing Page Headline

A B2B SaaS company is testing a new headline on their demo request page. They believe the new headline will clarify their value proposition and increase demo requests. They turn to the a/b test sample size calculator for guidance.

• Inputs:
  • Baseline Conversion Rate: 1.5%
  • Minimum Detectable Effect: 20%
  • Statistical Significance: 95%
  • Statistical Power: 80%
• Outputs:
  • Sample Size Per Variation: ~21,365 users
  • Total Sample Size: ~42,730 users
• Interpretation: Due to the low baseline conversion rate, detecting a 20% relative lift requires a much larger sample. The marketing team now knows they need to allocate significant traffic to this test to get a reliable result. Using an a/b test sample size calculator prevents them from launching a test that is doomed to be underpowered from the start.

How to Use This A/B Test Sample Size Calculator

1. Enter Baseline Conversion Rate: Input the current conversion rate of your control page or element as a percentage. For example, if 3 out of 100 users convert, enter ‘3’.
2. Set Minimum Detectable Effect (MDE): Decide on the smallest *relative* improvement you care about. An MDE of 10% on a 3% baseline means you want to detect if the new version hits 3.3% or higher.
3. Choose Statistical Significance: 95% is the industry standard. Selecting 99% will require a larger sample size but gives you more confidence in the result.
4. Select Statistical Power: 80% is standard. This means you have an 80% chance of detecting an effect if one truly exists. Increasing power to 90% or 95% requires more users.
5. Read the Results: The calculator instantly provides the ‘Sample Size Per Variation’. This is the number of users you need for both your original (A) and your new version (B). The total sample size is also displayed.

Use the output from the a/b test sample size calculator to plan your experiment’s duration. For instance, if your page gets 1,000 visitors per day and you need 20,000 total visitors, your test will need to run for at least 20 days.

Key Factors That Affect A/B Test Sample Size

Several factors influence the output of an a/b test sample size calculator. Understanding them is key to effective test planning.

• Baseline Conversion Rate: The lower your baseline rate, the more traffic you’ll need. It’s harder to detect a 10% lift on a 1% conversion rate than on a 20% conversion rate.
• Minimum Detectable Effect (MDE): This is the most critical lever. Wanting to detect a very small change (low MDE) requires a massive sample size. Being willing to only detect larger changes (high MDE) reduces the required sample size.
• Statistical Significance (Alpha): A higher significance level (e.g., 99% vs. 95%) means you want to be more certain you aren’t seeing a false positive. This increased certainty requires more data.
• Statistical Power (1-Beta): Higher power (e.g., 90% vs. 80%) means you have a better chance of detecting a real effect. This, too, requires a larger sample size as it reduces the risk of a false negative.
• Number of Variations: A standard A/B test has two variations. An A/B/n test with more variations will require the calculated sample size *for each variation*.
• Variance: A metric with high natural variance will require more samples to distinguish the signal (the effect of your change) from the noise (random fluctuation). Conversion rates are a proportion, and their variance is factored into the formula (p*(1-p)).

Frequently Asked Questions (FAQ)

1. Why can’t I just run my test until it’s significant?

This is called “peeking” and it dramatically increases your risk of a false positive. Statistical significance can fluctuate and will almost certainly be reached by pure chance at some point if you watch it long enough. The correct method is to determine the sample size with an a/b test sample size calculator first, and then run the test until that size is reached.

2. What if I can’t reach the required sample size?

If the required sample size is too large for your traffic, you have two options: either increase your Minimum Detectable Effect (be willing to only detect larger wins) or decrease your statistical confidence (e.g., lower power to 70%, though not recommended). An a/b test sample size calculator helps you see these trade-offs clearly.

3. How does this calculator differ from a significance calculator?

A sample size calculator is used *before* a test to plan it. A significance calculator is used *after* a test to analyze the results and determine if the observed difference was statistically significant.

4. What is the difference between relative and absolute MDE?

This calculator uses a relative MDE. If your baseline is 2% and you set a 10% relative MDE, you’re trying to detect a change to 2.2% (a 0.2% absolute lift). It’s crucial to understand this distinction when planning your tests.

5. Is a larger sample size always better?

While a larger sample size provides more statistical power, there are diminishing returns. It may not be worth running a test for six months to detect a 0.1% lift. The a/b test sample size calculator helps find the right balance between confidence and practicality.

6. Should I use a one-tailed or two-tailed test?

This calculator uses a two-tailed test, which is the standard for A/B testing. It checks if the variant is either better OR worse than the control. A one-tailed test only checks for an effect in one direction (e.g., better) and is generally not recommended unless you have a very strong reason to ignore a negative effect.

7. How many variations can I test?

The sample size calculated is *per variation*. If you test four variations (A, B, C, D), you’ll need four times the sample size shown by the calculator. Running many variations requires very high traffic.

8. What if my conversion rate is very low (under 0.5%)?

For very low conversion rates, the required sample sizes can become extremely large. In these cases, you might need to accept a higher MDE or focus on optimizing for micro-conversions (like clicks or form interactions) that happen more frequently. Our a/b test sample size calculator will show you the numbers needed, even if they are large.

Related Tools and Internal Resources

Expand your optimization toolkit with these related resources. Using an a/b test sample size calculator is just the first step.