In experimentation, sample size is the number of observations, usually visitors, each variant must receive to have a good chance of detecting a real difference of a given size. It depends on the baseline conversion rate, the smallest effect you want to detect, the significance level and the statistical power. Calculating it before a test starts tells you whether the test is worth running and how long it will take.
What drives the number
Four inputs decide the required sample. Lower baseline rates and smaller effects both demand more visitors. Stricter significance levels and higher power also raise the number.
- Baseline conversion rate of the control.
- Minimum detectable effect: the smallest lift worth finding.
- Significance level (alpha), commonly 0.05.
- Statistical power (1 minus beta), commonly 0.8.
A quick approximation
For a two-variant test on a conversion rate, with alpha of 0.05 two-sided and 80% power, a widely used rule of thumb gives the visitors needed per variant. It is an approximation; use a proper calculator for real planning.
n per variant ≈ 16 × p × (1 − p) / d²
p = baseline conversion rate (e.g. 0.04)
d = absolute difference to detect (e.g. 0.008)
16 × 0.04 × 0.96 / 0.000064 ≈ 9,600 visitors per variantDuration follows from sample size
Divide the total sample by the daily eligible traffic to estimate duration, then round up to whole weeks so that every day of the week is represented equally. If the answer is many months, the test as designed is not practical: target a bigger change, test on a higher-traffic page, or use a metric earlier in the funnel.
Stopping a test as soon as it looks significant, before reaching the planned sample, raises the chance of a false positive. Sequential testing methods allow early stopping, but only when designed in from the start.
Example
Deciding a test is not feasible
Hypothetical audit software Gristleford wants to test its enterprise demo form, which converts at an illustrative 2% on about 150 visits a day. A calculator shows that detecting a modest relative lift would need more than a year of traffic. The team instead runs user tests on the form and tests a larger change on its higher-traffic trial page.
Common mistakes
Skipping the calculation
Launching without a planned sample invites stopping when results look exciting, which is how false winners get shipped.
Counting page views instead of users
If the same visitor is counted several times, the sample looks larger than it is and confidence is overstated. Use the unit of randomization.
Questions
No single number applies. The sample needed depends on the baseline rate and the effect size you want to detect, which is why calculating it per test matters.
