A result is statistically significant when the observed difference would be unlikely if there were no real effect, judged against a threshold set in advance (often a p-value below 0.05). In analytics reporting, it helps answer whether a change in a metric, such as a rise in conversion rate after a page update, reflects something real or ordinary noise. Significance says nothing about whether the effect is large enough to matter.
Core ideas
These terms show up in any test or comparison.
- Null hypothesis: the assumption that there is no real difference
- p-value: how surprising the data would be if the null hypothesis were true
- Significance level (alpha): the threshold for calling a result significant, chosen before looking
- Confidence interval: a range of plausible values for the true effect
- Statistical power: the chance of detecting a real effect of a given size
Applying it to reporting
Week-to-week swings in conversion rate on small numbers are often noise. Before reporting a change as a win, check whether the sample is large enough and whether the confidence interval for the difference excludes zero. For before-and-after comparisons on a site, seasonality and other changes can also cause differences, which significance tests alone do not account for.
Two-proportion example (illustrative):
Page A: 120 sign-ups / 4,000 sessions = 3.0%
Page B: 150 sign-ups / 4,000 sessions = 3.75%
Test whether the 0.75-point gap is larger than chance would explain.Common traps
Checking results repeatedly and stopping as soon as they look significant inflates false positives. Testing many metrics or segments at once makes it likely that some will appear significant by chance. Decide the metric, sample and duration before you start.
Example
A headline change that was noise
Suppose Fallowmere, a forms SaaS, reports a jump in demo requests from one landing page the week after changing its headline. With only a few dozen requests, the confidence interval for the change is wide and includes zero. The team keeps the new headline but waits for more data before claiming a result.
Common mistakes
Equating significant with important
A tiny effect can be significant on huge samples. Look at effect size too.
Peeking and stopping early
Repeated checks without correction lead to false positives.
Questions
It is a convention, not a law. Choose a threshold that fits the cost of being wrong, and decide it before you look at results.
