An A/B test shows version A (the control) to one randomly assigned group of visitors and version B (the variant) to another, then compares a goal metric between the groups. Because assignment is random and both run at the same time, a meaningful difference can be attributed to the change rather than to seasonality, traffic mix or luck, provided the test has enough data.
How an A/B test runs
You define a hypothesis, a primary metric and the smallest effect worth detecting, then calculate the sample size needed. The testing tool assigns each visitor to a variant, usually keeping them in that variant on return visits. Once the planned sample is reached, you analyze the result with the statistical method you chose in advance.
- Hypothesis: what will change, why, and which metric will move.
- Primary metric plus guardrail metrics that must not get worse.
- Sample size and duration calculated before launch.
- Random, consistent assignment of visitors.
- Analysis at the planned end, not whenever the numbers look good.
Tools
Google Optimize, once a common free option, was sunset in September 2023. Teams now use dedicated experimentation platforms, feature-flag tools with experiment support, or server-side tests built in-house. Client-side tools that rewrite the page in the browser can cause flicker and layout shifts, so server-side or edge-based assignment is preferable for performance-sensitive pages.
A/B testing and SEO
Google's guidance says testing is fine as long as you do not cloak, meaning you must not show Googlebot different content than users see. For split URL tests, use rel=canonical pointing to the original and 302 redirects rather than 301s, and end tests once they finish rather than leaving variants live indefinitely.
Example
Testing a pricing page headline
Hypothetical scheduling app Fernhollow tests a pricing page headline that addresses team size directly against its original. It sets trial starts as the primary metric and paid conversion within 30 days as a guardrail, computes the sample size up front, and runs the test for full weeks so weekday and weekend visitors are both represented.
Common mistakes
Peeking and stopping early
Checking results daily and stopping the first time the variant looks ahead inflates false positives. Decide the stopping rule in advance or use a method designed for continuous monitoring.
Testing tiny changes on low traffic
A button color tweak on a page with few conversions may need months to reach a reliable answer. Test bolder changes or use qualitative research instead.
Ignoring downstream effects
A variant that wins on clicks but loses on qualified leads or revenue is not a winner. Include guardrail metrics.
Questions
Yes, on different pages or with tools that handle overlapping experiments. Running interacting tests on the same element without accounting for overlap makes results hard to read.
