Incrementality asks a causal question: how many conversions happened because of this activity that would not have happened without it? Attribution models assign credit among touchpoints but cannot answer that. Incrementality tests do, by withholding the activity from a comparable group (a holdout) and comparing outcomes. They are the most reliable way to check whether a channel, campaign or tactic is actually adding value.
Common test designs
The design depends on what you can randomize.
- User-level holdout: randomly exclude a share of an audience from a campaign or email
- Geo experiment: turn an activity on in some regions and off in comparable ones
- Time-based switchback: alternate on and off periods, used with care because of seasonality
- Platform lift studies: ad platforms' own randomized tests, run inside their systems
Calculating lift
Lift compares the test group with the holdout, scaled to group size.
Lift = (Test conversion rate - Holdout conversion rate) / Holdout conversion rate
Incremental conversions = (Test rate - Holdout rate) x Test group sizeIncrementality and SEO
SEO is hard to hold out, since you cannot hide pages from some searchers. Teams instead use split tests across groups of similar pages, before-and-after comparisons with control groups, or tests on paid brand search to see how much organic results would capture if ads were paused. Each of these has assumptions worth stating in the results.
Example
Pausing brand ads in some regions
Imagine Thistlebank, a recruiting software company, pauses paid brand search in a set of matched regions for four weeks while keeping it on elsewhere. Total brand sign-ups in paused regions barely change, as organic results pick up most clicks. The team reduces brand ad spend and rechecks the result the following quarter, since competitor bidding could change the picture.
Common mistakes
Groups that are not comparable
If test and holdout differ before the test, the result reflects that difference.
Tests too short or too small
Without enough volume, the result is noise. Estimate the needed sample first.
Questions
Attribution divides credit for observed conversions among touchpoints. Incrementality measures how many conversions a touchpoint caused, using a comparison group.
