Geo Holdout Tests: Measuring What Your Ads Actually Caused

Every attribution model answers the same question: given that a sale happened, which touchpoint should get credit? That's a useful question. It's not the question a budget decision needs.

"The budget question isn't 'which channel gets credit for this sale' — it's 'what happens to total revenue if I stop spending here.' Only a holdout answers that."

— Aashish Kasma, Co-Founder — Data & Analytics

What is a geo holdout test?

You split your market into two comparable sets of regions. One set keeps running the channel you're testing. The other set has it switched off entirely. After enough time to cover a full purchase cycle, you compare total revenue between the two — not platform-attributed revenue, total revenue from your order system.

The difference is the incremental contribution of that channel. Not its attributed contribution, not its reported ROAS: the actual revenue that existed because the spend existed.

Why not just use attribution or MMM?

They answer different questions at different costs. Attribution is cheap, fast, and structurally biased toward channels that sit close to the conversion — retargeting and brand search look excellent under any last-touch-leaning model because they intercept demand that already existed. Marketing mix modelling is unbiased in principle but needs substantial spend history and delivers correlational estimates rather than experimental ones.

A geo holdout is a real experiment. Its weakness is cost — you deliberately forgo revenue in the holdout regions for the duration — and it only works if you have enough geographic spread for the two groups to be genuinely comparable.

How do you set one up without wrecking the read?

1. Match the groups on pre-period behaviour, not on population. Pair regions with similar historical revenue, AOV, and seasonality, then split the pairs. A split by "big cities vs. small cities" produces a difference you can't interpret.
2. Run for at least one full purchase cycle, plus a buffer. Ending the test at two weeks when your median lag to purchase is three weeks measures the ramp-down, not the effect.
3. Hold everything else constant. No promotions, no new creative launches, no email campaigns that hit only one group. A concurrent sitewide sale is fine; a region-specific one destroys the test.
4. Measure on total revenue from your order system, split by shipping region — not on platform-reported conversions, which won't reflect the holdout at all.
5. Decide the success criteria before you start, including what result would cause you to cut the budget. Otherwise the result gets reinterpreted after the fact.

Turn the channel off completely in the holdout A "reduced spend" holdout is much harder to read than a clean on/off, because you can't separate the effect of less spend from the effect of a changed delivery pattern. If a full switch-off is too risky commercially, run the test on a smaller share of regions rather than a smaller reduction.

What do you do with the result?

Resize, don't celebrate or panic. A channel measuring at, say, half its reported contribution isn't a failure — it means the reported number was always mixing incremental and non-incremental sales, and now you know the ratio. Set the budget against the incremental figure, re-run the test once or twice a year (incrementality drifts as saturation and creative change), and use the ratio you measured as a correction factor on the platform's reporting in between tests.

The channels where holdouts most often change minds are retargeting and brand search — both of which tend to report far more than they add. That's not an argument for switching either off, but it usually is an argument for spending less there and more on prospecting.

Related guides

• ROAS and Attribution: The Truth About Blended vs. True Return
• Marketing Mix Modeling for D2C Brands
• Attribution Modeling: A Practical Guide
• Marketing Budget Allocation for D2C Brands

Holdout testing is the only method on this list that produces causal evidence rather than a credit assignment. It costs real revenue to run, which is exactly why it's worth doing on the two or three budget decisions large enough to justify the price of finding out.