Incrementality Testing in Practice: From Investment Case to Proof

Nitin, Haipeng, Susan

MAI Performance Marketing Team

Every mature performance marketing program eventually runs into the same problem:

Early on, finding growth opportunities is relatively easy. As an account matures, though, the obvious wins disappear. Every new dollar has multiple places it could go, and the trade-offs become much harder. Dashboards are great at explaining what already happened. However, they're much less helpful when you're trying to decide what to invest in next.

A mature ecommerce brand in Japan faced a familiar problem: the obvious account optimizations had already been made, but the team still needed to decide where the next marketing dollar should go. The account was already performing efficiently, so the challenge was not fixing an underperforming campaign. It was identifying where additional investment could still create incremental growth.

MAI utilized advanced ML technology to identify one such opportunity, built a quantified investment case around it, and then used incrementality testing to determine whether the recommendation should actually be funded.

---

Finding the Next Growth Opportunity

The client’s existing portfolio already possessed high ROAS, stable conversion volume and extraordinary brand-awareness. The central concern was incrementality, the classic brand search cannibalization question: were additional branded clicks creating new business, or simply capturing customers who would have converted organically?

Brand Search looked deceptively strong. It was already delivering excellent returns, but with impression share sitting at just 66%, there was still meaningful room to grow. The real question was whether spending more would actually generate new business or simply capture conversions that would have happened anyway.

Building the Investment Case

The first thing the MAI team did was to work with the business owner to understand how much to spend, what impact to expect, and what assumptions could prove the investment case wrong.

The MAI Intelligence system then translated the opportunity into a quantified investment case.

Brand Search did not surface simply because it had high ROAS. MAI evaluated the opportunity against other places the account could deploy budget, including existing efficiency, available auction headroom, expected marginal return, and the risk that paid clicks would substitute for organic demand. The proposed spending plan was then sized conservatively, using an 80% organic-substitution assumption to test whether the economics would still hold under an unfavorable scenario.

The objective was a proposal that would stay attractive even if the most important uncertainty moved against us, not the largest possible forecast.

Validating the Investment with Incrementality Testing

Even though the simulated numbers seemed compelling, we still did not fully trust them. Good investment proposals should make their assumptions visible and testable. So before increasing Brand Search budgets nationwide, MAI ran a geo-lift incrementality test across comparable treatment and holdout regions in Japan, validating first with an A/A test that the two groups behaved similarly. The question was then simple: would the additional investment increase total GMV beyond what would have happened anyway?

What the Experiment Revealed

The observed results were materially stronger than the conservative case used to justify the investment. The experiment ultimately deployed a larger spend increase than the initial proposal, which allowed us to see whether the opportunity continued to hold at greater scale.

The final experiment outcome did more than validate the investment case. GMV increased far more than expected, while spend grew at a much slower pace, pushing marginal ROAS from roughly 2x to about 6x. As a result, the decision changed from "Brand Search might be worth scaling" to "Brand Search is one of the highest-confidence places to deploy additional budget."

---

From Recommendations to Investment Decisions

Every marketing team has more ideas than budget. A proposal carries a higher burden: it must specify the action, quantify the investment, estimate the impact, and expose the assumptions that could make it wrong. The proposal determines what is worth testing. The experiment determines whether the business should act.

In this case, the next dollar belonged in Brand Search. But the broader lesson is that the best marketing decisions aren’t driven by dashboards alone. They’re built on quantified investment cases and validated with incrementality testing.

Which opportunities in your account deserve funding?

See how MAI identifies your highest-confidence investment opportunities and validates them with incrementality testing.