ROAS Is Lying to You. Here's the Test That Tells the Truth.

Your ROAS is lying to you. Not on purpose. Structurally.

Platform ROAS tells you what Meta claims credit for. It cannot tell you how much of that revenue would have happened anyway. Those are two completely different numbers, and the gap between them is where a shocking amount of ad budget quietly dies.

At small scale, you can get away with ignoring this. At real scale, ignoring it will have you confidently pouring money into a channel that's mostly taking credit for sales you were going to make regardless.

So on a large account, I stopped trusting the platform number for the one decision that mattered most, and ran a test to get the truth instead.

(Anonymized. Client identity, absolute revenue, spend, and iROAS are withheld. The test design and the relative lift are real.)

Attribution answers the wrong question

Here's the trap in one line. Attribution tells you where to assign credit. Incrementality tells you what your ads actually caused.

Most teams optimize against attribution as if it were causation. Meta reports a conversion, so Meta gets the money, so we give Meta more money. It feels like a closed loop. It isn't.

Because if a customer would have bought anyway, and Meta showed them an ad on the way, Meta reports that as a win. Your ROAS looks healthy. But the ad didn't create that sale. It just stood next to it and took the photo.

The question that actually decides a budget isn't "what did Meta get credit for?" It's: if I push more spend into this, does total new revenue move? That's a causality question, and attribution physically cannot answer it.

Why this got urgent at scale

Meta was already a major acquisition channel. Increasing spend meant buying more expensive marginal inventory, the next, less-efficient slice of audience.

If that extra spend was creating genuinely new customers, great, keep pushing. If it was mostly inflating attributed numbers without moving the business, then every extra dollar was a worse and worse deal, and the platform report would never tell me. A clean incrementality read could support or reject the case for expanding budget. That's a decision worth testing properly.

The test: a holdout, not a dashboard

The goal of the experiment was not attribution. It was causality. So the design was simple in principle and strict in practice.

1. Define the decision. Not "measure Meta." Test one specific thing: does increasing CBO spend create incremental new revenue? Design the test around the decision you actually need to make.
2. Use a holdout. A two-cell design through HAUS. One group got the spend increase. A holdout was used to estimate the counterfactual, what revenue would likely have happened without the extra spend. The holdout is the whole point. It's your control group for reality.
3. Measure business revenue, not platform conversions. The primary KPI was Shopify new revenue, not Meta-attributed conversions. You're judging the business, not the ad account's opinion of itself.
4. Evaluate the lift. Judge the budget increase on incremental revenue and efficiency. Then do it again to confirm.

What it showed

The extra spend produced a measurable +5.44% incremental new revenue lift. Real, caused, above the counterfactual. Not "attributed." Caused.

Then I ran a second spend-increase experiment, iterated from the first. It produced a stronger lift at better efficiency, even under greater spend pressure. So the budget expansion wasn't a guess dressed up in a healthy-looking ROAS. It was a decision made on causal evidence, and then re-validated.

That's the difference. Same money, but now you actually know whether it's working.

The full test design and the second-round validation are written up as a case study.

"Isn't this only for huge brands?"

The usual objection: incrementality testing sounds like something only a nine-figure brand with a measurement team can afford. It isn't, and believing that is how you keep flying blind.

You don't need a formal platform contract to start thinking incrementally. The cheapest version is a geo holdout: hold spend out of a set of regions, run it in a comparable set, and compare business revenue between them. It's not as clean as a two-cell design, but it's a real causal read, and it's within reach of almost anyone with enough scale to care about the answer.

The deeper point isn't the tool. It's the question. The moment you start asking "would this revenue have happened anyway?" instead of "what did the platform get credit for?", you make better budget decisions, with or without fancy software. The tooling can scale with your budget. The mindset should start now.

And here's the reframe on cost. An incrementality test feels expensive, because you're deliberately not spending in the holdout. But that isn't a cost. It's the price of knowing whether the rest of your spend is real. Guessing is far more expensive. It just hides the bill inside a healthy-looking ROAS.

How to use ROAS, MER, and incrementality without fooling yourself

You don't throw ROAS away. You put each tool where it belongs.

• ROAS is for in-account decisions. Which creative and audience to scale, day to day. It's fine for that.
• MER (total revenue over total spend) is for the business-level gut check. Is all your marketing, blended, actually producing growth? It catches what per-channel ROAS hides.
• Incrementality (holdouts, geo-lift) is for the channel budget question. Is this specific spend causing new revenue? Neither ROAS nor MER can answer that. Only a holdout or a geo test can.

Most people try to make ROAS do all three jobs. It can only do the first one. That's the whole reason budgets get set on fiction.

The takeaway

Before you scale a channel on the strength of its ROAS, ask the harder question: would this revenue have happened anyway?

The only honest way to answer it is a holdout. Define the decision, hold a group back, measure real business revenue against the counterfactual, and judge the spend on what it caused, not what it got credit for.

Do that once on a channel you thought you understood, and I promise the number will surprise you. It's the difference between managing an ad account and actually knowing whether your marketing works.

Related: once you trust the read, scaling the spend without breaking delivery is the next problem.