Most Marketing Mix Modelling (MMM) bake-offs I see run three vendors against two years of historical sales and pick the one with the best in-sample fit (the closest match between what the model says and what actually happened). I think this is the wrong question. In-sample fit rewards flexibility, and flexibility is cheap. Any competent vendor can fit your history. The question that matters is whether the vendor’s outputs would survive a Tuesday morning with your chief financial officer (CFO).
A worked example. We ran a bake-off for a UK retailer in 2024. Three vendors. All three reached R-squared above 0.88 on the historical window. On paid social, vendor A reported a return on investment (ROI) of 1.4, vendor B returned 2.1 and vendor C returned 4.6. The retailer was about to choose vendor C on the strength of the highest reported return – until we asked each vendor to defend their number in a 30-minute session with the CFO.
Vendor C couldn’t name the saturation point they had assumed (the level of spend beyond which the next pound returns less than it costs – where marginal ROI falls below break-even). Vendor B could, and showed the diminishing-returns curve with the break-even point flagged. Vendor A had modelled paid social as two separate channels – prospecting and retargeting – and could explain why retargeting’s ROI was lower but more defensible.
The retailer chose vendor A. Two years on, vendor A’s reads have held up. I don’t know what vendor C’s would have said. I do know the CFO wouldn’t have been able to act on them.
My recommendation: rerun your bake-off around defensibility under questioning, not historical fit. The decision will get easier.
The same defensibility test applies once the deck lands on your desk – see Three questions before you sign.
If you’re about to run a selection, or you’ve just run one and the answer feels off, drop a line to hello@themmmdoctor.com – happy to talk it through.