The Messy Truth About Measuring ROI#
It’s like trying to get those tangled up earphones out of your pocket.
The Good Old Days (Which Weren’t That Good)
It was easy before the digital age:
TV ads—Keep an eye on sales patterns and hope they match up.
Print ads: Add a coupon code and keep track of how many people use it.
Billboard—Hope for those eyes
Everyone was measured at the “cohort or channel level.” For example, “We spent $100,000 on TV in Q3 and sales went up 15%.” Did the advertising on TV do it? What are the trends for each season? No one knew.
The New Way#
We have a lot more information now:
Tracking at the pixel level - Dashboards that update in real time - Attribution models based on machine learning
Graphs that can be used on more than one device
Giving credit for more than one touch
This is excellent, but it just means that people are now arguing more over what is really making sales happen.
The Real Problems
Correlation does not equal causation (the eternal problem)
People who viewed your ad and then bought your product didn’t necessarily buy it because of the ad. They could have: Were already intending to buy— Saw a post on social media that was real— A friend told me about this, although I would have found it anyhow by searching on Google.
It’s hard to identify the difference between the “incremental impact” (what the ad actually added) and the “baseline impact” (what would have happened anyhow). Incrementality tests are useful, but they are expensive and most businesses don’t do them.
Who Gets the Credit? Multi-Touch Attribution
This is what a modern customer’s journey looks like:
See a commercial on YouTube 2. Don’t pay attention to it 3. A week later, see an ad on Instagram 4. Search for the item on Google 5. Click on an ad for a search engine 6. Go without buying
Go to a news site that has a retargeting ad
Come back straight away and buy
Which point of contact “made” the sale? The first one that got people talking? The last one before you buy it? A mixture with some weight?
Attribution models try to figure this out:
Last-click: The person who clicked on them last gets all the credit (not very nice, but frequent)
initial-click: The initial interaction receives credit, and everything else is ignored.
“Linear”: Give all touchpoints the same amount of credit (fair but not right)
Time-decay: Newer touches receive more credit (better but random)
Data-driven: Let machine learning figure things out (sounds smart, but it’s a black box)
Chaos across all channels
It’s hard to make sense of it all if you’re a huge company running commercials on TV, the internet, podcasts, influencer relationships, and retail media networks all at the same time.
Different systems use different approaches to measure:
TV gets its ratings from Nielsen and information from set-top boxes. - Digital employs pixels and cookies.
Surveys and promo vouchers are used in podcasts
Influencers use unique links to maintain track
Retail media is held in walled gardens, like Amazon
Most marketing teams don’t have the data technical capabilities or the money to pay for integration that you need to get a unified perspective.
Privacy is killing third-party tracking
Do you remember when cookies let you stalk people online? Yes, that’s dying thankfully:
GDPR in Europe needs permission - CCPA in California allows individuals opt out - Safari disables third-party cookies by default - Chrome is steadily getting rid of them - iOS made IDFA opt-in, but no one does.
Measurement went from “we can see everything” to “we can see 40% of users and guess about the rest” all at once.
The platform data is only meaningful to the platform itself.
When ad networks say their advertising led to purchases, it’s hard to identify who really made the transactions.
Self-attribution models are used by platforms to give themselves the maximum credit. It’s like asking the car dealer if the price they gave you is fair. Yes, the answer is always yes.
Best Practices (If You Want to Get Useful Results)
Before you start, make plans for how to succeed.
Don’t launch a campaign and then try to find out what to track. Choose ahead of time:
What do you want to get out of this? (Do you know the brand? Sales? Leads?)
What is the key performance indicator (KPI)? (Conversions? Impressions? Money?)
What is the goal? (10% increase? $5 CPA? 3x ROAS?)
If you don’t know what success looks like, you’ll never know if you have it.
Invest in the correct infrastructure
This indicates you need to double-check your website’s tags to make sure they are proper.
A data warehouse that combines information about ad spending and sales - Engineers that can genuinely integrate datasets appropriately - A single source of truth for metrics (not five distinct dashboards showing five different values)
I’ve seen companies spend millions on marketing while fighting over whose spreadsheet is correct. Don’t be that kind of business.
Get to know people both online and in person
If you run digital ads yet 80% of your sales occurs in physical stores, you need to develop a means to keep track of store visits and purchases. Options:
Store visit attribution (Google and Facebook can accomplish this with data on where people are)
Matching CRM (using customer IDs to connect online marketing to offline sales)
Promo codes (they’re old school, but they work)
Test and control groups (the cleanest but needs discipline)
Understand how AI will effect you (or not)
AI and machine learning can help you uncover trends, make predictions, and make things better. But they can’t fix:
Wrongly setting up tracking
Attribution models that are not fair - Data that is not complete - Not being able to tell the difference between correlation and causation
AI isn’t magic. AI just makes your lousy data worse.
There are other things that matter besides sales. You need to measure now and then:
Brand lift (Do more people know about us?)
“Thought” (Are we one of the things they can buy?)
Feelings (Do they like us or think we’re a bother?)
Voice share (How much of the conversation do we own?)
These “soft” metrics are tougher to quantify, but they are typically more helpful than fretting about conversions from the last click.
The Truth That Makes You Feel Bad
There is no such thing as a flawless measurement.
Not trying to be flawless, but to make “good enough” decisions. You want to find out how much that is:
appropriate in the appropriate way - Reliable over time - Trustworthy enough to act on - Better than guessing
Companies who are too focused on attribution models and dispute about how to do things miss out on greater opportunities, like stopping campaigns that are clearly not working or putting more effort into channels that are clearly succeeding.