As marketers, we think we’re immune to data manipulation and…storytelling. But we’re not! So here are five data manipulation “dark patterns” to watch out for.
This content was originally published in the No Best Practices newsletter on 10.13.2024.
“More lies have been written in Excel than in Word.” – Marilyn Monroe
But seriously–it’s easy to manipulate numbers. As marketers, we think we’re immune to this kind of trickery. But let me tell you from experience: we are not.
A few weeks ago I asked my X followers: what are some of the most misunderstood and/or abused KPIs in eCommerce? I had a few of my own opinions, which were further confirmed by the post.
I’m going to share 5 KPIs and metrics that are most frequently abused in eCommerce, so you don’t get fooled in your next marketing meeting or vendor pitch.
#1: ROAS
This was the most frequently shared answer in the thread, and it was the first KPI that came to my mind when I developed the idea for this edition of the newsletter.
ROAS doesn’t measure profitability. It is simply revenue divided by ad spend. ROAS typically improves when a brand runs sales or promotions. But the variable costs associated with the order remain the same. For that reason, higher ROAS can erode profitability.
ROAS does not reflect the relative ease or difficulty of converting users in the channel. Converting new customers is always going to be more expensive than remarketing to existing customers. But you still need new customers to grow.
When a business optimizes for ROAS without keeping these dynamics in mind, they typically wind up over-indexing on remarketing and promotions. Do this long enough and the business stops growing.
I wrote about this in my “how to fail at eCom” series here and here.
#2: Conversion Rate
Another frequently shared answer in the thread. I consider Conversion Rate a “temperature taking” metric, not a “north star” metric.
Conversion rate is useful for understanding how the performance of an individual campaign, channel or marketing activity is changing over time. But you shouldn’t compare CVRs between campaigns as a way to assess their relative value or performance.
Your conversion rate on an email campaign targeting VIPs is always going to be higher than the conversion rate on your email Welcome Series that targets prospects. Does that mean one campaign is “better” than the other? No. They’re targeting different audiences.
Sitewide conversion rate is another boondoggle. It’s going to vary widely for a number of reasons: seasonality, merchandise adoption, the traffic mix you’re driving to the site. If you analyze and optimize sitewide CVR independent of this context, you’re going to be chasing your tail.
When you’re analyzing and comparing conversion rates, always ask yourself: are the audience sources in these two samples comparable?
#3: Lifetime Value (LTV)
If you want to see how LTV gets abused, go check the lifetime share price history of Allbirds, Casper (oops, you can’t, it went private) or other “DTC darlings” that went public.
The VC-supported thesis behind these companies: we can acquire new customers at a loss because they will come back and spend with us over the next 90 days, 6 months, etc.
One major problem with that approach: when it comes to LTV, the past is not a good predictor of future behavior. The faster a brand scales paid acquisition, the more likely they are to access low-context customers. These customers are much more likely to be “one and done” impulse shoppers.
Another way LTV is abused: when marketers fail to factor in the costs of retention, or fail to look at LTV on the contribution margin level.
Tons of marketers and SaaS-sters will push for promotions and loyalty programs as levers for increasing LTV. But they fail to account for the margin hit and operational costs these programs produce. And they rarely, if ever, run testing to determine if these programs drive incremental sales.
#4: Attribution
If there is one thing DTC marketers love to do, it’s debate attribution (while secretly being sponsored by attribution vendors).
And if there is one thing some SaaS and martech providers love to do, it’s abuse attribution to make their products look more successful than they actually are.
When you use last-click attribution, the channel or activity closest to the sale will always appear to perform the best. Typically, those are channels that did very little to influence the actual conversion.
Branded search crushes last click, but what marketing activity got your brand name on the consumer’s mind in the first place?
Another slimy attribution trick: using really broad time windows to claim credit for sales the marketing channel didn’t influence.
No, your email and SMS provider should NOT claim credit for all the sales your subscribers made within 30 days of receiving an email. No, your direct mail vendor should NOT claim credit for all the sales the mailed audience made within 30 days of the campaign distribution.
Incrementality is the gold standard in “attribution” modeling. It isn’t realistic (or necessary) to test the incrementality of every single marketing activity you run…as long as you’re using common sense. Insert bell curve meme here.
#5: Anything Related To “Testing”
CRO is a swamp of grift and statistical misunderstanding. And I’d say that only ~50% of that grift is purposeful. The other 50% is people who studied the discipline without a solid understanding of statistics.
You cannot run an A/B test for 5 days and declare a winner. You cannot declare an A/B test a winner after 100 conversions. You cannot extrapolate the lift from two weeks of testing and then say “this site optimization will drive $X million in lift this year.”
Shane Rostad is the GOAT of honest, stats-informed CRO. Go check out his appearance on the Marketing Operators podcast for a full rundown of the right way to run tests on your site.
Bonus: Percentages (%) Offered Without Context
If you work in eCommerce on the brand side, you’ve probably received a cold email pitch like this:
“Please demo my SaaS bro. We can 5x your ROAS bro. We did it for [insert brand you’ve never heard of]. Bro. Please book a meeting. My family is dying.”
An amateur (or a desperate entrepreneur) will immediately zoom in on that “5x your ROAS” line and smack the “book a demo” button.
A seasoned pro will immediately ask him or her self “5x off of what baseline?” Hint: if the baseline isn’t mentioned, that is usually because the baseline is small.
If a metric appears too good to be true, that’s probably because it’s lacking the context that would render it much, much less impressive.