Data and testing are critical for success in marketing and advertising. We’ve all heard it, we all know it. “Data is the new oil.” “Data-driven marketing is the future.” “Data is better than opinion.” Maybe all true, but it needs to be said: Data is not a strategy.
You’re not going to test your way to success. You’re not going to resegment Facebook audiences until you find a silver bullet. You’re not going to A/B test your way to the perfect tagline. Sorry. I’m probably raining on a lot of parades right now.
Why spoil everyone’s fun?
Look, the fun is always up sooner or later anyway. Data-driven techniques – A/B testing, automated tools, segmentation, etc. – have their place, but they also have their limitations. Failing to recognize and work within those limitations always leads to pain.
How? This post is broken into two parts. In part one, we break down some of the key limitations of data-driven marketing techniques and the tools available to help measure performance. Part two will lay out some best practices for how you *should* use data and testing.
Limitations of data-driven approaches
More ways to fail
For any aspect of a media plan you want to test – the likelihood is there are more ways to fail than there are to succeed.
There are more Facebook audiences that will *not* resonate than ones that will. There are more Google Ads copy options that will *lower* CTR than raise it. There are more randomly-paired image/copy pairs that will *not* fit well together than will.
This is a key reason why trying to test your way to success across a whole media plan is likely to fail. The odds are, at any given time, more of your ad dollars will be spent suboptimally than optimally.
Complexity
Marketing is complex – in the sense that marketing campaigns are part of complex systems. Small changes to inputs can create large and unpredictable changes in outcomes.
Tests/optimizations on small parts of the system (i.e. coupon redemption) can have unpredictable results on the overall system (i.e. brand perception and total sales).
As a result, tactical tests don’t provide strategic learnings*. Failing to consider the big picture can have big consequences.
*Not never – but not often
Human behavior
A major driver of complexity in marketing is human behavior. People rarely actually work according to nice linear functions like x clicks = y dollars.
Just a couple examples include familiarity bias and narrative cognition.
Familiarity bias means that people have a tendency to view familiar things (including brands and ads) more favorably. Someone who has seen your ad six times may respond to a new variation very differently from someone who has never see an ad.
Narrative cognition means that people often seem to think in terms of stories. They tell themselves stories mentally about their lives and experiences, including those with brands. Changing one touchpoint in a much bigger story may not change that much.
There are many other cognitive biases that can drive people to interact with marketing in unexpected ways. These are tough to account and even tougher to control for.
Expense
Testing done properly is expensive. It requires significant amounts of staff or agency time to design, run, and analyze. The bigger and more important the test, the more variables need to be controlled for and the more time must be dedicated to test design.
Gathering enough observations to make statistically sound observations can also require large numbers of impressions, clicks, or site visits – and therefore large media budgets to drive these actions.
For example, Croud’s test planning tools – built on popular A/B statistics testing methods – often show that testing for small variations in audience conversion behavior can require 5,000-10,000 visits to generate concrete learnings. Multiply that by any common CPC (cost per click) and you can understand how testing can quickly become expensive.
Random amplification
If the test isn’t carefully designed or carried through to statistical significance, another issue arises.
Test “learnings” are taken and the buy is changed or optimized accordingly. However, the learnings may have been the result of random noise, so the the resulting “optimizations” are only amplifying random noise and may even be hurting the campaigns.
Data isn’t infallible
For any test based on data – no matter how thorough the design or how many observations accrued – the results will only be as good as the data.
Pixel issues, multiple conversion types, shoddy third-party data segments, low cookie match rates, and iffy attribution models can all wreak havoc with data-driven techniques.
How can you measure whether performance improved if nobody is actually sure what performance is or how it’s being measured?
Scale
Now, imagine that you’ve handled all these areas discussed above. You’ve designed an excellent test, funded it fully, and have a high degree of confidence in the measurement. You’ve discovered an audience segment that performs at double the rate of your basic target demo, but… it’s only 20,000 people.
If your company goal is to move another two million units, that probably doesn’t do you much good does it?
But many tests are designed this way. They test tiny audience segments, messaging for low-volume keywords, or low population geos. Even when the results are spectacular, they often don’t close the gap between reality and company goals.
From yet another angle, tactical tests don’t drive strategic learnings.
Tools
Can’t you use tools to overcome these challenges? Major vendors like Facebook and Google provide all kinds of interesting testing tools and automated, data-driven buying methods; aren’t they a solution to these problems?
Sort of. While these tools have immense potential, most are early in their life cycles and they are also subject to their own special limitations.
Poor multipolar optimization
Many tools can optimize very effectively for single KPIs. However, very few have yet developed the capabilities to fully consider constraints and secondary goals in their optimization.
For example, an automated bidding tool may be able to hit a very low target CPA, but may not be able to do it while still spending the budget.
This can easily lead to over-optimization, in which a tool has found a buying space which excels at its given KPI, but harms the overall health of the marketing plan.
Limited scope
Additionally, no existing tools have a wide enough breadth to consider the full context of a buy. They may not see spend from other vendors, ignore important site-side signals (like stock levels), or fail to consider context like offline spend.
Few tools or automated buying options can react fast enough to major shifts in the market. Failures we have seen include tools that did not ramp spend up fast enough when an effective TV campaign was in market, or tools that continued to bid at high levels after the Black Friday/Cyber Monday timeframe ended.
Control/Transparency
The tradeoffs for many tools also include a lack of control and transparency. Some tools do not allow measures such as negative keywords, site blacklists, or creative control.
These may be acceptable tradeoffs for some advertisers, but for brands which are sensitive about image or brand safety, these are serious concerns which should not be glossed over.
Limited outputs
Just as many automated tools and testing platforms limit inputs, they also limit outputs. Some tools have made big strides here recently, but in many cases, it must be assumed that using automated tools will give you far less data to work with after the buy.
There may be limited learnings on best-performing variants or audiences, but full, rich, CSV tables full of detailed histories and change logs will almost certainly not be available.
In the case of any vendor offering, their goal is to keep more money moving to their platform. They are rarely interested in enabling you spend more money or spend more effectively in other parts of your media mix.
Data is common
Finally, remember that data is common. All of your competitors have the same Google Ads keyword data, the same Facebook targeting options, the same third-party data brokers, and the same ad networks you do.
They have the options to run the same automated buying methods, the same optimizations for dynamic creative, and the same attribution models you do.
It isn’t enough to just test – everyone is testing. It isn’t enough to use data – everyone is using data. You have to test and use data *better*.
So, how do you do that? We will be sharing some best practices for how you *should* use data and testing in part two of ‘Data is not a strategy’. In the meantime, if you would like to discuss how to improve your strategy, contact us.
