A master craftsman will have intuitions about their craft which they won’t be able to explain to you
Supposedly data-driven decision making can lead to a passive aggressive style where disagreements are hidden by dishonest, manipulated numbers
(What follows is an excerpt from the book I’m working on, regarding advice to entrepreneurs and managers.)
Most successful product launches are overseen by someone who is deeply committed to getting that product off the ground. That person will have devoted years of their life to understanding the customer who needs and wants that particular product. The person overseeing that product launch will eventually build up dozens of important intuitions which can be difficult to verbalize in a language that the CEO will understand (likewise, an entrepreneur launching their own startup will often have trouble explaining to investors exactly why customers love a given product). If “data driven decision making” refers to careful studies of massive sets of user statistics, then data driven decision making can never function correctly for truly innovative new products, as there simply won’t be enough data for the CEO to study.
This point is subtle and easily misunderstood: entrepreneurs and product managers will spend a great deal of time talking to customers and thus gaining data and insights into what the customers actually want. But this kind of tacit knowledge is not what is typically meant when CEOs talk about data driven decision making.
A great designer will crave data and try to get as much as possible: ethnographic studies, customer interviews, formal A/B testing at sufficient scale to rule out statistical anomalies, simple multiple-choice surveys, heat-map studies to show usage patterns, and so much more. A master of user experience (UX) is always looking for new tools and methods to help them get more insights about the customer. And then this expert will combine the data with everything they've learned over the course of their career, and they will synthesize all of it to a few intuitions that will guide their work. They may have trouble verbalizing the process by which their intuitions are derived, but the correctness of those intuitions should be deducible from the final result: are customers happy?
In other words, the correct place for data-driven decision making to happen is inside the mind of the talented experts that you have hired. Such people use a lot of data to form their opinions. Once they give you their opinion, there is no need to re-hash the data behind that opinion. Having a large, raucous meeting where people quote statistics at each other can only undermine, and never enhance, the expertise of the professionals that you've hired.
The more emphasis given to certain metrics, the more that employees will manipulate that metric
As soon as the leadership agrees that certain metrics are the crucial ones for measuring success, then there is the risk that employees will manage the metric itself, instead of the underlying reality. As Page Laubheimer and Kate Moran have written:
One of the most misquoted sayings in business is “if you can’t measure it, you can’t manage it”. This statement (and its variations) is often meant to say that, to improve something, we need a precise metric that captures it and that should be tracked in order to understand if our efforts to improve it are effective.
It is interesting that this “quote” is actually the complete opposite of the original, which was:
“It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” — W. Edwards Deming (The New Economics).
This difference between the original and the commonly used version highlights why it’s dangerous to rely on a single metric to assess how well a business is performing: that one metric can be manipulated in ways unrelated to what it is supposed to measure. This is the phenomenon described by Campbell’s law.
Campbell’s law states that the more important a metric is in social decision making, the more likely it is to be manipulated.
In other words, when a single metric is used to determine success or failure, human beings are likely to try to optimize their behavior to improve that metric — sometimes with ridiculous or dangerous consequences. People manage the metric, rather than using the metric to help manage the underlying issue of interest.
...A metric is a signal that reflects the goal or outcome you’re seeking, but is not the full picture. There is no one truly absolute metric that completely captures a real-world behavior or phenomenon. Just because a measure is quantitative, does not mean that the data collected will be free from bias.
https://www.nngroup.com/articles/campbells-law/
All companies have always valued data, since the beginning of time
As far as I know, there has never been a company that said “We want the worst informed people to make the decisions” so in a sense all companies have always valued data. But they didn’t make a fetish out of it. They simply expected people to be well informed, and to make intelligent arguments, based on what they knew. That would have been true at General Motors in 1950 at least as much as at Google in 2015. Indeed, this much has probably been true at most companies for centuries. When management says that the company is going to be “data driven” they are implicitly asking for a particular type of interaction to happen in meetings, an elaborate dance where people pantomime the objectivity of scientists while still relying on the implicit knowledge they’ve gained over the many years of their career.