I am testing a new social app (under NDA) and I am in a group of PMs working for different companies, someone asked this interesting question
What according to you are errors that kill data driven decisions?
What I learned in the past year since I stepped into real product management at booking.com is that there are two main errors that kill data driven decisions.
Opinions
Opinion is the opposite of data.
The problem with opinions is that sometimes it is hard to suppress your own opinion because you want to do something so badly so you either decide to ignore the data, or try to find data that supports your opinion (I call this data driven confirmation bias).
It gets more problematic when the opinions come from someone higher up in the hierarchy (Your manager). You get into the dilemma of: Is she more experienced so probably she knows better? Will she not like it if I didn’t follow what she said?
Fortunately I saw this very few times and it never happened with me. Be careful with opinions.
Is the effect real?
We use data to validate our hypothesis. The question becomes: Is the effect of the change I am seeing through the data statistically significant and I can base a decision on? Or it is a random effect?
There will always be the probability of identifying unreal effect as statistically significant (false positive), however not questioning the significance you are seeing through the data might drive you to misleadingly taking decisions based on effects that aren’t real. Which eventually kill the data driven decisions.