Tag: Data

Posted on

Errors that kill data driven decisions?

I am testing a new social app (under NDA) and I am in a group of PMs working for different companies, someone asked this interesting question

What according to you are errors that kill data driven decisions?

What I learned in the past year since I stepped into real product management at booking.com is that there are two main errors that kill data driven decisions.

Opinions

Opinion is the opposite of data.

The problem with opinions is that sometimes it is hard to suppress your own opinion because you want to do something so badly so you either decide to ignore the data, or try to find data that supports your opinion (I call this data driven confirmation bias). 

It gets more problematic when the opinions come from someone higher up in the hierarchy (Your manager). You get into the dilemma of: Is she more experienced so probably she knows better? Will she not like it if I didn’t follow what she said?

Fortunately I saw this very few times and it never happened with me. Be careful with opinions.

Is the effect real?

We use data to validate our hypothesis. The question becomes: Is the effect of the change I am seeing through the data statistically significant and I can base a decision on? Or it is a random effect?

There will always be the probability of identifying unreal effect as statistically significant (false positive), however not questioning the significance you are seeing through the data might drive you to misleadingly taking decisions based on effects that aren’t real. Which eventually kill the data driven decisions.

Posted on

Data Driven Confirmation Bias

Two weeks ago I had a presentation at work as part of a training. The presentation meant to show a problem and my team’s proposed solution to this problem. I wanted to show at the beginning of the presentation how the problem is growing. I pulled data showing the month over month growth for the past year and a surprise was waiting for me.

It wasn’t growing. There was no pattern. Random fluctuations of ups and downs. No problem, let’s pull the data from the year before and compare the same month from the two years. Voila! We have a nice growth trend.

I just finished the book “How to lie with statistics?”. It is a nice short read about how statisticians, and politicians manipulate the way they present statistical facts to different audiences to convey a message.

What I found myself doing on this day is applying what’s in the book subconsciously because I was enthusiastic about proving my point, while what I did on this day wasn’t mentioned in the book, but I kept thinking about how to make the problem looks growing, regardless of the fact that there is no monthly trend. Someone even recommended using the cumulative numbers to display a nice growth chart.

Before this, I was telling a friend that being data driven doesn’t mean you are not biased. Most of the time you will find data to support your case (unless it is extremely illogical).

Our biases drive us to find the data that support our opinion, ignoring data that doesn’t. It is up to one’s self and to their self awareness to realize whether they are really looking for the truth, or dragged into a data driven confirmation bias.