“That number doesn’t look right.”

“I think we need to reframe the question.”

“That’s a disappointing result.”

Have you ever heard statements like this during your analytics and insight career? They represent the level of distrust/disdain for the results of your insights that can’t be explained by a problem with the data/analysis, because you know the model is sound…

In my experience, the reason is often this: insight consumers are not looking for an answer; they are looking for ‘proof’.

Answer vs Proof

Simply defined, an answer is “a thing that is said, written, or done as a reaction to a question, statement, or situation.” In data world, it’s the unabashed truth, the statistic, the solution to the business question.

Imagine the following conversation between an imaginary Marketing Manager and an Analytics Manager:

Marketing Manager: How much time did our product save the average customer over a given year?
Analytics Manager: Our product saved the average customer 5% of time a year.
Marketing Manager: That doesn’t sound right.
Analytics Manager: It is right. We spent a week modelling it and testing it, it’s right.
Marketing Manager: Hmmm ok, I think we need to reframe the question.
Analytics Manager: Ok, what is it you are trying to do?
Marketing Manager: Well, we know that our competitors are quoting a 20% time saving so we were hoping that we could cite a similar or higher number.
Analytics Manager: Ok so what you want to do is ‘prove’ we are as good or better than our competitors. Give me three days, and I’ll return with a viable upper limit you can quote, something like ‘Our customers see time savings of up to 25%’. Sound good?
Marketing Manager: Sounds perfect!

So, in the (highly simplified!) imaginary exchange above the Marketing Manager never really wanted the answer to the original question, they wanted proof of an already existing assumption or business need.

Lies, Damned Lies, and Statistics

Simply defined, proof means ‘evidence or argument establishing a fact or the “truth” of a statement’. In our world, however, ‘proving’ something doesn’t always explicitly align with the cold hard ‘truth’. It can sometimes align with the truth as the organisation defines it, or needs it to be.

Now, I’m aware that what I am saying here could have a myriad of implications on data culture and ethics. However, those debates are for another day, because the reality of business and most organisations is that data, analytics, and Insight are ultimately used to the organisation’s direct benefit.

If we don’t understand what our Insight consumers want to do with their insights and don’t ascertain if we are providing an answer or proof, then a tremendous amount of time is wasted. In fact, I’d be willing to wager that not knowing what our Insight Consumers truly need is another reason behind the perceived high failure rate of Data, Analytics, and Insight projects.

We need to make it clear to our Insight consumers that it is ok (within reason) to ask for ‘proof’ rather than an answer. In the ‘perfect’ organisational Data Culture, there would be no embellishment, no framing, nothing but the hard unfiltered truth, but that is not the reality we live in yet, and data driven leadership needs to accept this truth. Insights organisational value is better amplified knowing whether our Insight consumers need an answer to their question or ‘proof’ of something so that they can deliver value to the organisation. It saves time, it reduces dissatisfaction and re-work, and it increases commercial awareness and value.