As Rolf Dobelli puts it in his exceptional book The Art of Thinking Clearly, Confirmation Bias is the mother of all misconceptions and father of all fallacies.
If you know what it is, then I am sure you already know that you and I and everyone is a victim of confirmation bias, one way or the other.
One interesting example that explains it beautifully, although partially.
If you are told that the following paragraph summarises your personality, how would you rate the accuracy of this description of you?
"You have a great need for other people to like and admire you. You have a tendency to be critical of yourself. You have a great deal of unused capacity which you have not turned to your advantage. While you have some personality weaknesses, you are generally able to compensate for them. Your sexual adjustment has presented problems for you. Disciplined and self-controlled outside, you tend to be worrisome and insecure inside. At times you have serious doubts as to whether you have made the right decision or done the right thing. You prefer a certain amount of change and variety and become dissatisfied when hemmed in by restrictions and limitations. You pride yourself as an independent thinker and do not accept others' statements without satisfactory proof. You have found it unwise to be too frank in revealing yourself to others. At times you are extroverted, affable, sociable, while at other times you are introverted, wary, reserved. Some of your aspirations tend to be pretty unrealistic. Security is one of your major goals in life"
Most of us think that its an highly accurate depiction of our own personality.
This was the experiment crafted by a psychologist Bertram Forer in 1948. While it doesn't explain the confirmation bias fully, it brings out some fundamental aspects of that.
We tend to interpret new information in such a way that aligns with what we already know or what we wish is true. Confirming to our assumptions and beliefs.
It is the most common fallacy that we fall for in our everyday life. This is so prevalent that it affects our work and personal life, how we interact with people, how we assess different situations and hence how we make decisions in response to those situations.
If you are involved in product management in any role, you already know how hard it is. You know how subjective & ambiguous the situations can get on a day-to-day basis. Most of them involve many assumptions and those situations are extremely critical in the early stages of products. Possibly due to the existential outcomes of its impact, they tend to get very personal as well.
Most of the product managers, especially founder product managers, are highly opinionated about problems their product solve and how to solve it. It is necessary for them to be so, in order to deal with the ambiguity & unknowns, otherwise there is no starting with something and there is no product in the first place.
But, a combination of many assumptions & a high amount of conviction can pull in a lot of confirmation bias. It can cloud the thinking that goes in developing the product strategy. It can obscure our learning from every interaction - inputs from early adopters, feedback from existing customers, initial sales discussions with potential customers - that feeds into our decision making towards product strategy & other areas.
Just try to recall any recent response from your team or yourself, to the question ‘how did the meeting go with someone-interested-in-your-product’. Most of the times, not always, we tend to remember the affirmations as compared to any counter opinions. And when we remember the disconfirmations, we quickly reason those out by adding our thoughts on ‘why those are more of an exception’ or 'why its a unique situation and not a general representation’.
We all like to get positive validations and stay-course. For us, affirmations are the norms and disconfirmation are exceptions. When the confirmation bias is at play, we tend to filter out the information that is not aligning with our own thoughts, hypotheses & wishes.
From products perspective, we tend to give disproportionately higher weightage to the information that confirms our ideas about the problems & how our products solve those better than others.
There are three concerns that I think we need to be aware of and we need to counter those:
(1) Undue higher ranking of confirming inputs.
(2) Ignoring the disconfirming inputs.
(3) Realising the absence of any inputs.
All of these to be checked continuously against the value proposition of the product, which of course is further divided into different user personas level from different potential customer segments, depending on the product.
While its not possible to be fully objective in discerning these concerns, it maybe possible to counter it with a systematic approach.
I suggest a system of three things:
1.Definition / redefinition of our bias-vulnerabilities
This is the simplest one. It goes hand in hand with what we already do with our product iterations, our definition of customer pain-points, gain-areas. While we do that, identify the most ambiguous areas or areas that involves many assumptions, ones that are most vulnerable to confirmation bias. Prioritise those areas and create specific Tests i.e. the questions that needs to be answered from the future interactions with (current or potential) users or buyers of the product.
2.Thorough note taking
I find this to be the most difficult one to implement as it needs a disciplined execution.
For every such interaction, rigorously noting the answers or opinions, against those Tests. Subjectively analysing the responses against three concerning areas originating from confirmation bias. Then, objectively scoring each Test for three things, let’s say- Confirmation Score, Dis-confirmation Score and Nothing-at-all score.
Additionally note these along with the profile and/or characteristics of the user or buyer.
3.Progressive but infrequent analysis
Most important part is to then distill the information & scores captured from every interaction, to surface the true picture of what’s being ‘said’ about the product by people that matter the most.
I think it should be done at two levels, one with all interactions treated as equal and second by including information about the customer profile or characteristics. Second one can surface the analysis by 'segments' of customers.
Doing it infrequently i.e. NOT after every interaction is also important as it can remove the recent conversation bias. It could be after every 5 interactions or more, depending on stage of the product.
To be even more effective in implementing this system, I think it can help to add two more ideas
4.Designated role playing
It may help to identify an designated member of the product team, other than product manager, to do a few things:
Capture inputs from interactions,
Score those Tests objectively and
Provide a comparison between what the Analysis outcomes say and what Actions are being taken by the product team. Actions here means cross-checking product roadmaps for priorities, feature backlog, team composition etc. Basically, idea is to make it real.
5.Cultural Shift
This is very subjective and depends on how well the bias is countered in the team already.
It may help to:
Educate the team on confirmation bias, why it is important to counter it for your product and how you are trying to counter that systematically.
Use these terms in day to day interactions like 'is the confirmation bias at play here?'
Reward or recognise members who surface #2 Disconfirming information or #3 Absence of any inputs.
Lastly, it's not only about the product. Confirmation bias kicks in the most when we talk to own teams. I have realised that it can spoil a potentially interesting discussion and make it completely unproductive, especially if we are working remotely, not interacting face to face or with videos-on.
Most of us form some opinions about people we interact with on a day to day basis, maybe with an exception of people that we just met only once or twice ..or never. Actually even for those, we sometimes develop opinions based on their social media profiles that we look up before meeting them.
During our discussions we pick on points that support our preconceived notion about the person. Subconsciously, we tend to assess the merit of their arguments & suggestions through the same lens and may filter out important points being made. Those could be the most valuable or interesting ones.
One hack that works for me is just to make myself aware, every time, that I carry such notions about the team members I am meeting. I do that with a practice of just thinking and sometimes scribbling it before the call - about the specific members of the team. Somehow, just being conscious about it, removes it. It helps me to listen more & better.
How we subconsciously filter out what we are NOT looking for explicitly. This video on The Monkey Business Illusion makes that point brilliantly.
Hope you liked this one. Please send your comments & your ideas to counter confirmation bias.