Dug Campbell

Optimism and The Base Rate Fallacy

Some people get a reputation for optimism. Others are labelled as pessimists. And it’s often confused me because I’ve had both seemingly contradictory criticisms directed at me.

Whether the vast body of research into the subject agrees with me or not, it seems that the specific label that you apply to another can only be subjective. Whether you view someone as optimistic or pessimistic depends on a huge range of factors. What specific area of their life are you commenting on? Are you comparing their attitude to a cohort who are living similar lives – or are you simply comparing them to you?

And are the two entirely unrelated in any event? Just think of the advice: “Hope for the best but prepare for the worst”. Or read any biography of a successful repeat entrepreneur: the story’s usually pretty clear, a tale of someone who takes chances, sometimes huge, but at the same time works consistently on minimising any downside risk. Or the best investment advice: think of a Buffett/Munger warning against anything that doing anything that could potentially totally wipe you out.

When it comes to the question of why people provide me with different assessments of my character, I often wonder how much my legal training plays a role. I might not have been the best lawyer in the world (to say the least) – but I did spend close to two decades (combining law school, training and practice) being paid to think of the risks. Whether that was a good use of my time is another post altogether…

All of which leads me to a tangental point: how often are pessimists simply people who hold more information? It’s nothing to do with intelligence. Or even conscientiousness. It’s just a question of ‘being realistic’.

Daniel Kahneman speaks of something called the ‘Base Rate Fallacy‘ in his amazing book ‘Thinking Fast, and Slow’. It’s another of those inbuilt, unconscious biases that we all, to a greater or lesser extent, fall prey to. There’s a great post on Albert Wenger’s Continuations blog that explains the issue:-

“The classic example… is guessing which job someone has based on a description of their characteristics: “Steven wears glasses and has a meek demeanour. Is Steven more likely to be a librarian or a truck driver?”

“Our story telling brain wants to jump to the conclusion that Steven seems to fit our stereotype of a librarian much more than that of a truck driver. But in the US there are 3.5 million truck drivers and only 170 thousand librarians. So the base rate is that Steven is 20x as likely to be a truck driver than a librarian! It is therefore quite unlikely that the information about glasses and his demeanour is enough to suggest that Steven is actually more likely to be a librarian.”

The fallacy is responsible for launching thousands of misleading – or just plain wrong – news headlines every year.

Could it be the case that sometimes the so-called optimist is simply blundering ahead blindly with wildly incorrect assumptions (i.e. believing Steven is a librarian)? Whereas the so-called pessimist has a stronger understanding of the reality (i.e. how many lorry drivers there actually are)? That would certainly seem to tie in with evidence shown repeatedly by Nassim Nicholas Taleb that the vast majority of people simply don’t understand the way that statistics work.

So provided it doesn’t prevent you from taking action in the first place, perhaps a healthy dose of pessimism is actually just what the doctor ordered after all.