The Knowledge Illusion part 2

This post is on the book The Knowledge Illusion by Steven Sloman (cognitive scientist and professor at Brown University) and Phillip Fernbach (cognitive scientist and professor of marketing at Colorado's Leeds School of business).

This post is the second in a series of sixteen that address The Knowledge Illusion and unless otherwise noted all quotes are from The Knowledge Illusion. I recommend reading all sixteen posts in order.

I have written on numerous other books on psychology, social psychology, critical thinking, cognitive dissonance theory and related topics already but discovered this one and feel it plays a complimentary and very needed role. It helps to explain a huge number of "hows" and "whys" regarding the other subjects I mentioned, all of the subjects.

The authors in the introduction take on the origin of cognitive science and explain how in the 1950s how we think and do actions was explored.

The authors have spent years researching cognitive science and remarked "We have seen directly that the history of cognitive science has not been a steady march toward a conception of how the human mind is capable of amazing feats. Rather, a good chunk of what cognitive science has taught us over the years is what individual humans can't do - what our limitations are." (Page 4)

 "The darker side of cognitive science is a series of revelations that human capacity is not all that it seems, that most people are highly constrained in how they work and what they can achieve. There are severe limits on how much information a person can process (that's why we often forget someone's name seconds after being introduced). People often lack skills that seem basic, like evaluating how risky an action is, and it's not clear they can be learned (hence many of us - authors included - are absurdly scared of flying, one of the safest modes of transports available). Perhaps most important, individual knowledge is remarkably shallow, only scratching the true surface of the complexity of the world, and yet we often don't realize how little we understand." (Page 4)

 "The result is that we are often overconfident, sure we are right about things we know little about." (Page 4)

Now that is a lot to take in and leads to an obvious question which people have given me repeatedly when I write about the limitations of the human mind - how do we know so much and have so much technology and art and science if we are so limited ? The achievements of humanity and the confidence an individual can have in their personal knowledge seem to contradict this - but upon closer examination they don't.

The authors explain one part "The human mind is not like a desktop computer, designed to hold reams of information. The mind is a flexible problem solver that evolved to extract only the most useful information to guide decisions in new situations. As a consequence, individuals store very little detailed information about the world in their heads. In that sense, people are like bees and society a beehive: Our intelligence resides not in individual brains but in the collective mind. To function, individuals rely not only on knowledge stored within our skulls but also on knowledge stored elsewhere: in our bodies, in the environment and especially in other people. When you put it all together, human thought is incredibly impressive. But it is a product of a community, not of any individual alone." (Page 5)


The authors described the Castle Bravo nuclear weapons tests of 1954 and how they much like the Manhattan project before required thousands of people including engineers, physicists, doctors, nurses and on on. No one person understood it all.

As a useful exercise the authors explain that modern planes and cars are too complicated for most of us to understand. I confess I don't know how they work. Not even a little bit. I know that they exist and work but have no clue how.

The authors then demonstrate their point further by asking if we as a reader know how modern toilets work. I again confess I don't know how. They even have a few paragraphs to explain it but they leave out so much that I still don't actually know how a toilet works even after reading the answer.

They go on to explain a complete understanding of toilets would include a lot more and get into things like why people buy certain toilets and economics and on and on.

They comment "Nobody could be a master of every facet of even a single thing. Even the simplest objects require complex webs of knowledge to manufacture and use." (Page 8)

Then they contrast really complicated stuff like bacteria, trees, hurricanes, love and reproduction. They mention how most of us can't tell how a coffeemaker works or how glue holds paper together or how a focus on a camera works.

If we look around a modern workplace or home there are hundreds of things we usually do not understand. Computers and lights and machines and architecture and on and on.

 "Our point is not that people are ignorant. It's that people are more ignorant than they think they are. We all suffer, to a greater or lesser extent, from an illusion of understanding, an illusion that we understand how things work when in fact our understanding is meager." (Page 8)

 "We wager that, except for a few areas that you've developed expertise in, your level of knowledge about the causal mechanisms that control not only devices, but the mechanisms that determine how events begin, how they unfold, and how one event leads to another is relatively shallow." (Page 9)

 "We can't possibly understand everything, and the sane among us don't even try. We rely on abstract knowledge, vague and unanalyzed. We've all seen the exceptions - people who cherish detail and love to talk about it at great length, sometimes in fascinating ways. And we all have domains in which we are experts, in which we know a lot in exquisite detail. But on most subjects, we connect only abstract bits of information, and what we know is little more than a feeling of understanding we can't really unpack. In fact, most knowledge is little more than a bunch of associations, high-level links between objects or people that aren't broken down into detailed stories.

So why don't we realize the depth of our ignorance? Why do we think we understand things deeply, that we have systemic webs of knowledge that make sense of everything, when the reality is so different? Why do we live in an illusion of understanding?" (Page 10)

These are big questions, profound questions. The authors believe thinking evolved to help us with action. Lots of living animals do various actions and have likely been doing actions for far longer than thought has existed.

 "Thought allows us to select from among a set of possible actions by predicting the effects of each action and by imagining how the world would be if we had taken different actions in the past. " (Page 11)

 "We will see that humans specialize in reasoning about how the world works, about causality. Predicting the effects of action requires reasoning about how causes produce effects, and figuring out why something happened requires reasoning about which causes are likely to have produced an effect. This is what the mind is designed to do. Whether we are thinking about physical objects, social systems, our pet dog - whatever - our expertise is in determining how actions and other causes produce effects. We know that kicking a ball will send it flying, but kicking a dog will cause pain. Our thought processes, our language, and our emotions are all designed to engage causal reasoning to help us act in reasonable ways.

This makes human ignorance all the more surprising. If causality is so critical to selecting the best actions, why do individuals have so little detailed knowledge about how the world works? It's because thought is masterful at extracting only what is needed and filtering out everything else. " (Page 11 - Page 12)

"Your causal understanding is limited to only what you need to know: how to make the thing work (with any luck you've mastered that)." (Page 12)

So the authors see us as individuals as being very limited in our knowledge and usually very unaware of how little we each really know. They see thinking as a tool to help us carry out actions. If we are good at picking which actions to do and do them we have a better chance at surviving in evolutionary terms. So a function of thinking is defining the world well enough to accurately enough choose actions and carry out actions that are better for survival than other options. Thought is designed to predict possible futures and work towards more desirable ones that can be achieved regarding survival.

The enormous amount of the world we don't know or understand is treated mostly as irrelevant to making the decisions required for action, especially if we have enough knowledge to make decisions.

This is a profound proposal - we don't need the irrelevant and treat much of what we don't know as irrelevant. It is an amazing revelation to me that we as individuals know so very little because we focus on just knowing enough to get by. I know just enough about my car to drive it in good conditions and if it is in good order. I know just enough about my job to do most of what I need to but sometimes have to ask questions. I know just enough about the vast majority of things that I deal with to engage with them on a very superficial level. Wow.