How intuition makes us better engineers
“The intuitive mind is a sacred gift and the rational mind is a faithful servant. We have created a society that honors the servant and has forgotten the gift.”
What is intuition and why is it important?
Intuition is a feeling of knowing that comes without consciously thinking.
It is a snap judgment about something or a situation. You know it feels right but can’t exactly explain why.
Intuitions pops out of nowhere. It is the result from subconscious thought processes too quick for the conscious mind to understand.
In the past few decades, intuition has received increasing interest from scientists. And what they found is that intuition does exist.
Scientist and Nobel Prize winner Daniel Kahneman wrote the book Thinking, Fast and Slow.
The book explains how the brain has two different operating systems; system one and system two.
System one is fast thinking and often dictates our subconscious way of operating. It is where feelings and intuition dominates.
System two is our slower, more analytical way of operating.
System one, where intuition dominates, was proved to know the right answer in a wide range of situations long before system two.
E.g., people who had to make a quick, intuitive decision about their car purchase later found out to be satisfied with their purchase 60% of the time. Meanwhile, buyers who had a lot of time to think carefully about all the different options were only happy 25% of the time.
Our intuition is based on emotion, and it derives from things that have already happened to us.
Our intuition remembers, and the more we use it, the more accurate it can become.
According to Robin Hogarth, intuition offers the ability to respond with confidence in our knowledge and decision making with little apparent effort, and without conscious awareness.
Robin Hogarth argues that such decisions rely heavily on learning, in particular learning specific to the environment in which we operate. Moreover, intuition may have emotional origins such as fear.
Intuition is essentially a shortcut that helps us make quick decisions based on minimal information. It’s the ability to detect patterns quickly and without thinking.
This is why people who devote years to their work, skills, or talent become intuitive about it. Intuitive people gain the ability to make the right decisions with minimal information.
While it may take a novice a large amount of information to make a particular conclusion, an expert will only need half of it.
Did you know that before becoming an active-duty firefighter, one needs to spend approximately 600 hours in training? And that’s just the beginning. After that, some firefighters — according to reports — spend over 80% of their active-duty time in training.
That’s because when firefighters go under live-fire conditions, they need to have an intuition for the fire they are fighting against. To acquire that lifesaving intuition, they need to collect an incredible amount of training hours.
Like the old adage says, practice makes perfect.
“They seem to get inside the head of the fire, sort of like a Dr. Phil for a fire”
What is even more interesting, is that both intuition and creativity appear to be correlated, especially during the early stages of the creative process, including idea generation and evaluation stages.
Since engineering is inherently creative, developing intuition is very important. If we weren’t creative, how could we ever develop new technologies and solve problems?
Finally, engineers often have to deal with ambiguity. There are always, always, to many tasks to work on, and sometimes we don’t have all the data required to make informed decisions. This is were intuition can help. Having a hunch for what might be important, and what you should be focusing on is critical. I see this everyday at work.
While intuition usually helps us, it may lead to bias and prejudice in our decision making that can be countered by rational thinking.
By deliberately developing our intuition, we can become more accurate, more creative, but also less biased.
To develop our intuition, we need to primarily focus on three important aspects:
1 — Practicing a wide range of skills
2 — Listening to anecdotes
3 —Understanding and fighting biases
Let’s dive into each of these.
1 — Practicing a wide range of skills
In his book Range, David Epstein explains:
“In a wicked world, relying upon experience from a single domain is not only limiting, it can be disastrous.”
He continues:
“Modern work demands knowledge transfer: the ability to apply knowledge to new situations and different domains. Our most fundamental thought processes have changed to accommodate increasing complexity and the need to derive new patterns rather than rely only on familiar ones.”
And I could not agree more.
The most successful engineers I have ever met have been those with very diverse backgrounds and roles.
They gained skills from a wide range of experiences. By doing so, they also acquired the knowledge required to learn and derive new patterns.
These patterns are what feeds intuition.
David Epstein calls that Interleaving and it refers to the mixing up of the types of problems you train on so that you don’t know what kind of situation is coming next.
So, instead of practicing, or repeating, the same procedure over-and-over again, we’re forced to understand the structure of the problem.
By doing so, we are developing our own pattern recognition algorithm, trained from our own experience.
The more diverse our experience is, the more accurate our algorithm will be.
2 — Listening to Anecdotes
Another important aspect for increasing awareness and developing our intuition, is engaging with people, asking questions, and especially listening to anecdotes.
We do that a lot at Amazon.
While engineers typically rely on data and metrics to make decisions, we also spend a large amount of our time talking to customers to get anecdotes.
Jeff Bezos — founder and former CEO of Amazon — explained why he spend a lot of time reading customer emails and forwarding them to the appropriate team.
“I’m a big fan of anecdotes in business. Often, the customer anecdotes are more insightful than data. […] I’ve noticed that when the anecdotes and the metrics disagree, the anecdotes are usually right”.
Science is very unlikely to follow anecdotal observation. And yet, medical discoveries based on anecdotal observations use to be common place.
For example, the toothbrush was developed from A.T. Pitts’ observations.
Of course, anecdotal observations should not be used alone, but they can provide the stimuli for potentially important ideas.
Anecdotes feed the creative mind.
How to get more anecdotes?
My favorite question to ask engineers, and get anecdotes, has become:
“What are you worrying about?”
That is a very powerful question — because often, if engineers worry, it is intuition-based — and it is serious.
In fact, this question is so serious that — “What are you worrying about?” — is part of the Operational Readiness Review (ORR) we do at Amazon before launching a new service.
Listen to these anecdotes. If several team members worry about the same thing, rank it up and look into it.
All these years, I have learned that listening to intuition has never been a waste of time. Quit the opposite.
But— there is always a but.
Have you heard the saying?
“Dogs Not Barking”
The reference is to a Sherlock Holmes story by Sir Arthur Conan Doyle called The Adventure Of Silver Blaze.
At one point, inspector Gregory asks Holmes:
Gregory: Is there any point to which you would wish to draw my attention?
Holmes: To the curious incident of the dog in the night-time.
Gregory: The dog did nothing in the night-time.
Holmes: That was the curious incident
The lesson of ‘dogs not barking’ is to pay attention to what isn’t there, not just what is.
Absence is just as important and just as telling as presence.
Paying attention to absence requires intentional focus.
So also ask:
“What are you not worrying about?”
If you think about it, increasing our awareness is the higher form of increasing monitoring and observability, simply not limited to software engineering.
Now let’s talk about biases.
3- Fighting biases
Interestingly — intuition often gets confused with bias.
Intuitions like:
“It never happened before, it shouldn’t fail”
This is where intuition gets confused with bias.
Intuition is our genius insight, but we shouldn’t blindly follow it.
How to distinguish bias from intuition?
When you hear overly optimistic intuition, put it up to testing and validation as much as the rest — even more. Look for evidence that the intuition might be wrong.
Ask yourself if it looks like something you’ve seen before. If it isn’t, it’s a warning sign that it might not be intuition but bias thinking.
In my experience, the two most common biases with “intuitive” engineers are the confirmation bias and the sunk cost fallacy.
The confirmation bias is the tendency to search for, interpret, favor, and recall information that confirms or supports your prior personal beliefs or values.
On the other hand, the sunk cost fallacy is the tendency to believe that current investments justify more expenses.
Experimenting on a system — using chaos engineering for example— helps fights and challenge our assumptions and biases because it forces us to confront our intuition with the reality and data.
But be careful, biases are tricky and even dangerous because of the different factors that influence them. This is particularly true when people are deeply invested in a specific technology.
So remember, handle biases with care.
Developers love their code, architect love their architecture, and designers love their designs, it is their baby.
Final words
As engineers, we are often told that intuition is something we shouldn’t rely on. We are told that Data is the new gold from which innovation, products, and services will emerge.
While it might be true, as engineers, we need expertise, creativity, and intuition to begin translating data into insights.
Today, the average person consumes approximately 50GB of data every day.
How do we deal with the complexity derived from this abundance of information?
How do we cut through it all and make essential decisions in life and business?
Without expertise, creativity, and intuition, we are often stuck between the too many options presented to us, not knowing where to start.
As engineers, we need to widen our perspective, read non-technology related articles and books.
We need to actively search for information that is not related to our main field of interest. We need to realize that we are typically fed by algorithms that reinforce our interest. This algorithm typically create an echo chamber in which our ideas are magnified.
As engineer, we need to be humble, self-critical, and curious about our intuition. While it may be our greatest ally, it may be wrong.
If we are too firmly rooted in our own opinion, it limits our ability to realize that another idea might be as important or more important as the one we are currently holding.
Having a false perception of knowledge can only limit our curiosity because we think we have the answer.
“The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge.”