I deleted a database in production

“I am done. It is the end and I will be fired now.”

These were the words I kept repeating to myself. I had just deleted a production database critical to our customers. It was late in the evening, and it had been a long day. I had been trying to fix an issue that was impacting our customers. But instead of fixing it, I made it worse by mistakenly connecting to the production environment and deleting the database.

Suffice it to say neither my leadership nor our customer was happy.

Fortunately, we had a database backup, and recovering it worked more or less as expected. We lost some data, but considering I had deleted the production database, we were “happy.” It could have been a lot worse. Saying I was traumatized by what happened that day is an understatement.

I felt guilty. I felt terrible, and I felt like a failure.

After recovering our service, I called our customer and apologized for my mistake. When I hung up the phone, I broke down. It took me a long time to recover. Yet, here I am. I am happy to report that I survived. But I was lucky. Many engineers have been publicly shamed and fired for making mistakes. Instead, I stayed at my job. And I’d like to think that I thrived.

Rather than focusing on the failure, we looked at ways to prevent it from happening again. We looked at improving our tooling, culture, and mechanisms instead of concentrating on my mistake. We focused on the how instead of the what. We turned failure into a learning experience.

You can read the full story “What I learned from deleting a production database.” It includes details for tech-savvy folks.

When failure hits

Behind the fear of failure hides the worry about doing something wrong, looking foolish, or not meeting expectations. Here are a few of the fear that I, still today, often get:

• Fear of embarrassment

• Fear of letting others down

• Fear of the hard work not paying off

• Fear of not being good enough

• Fear of not knowing enough

Fear, like any emotion, has an evolutionary purpose. In particular, the fear of making mistakes remind us that we’re in a challenging situation. Of course, we all know that errors are unavoidable, and everyone makes them. The issue comes when we are blamed or shamed for our mistakes — by others or ourselves. That is when mistakes and failures become a pain. Unchecked, a painful experience can grow until it stops us from moving forward.

It may even trigger total paralysis because our brain is hardwired to avoid pain. Fear traps us since it ensure we direct all of our attention to the potential danger. That is by design so that we don’t ignore threats. It’s part of our genetic makeup and is related to our fight-or-flight response system. This response system, also called acute stress response, was first described by Walter Cannon in the 1920s. The theory behind it suggests that animals and humans react to threats with a general discharge of the sympathetic nervous system.

For early humans roaming the earth, the fight-or-flight response system was supposed to protect them from the dangers of the environment, such as the saber-toothed cat. Today, saber-toothed cats are no longer a threat, but our fight-or-flight response system hasn’t changed. And rather than focusing on physical dangers, it now focuses on our emotions.

The problem is that our body doesn’t know how to distinguish between physical and emotional dangers. Worth, because that response system now focuses on emotions, it has become harder to understand. We can’t quickly point at emotions with a finger as we can point at a saber-toothed cat,

This relatively new emotionally-focused fear avoidance mechanism is, in fact, so extreme that it can lead to depressive disorder and make us unmotivated to look for pleasure.

In 2016, a Challenger engineer — Bob Ebeling - still blamed himself for the event even though it was officially ruled out that he couldn’t do anything to prevent it. Before the launch, NASA managers had all the information and were warned against it by Ebeling and his colleagues. But they still decided to proceed.

No safety rules were broken, and no single individual was at fault. The cause of the disaster is a story of the banality of organizational life, what Diane Vaughan calls the Normalization of deviance.

Ebeling, 30 years after the explosion, still took it as his failure. Sadly, he suffered deep depression and could never lift the burden of guilt.

Perfectionism and the fear of failure

Research published in the Psychological Bulletin suggests that perfectionism might be part of the problem, especially for the younger generation.

According to a 2015 World Health Organization survey, a record number of young people worldwide suffer from depression or anxiety disorders. It turns out young people increasingly hold more perfectionist ideals for themselves when it comes to academic and professional accomplishments, their looks, and possessions. The data presented in that study suggests a worrying 14.9% increase since 2005.

Perfectionists often set themselves up for failure. They typically have unrealistically high expectations and often experience the dreadful fear that they won’t live up to those high standards, and they never feel good enough.

Unfortunately, we grow up praised for achievement and performance. We must excel in academics, sports, friendships, social media, family, and extracurricular activities. Just look at updates published by your network on your favorite social media platform. Or search for motivational quotes. You will quickly realize that being average is apparently an insult to humankind.

“All I knew is that I never wanted to be average.”

“Normal is the wrong name often used for average.”

“The average man does not want to be free. He simply wants to be safe.”

“Success and average don’t have nothing to do with each other.”

“Never settle for average.”

“Average is for losers.”

I am all for motivation, but I prefer when it doesn’t stigmatize others as failures. I am an average software developer, but I am not a loser.

No one can be perfect all the time, and probably shouldn’t even try to be. According to research, perfectionism won’t make you happier, freer, or more successful, as the authors of these quotes, probably perfectionist themselves, claim.

In The Dangers of Perfectionism, the author explains:

“Perfectionists think they have to be perfect; when they are not, they get angry. They also expect other people to be perfect — and get angry or impatient when others prove imperfect.”

Perfectionism is a trait often cultivated in our society, whether at school or the workplace.

“Perfectionism is regularly thought of by managers and employees alike as a positive trait which enables an employee to strive toward a perfect performance/product.” — Role of Perfectionism in the Workplace

Unfortunately, it gets worse for women.

Journalists Katty Kay and Claire Shipman, authors of The Confidence Code, explain that women are more likely than men to be perfectionists. Women are holding themselves back from answering a question, applying for a new job or asking for a raise until they’re absolutely 100 percent sure. As a result, women asked for a promotion only when they met 100 percent of the qualifications required for the position, while men applied even when they met only 50 percent. Women are also a quarter as likely as men to negotiate a raise.

The perfect engineer

Perfectionist tendencies are widespread among engineers.

Hear me out, though. I am not saying that every engineer is a perfectionist; simply that engineering, as a discipline, nurtures perfectionism.

It is not a new idea. Simon Winchester uses this idea to discuss the origins of technological precision and the implications of our modern obsession with it. Winchester explains that many aspects of everyday life would only exist with perfectionists.

While most of his examples are from mechanical engineering, it is not difficult to draw similarities with other fields of engineering — such as software engineering.

In a 2007 essay called “Programming can ruin your life,” the (unknown) author explains software engineers’ perfectionist issues well.

Programmers become obsessed with perfection. This is why they are constantly talking about rewrites. They cannot resist optimum solutions. Perfection requires tossing aside mediocre ideas in search of great ones. A good programmer would rather leave a problem temporarily unsolved than solve it poorly. A good solution takes into account all predictable outcomes and solves the largest number of them in the most efficient way. This mindset prevents you from writing code with limited utility and life span. While it’s a wonderful trait to have in programming, the demons of scope and efficiency will start to assert themselves on your ordinary life. You will avoid taking care of simple things because the solution is inelegant or simply feels wrong. Time to think will no doubt yield a better result, you’ll say.

Software engineers love to be remembered for the beautiful code we produce, and we love to be recognized for the intelligent choices we make. It is not uncommon for software engineers to refer to coding as art.

Paul Graham, best known for his work on the programming language Lisp, discusses that idea in his book Hackers & Painters, in which he compares hackers (software engineers) to an artist.

He explains that both artists and hackers start from a blank canvas and leave some of their personalities in their art. Both artists and hackers often strive for perfectionism.

Embracing failure

You might not define yourself as a perfectionist, but if you are an engineer, chances are you have perfectionist tendencies.

We all need to be aware of these perfectionist tendencies. They may have served us well until now but are damaging to our career and fulfillment.

I could tell you that the strategies that can help you overcome perfectionism and manage failures include; Thinking positively; Not comparing yourself to others; Creating a check-list; Thinking of the worst-case scenario; Having a plan B; Becoming an expert; Practicing mindfulness; or Challenging your thoughts with cognitive behavioral therapy, etc.

While these are great strategies, and I encourage you to look into them, I have found them hard to implement in practice, in my everyday life, and at work.

So instead, I will suggest three strategies that I found to work well — for me. And maybe they work for you too.

1— Make it cool to fail

2 — Be the best at failing by failing fast

3— Be the best at learning from failure

1 — Make it cool to fail

I read about the idea of Failing Well a few years ago from an article in the New York Times. Failing Well is a program run by Rachel Simmons at Smith College, an all-women’s school in western Massachusetts. That program aims at teaching students that failing is part of learning. When students enroll in that program, they receive a Certificate of Failure. Rachel Simmons explains that by accepting that certificate, they are given the permission to fail while remaining “totally worthy, utterly excellent human beings.”

I loved that idea, but I wonder if my CEO would give me such a certificate. Instead, my coach suggested I create my own, and I called it my Journal of Failures. In this journal, I keep track of all my failures. Every time I fail, I first turn that failure into a little story and add it to my collection. Then I quietly celebrate it. This little ritual has given me permission to fail, and it is surprisingly energizing because I get excited by sharing these stories with others. Writing these stories down is also a way to externalize them and forgive me. Keeping my journal helps me redefine what failure means to me. It is like becoming an anthropologist of my own life.

So ask yourself these two questions:

How do you personally define failure?

Can you make it cool to fail?

2 — Be the best at failing by failing fast

To fail fast means we have a mechanism for gathering feedback as soon as we start working on a project. We first break a project into smaller tasks called milestones. At each milestone, we continuously evaluate whether to continue working towards the original goal or take a different approach. The key here is deliberate adaptation. If a project or milestone isn’t working as expected, it is best to realize it early rather than wait too long and waste time and money.

Remember the book Hackers and Painters discussed above? While the book compares software developers to an artist, the author explains that as opposed to the artist, for the software developer, the fastest way to get better is to get the work in front of others early and fast and get feedback.

That is basically how the Instagram application was born. Systrom and Krieger created a prototype application called Burbn, which allowed people to check in with their mobile phones. That prototype could have successed, but was found cluttered and overrun with features. In the spirit of failing fast, the founders stripped out all the features from the application except the one element that people seemed to use and like — uploading and commenting on photos. They renamed the application Instagram which went on to be bought by Facebook for $1 billion in 2012.

By failing fast, we gain insights into how to proceed differently in the future. Here are a few ways you can use that practice daily at work.

For example, when you develop a feature, prepare a document, or write a blog post, get feedback before it is ready. Get about 70% of your ideas on the paper and start circulating that paper around. Get input from as many sources as possible. The more diverse your sources, the better. Make it clear that it is a Work-In-Progress (WIP) and that you are gathering feedback. I found that people typically love giving others feedback. Sometimes it will be critics rather than feedback, but don’t worry. Take the good and the useful, and leave the rest out.

Do you want to talk publicly about a great idea of yours? Try yourself with a few trusted friends first. Then grow your audience to your team or a small group. Once you have received enough supportive feedback, go big.

But it takes time, so be patient. The key here is to find your network of people with whom you can circulate half-baked ideas. People that will give you honest feedback. It also means that you have to be ready to hear the truth. So detach yourself emotionally from your idea and genuinely listen to the feedback. It is a good sign if a few reviewers think you are onto something. If none of them do, it is also a sign; you either need to go back to the drawing board or expand your network until at least one person understands and helps you polish your idea.

The presence of failure is okay, but what is wrong is not being prepared to handle the feedback.

3 — Be the best at learning from failure

Be perfect at learning from your mistakes rather than trying to avoid them. Research has shown that employees who take on an avoidance focus become twice as mentally exhausted as their approach-focused colleagues.

The key here is to change our questions from a “Judger” mindset to a “Learner” one. In Change Your Questions, Change Your Life, Marilee Adams explains that learner questions are open-minded, curious, and creative. They promote progress and possibilities and lead to discoveries, understanding, and solutions.

By contrast, Judger questions are more closed-minded, confident, and critical. They focus on problems rather than solutions, often leading to defensive reactions, negativity, and inertia.

Learner questions facilitate progress by expanding options. Judger questions stop progress by limiting perspectives.

When we fail, it is natural to ask Judger questions, and that’s the emotional response to fear. But because we judge instead of learning, results suffer. If you can effectively distinguish between the two, your performance, productivity, and morale will only grow.

Research has shown that managers who shift to a learning mindset are more likely to recognize changes in employee performance and spend more time coaching, mentoring, and developing their employees.

Changing the questions we ask can shift mindsets and behavior to produce remarkably more positive results.

In the book Behind Human Error, researchers show that failures occur in complex socio-technical systems and that behind every “human error,” there is a second story.

In engineering, learning from mistakes by asking the right questions is often done by doing a post-mortem. At Amazon, we call that process a Correction-Of-Error (COE). The key to learning with that process is to be blameless and not focus on human error.

Blameless means we investigate mistakes by focusing on the circumstantial factors of a failure and the decision-making process of individuals and the organization involved.

The key to blameless learning is again to ask the right “Learner” questions and to create an environment that is safe to fail without fear of punishment. Creating a safe environment to fail is critical because engineers who think they will be fired are disincentivized to learn from their mistakes and prevent their re-occurrence. That is how failures repeat themselves. By being blameless, organizations come out safer than they would if they had punished people for their mistakes.

To learn more about the blameless post-mortem, I suggest you read Blameless PostMortems and a Just Culture by John Allspaw.

Circling back to deleting the production database

Looking back, what happened that day, more than ten years ago, when I deleted the database in production, made me who I am today.

Now, I am not suggesting you delete your company’s database. That was a painful experience, and I don’t recommend it to anyone.

We must realize is that everyone makes mistakes and that failures does not make us failures. Instead, we must make it safe to fail.

Failure is not a bug; it’s a feature.

We are all Work-In-Progress, so be kind to yourself and others.

It always saddens me to see talented people paralyzed by failure. I always wonder what happened to that child who learned the most challenging things in life — talking, walking, and running, but is now scared of failing, or trying anything new or different.

If you think about it, it takes newborn babies between 9 to 12 months to take their first steps and another 3 to 5 months to walk well.

How many times did the ‘you’ baby fall?

Did we stop trying because we failed to walk the first time properly?

When we were a baby, we didn’t question failures too much. We didn’t overthink it, and we just did our thing. We failed and tried again right away.

Why can’t we be like that anymore?

The truth is we will only succeed if we fail.

To paraphrase what Edison said: