Yes, I know. Everyone and their dog has written about survivorship bias. The story of the Second World War planes and the mathematician Abraham Wald has become as much of a meme as Schrödinger's cat: it is retold at every other training session, and everyone nods knowingly. A quick reminder. The military studied the bullet holes on bombers that came back and wanted to add armour where the holes were most frequent. Wald suggested the opposite: armour the places with no holes. Because the planes hit in exactly those places simply did not return, and there was nobody left to tell about it.
Everyone nods. And then they go off and do exactly what they just laughed at. That is why I am writing this article after all. Survivorship bias is worn out in theory and still very much alive in practice. Especially in business, and especially in IT.
What it looks like in our world
In software development survivorship bias is everywhere, just dressed differently.
A startup sees that a large platform is built on microservices and decides microservices are the path to success. It does not see the hundreds of companies that drowned in the complexity of a distributed system long before they grew into the load that complexity is meant for. They do not speak at conferences. Nobody listens to their talks, because there are none.
A founder reads an interview with a billionaire who dropped out of university and concludes that education is overrated. He does not read interviews with the thousands of people who also dropped out and are now very much not billionaires. Nobody interviews them.
A team says: "We built a project without tests for five years, and everything worked." Maybe. It is also possible that three of their neighbouring projects without tests died quietly and are simply never mentioned. Or that this project did work, but every change cost three times what it could have, and nobody ever counted.
Success stories are loud because survivors tell them. Failures are quiet because they have no narrator. If You only hear the loud ones, You are hearing a very distorted picture.
You have planes that did not return too
The most treacherous place survivorship bias hides is not in books about success but in your own business. Look at where You get your feedback. From clients who stayed. They write to support, ask for new features, complain about little things. And You honestly work through those complaints, patching holes in the wings and tail.
Clients who left usually write nothing. They just stopped coming. They signed up, poked at the interface for three minutes and closed the tab. They called once, did not get an answer and went to a competitor. These are your planes that did not return. And their reasons for leaving are exactly the places that need armour. But those are precisely what You know least about.
I often ask clients to look not at those who bought but at those who did not. Where do people drop out of the funnel? At which step of sign-up do they close the page? Which leads did managers mark as "not relevant", and why? Almost always something turns up there that the team could not see right in front of them, because they were looking at the satisfied ones.
How not to become the error
If You translate Wald's lesson into practical rules for business, You get something like this.
- Study failures more often than successes. When You want to repeat someone's path, ask: how many people tried the same thing and failed? If You cannot answer, You are only seeing the survivors.
- Look for quiet data. Clients who left, rejected leads, abandoned carts, employees who quit after a month. Talk to at least a few of them.
- Do not copy decisions without context. A technology that suited a company with a million users and a hundred developers can kill a team of five.
- Review not only failures but successes. Sometimes a project went well despite the decisions, not because of them. If You do not examine it, You will cement chance as a method.
- Distinguish "it worked" from "it was right". A good decision can lead to a bad result through bad luck, and a bad one to a good result through good luck. Judge the quality of the reasoning, not just the outcome.
But You still have to survive
This topic has a flip side that is mentioned less often. Understanding survivorship bias can turn a person into an eternal sceptic who does nothing because "it's all a systematic error and the odds are slim". That is the other extreme, and it is no better. Planes still have to fly. Wald did not suggest stopping the missions; he suggested armouring them correctly.
Surviving in business means not avoiding risk but distributing it sensibly. Making bets You can afford to lose. Armouring what is truly critical: money for a rainy day, key people, relationships with your main clients, data. And flying anyway, because a plane is safe on the ground, but that is not what it was built for.
At Ingello, by the way, we use this approach when auditing systems: we look not only at what broke and has tickets, but also at where users silently work around the system, keep spreadsheets on the side or make a phone call instead of pressing a button. These workarounds are the invisible holes.
What to do tomorrow
Find five clients who stopped working with You in the last six months, or never started even though they came close. Call or write to them personally, not with a sales pitch but with one question: why? Do not argue or justify yourself, just write down the answers. Almost certainly these five conversations will contain more useful information than a month of working with reviews from happy clients. That is your map of the places without holes, where the armour needs to go.
