Business advice
plagued by survivor bias

November 2, 2009 | Jason Cohen

Do you read business blogs where the author has failed three times without success?

No, because you want to learn from success, not hear about “lessons learned” from a guy who hasn’t yet learned those lessons himself.

However, the fact that you are learning only from success is a deeper problem than you imagine.

Some stories will expose the enormity of this fallacy.

Bullet holes: A brain teaser

During World War II the English sent daily bombing raids into Germany. Many planes never returned; those that did were often riddled with bullet holes from anti-air machine guns and German fighters.

Wanting to improve the odds of getting a crew home alive, English engineers studied the locations of the bullet holes. Where the planes were hit most, they reasoned, is where they should attach heavy armor plating. Sure enough, a pattern emerged: Bullets clustered on the wings, tail, and rear gunner’s station. Few bullets were found in the main cockpit or fuel tanks.

The logical conclusion is that they should add armor plating to the spots that get hit most often by bullets. But that’s wrong.

Planes with bullets in the cockpit or fuel tanks didn’t make it home; the bullet holes in returning planes were “found” in places that were by definition relatively benign. The real data is in the planes that were shot down, not the ones that survived.

This is a literal example of “survivor bias” — drawing conclusions only from data that is available or convenient and thus systematically biasing your results.

Doesn’t most business advice suffer from this fallacy? You read about successes, but what about the businesses that “never made it home?” Like the downed planes, could failure contain more lessons than success?

Burying the other evidence

Scientific journals like to publish extraordinary results, so studies that don’t show anything of statistical significance aren’t published but rather are abandoned or silently stowed away in academic filing cabinets.

This practice is called the “file-drawer effect,” and it’s a particularly insidious form of survivor bias because it’s invisible.

Peter Norvig sums it up nicely:

“When a published paper proclaims ‘statistically, this could only happen by chance one in twenty times,’ it is quite possible that similar experiments have been performed twenty times, but have not been published.”

Pharmaceutical companies have exploited this effect to intentionally skew results. It’s gotten so bad that journals are calling for a public database to prevent fraud, according to a Sept. 9, 2004, Washington Post news story, which said in part:

“More than two-thirds of studies of anti-depressants given to depressed children, for instance, found the medications were no better than sugar pills, but companies published only the positive trials.

“If all the studies had been registered from the start, doctors would have learned that the positive data were only a fraction of the total.

Doesn’t most business advice suffer from this fallacy? Harvard Business School’s famous case studies include only success stories. To paraphrase Peter, what if twenty other coffee shops had the same ideas, same product, and same dedication as Starbucks, but failed? How does that affect what we can learn from Starbucks’s success?

Experimental proof of ESP

Dr. Joseph Rhine brought the rigor of experimental psychology to the study of the paranormal, and ESP (Extra Sensory Perception) in particular. He made waves in the 1930s with controlled experiments testing whether a person was able to predict the order of the cards in a shuffled Zener deck (with symbols like circle, square, star, and wavy lines).

In a typical experiment, 500 people are screened for “strong telepathic ability,” measured by significantly above-average performance in a 25-card deck. Those selected are tested again, and more drop away. Tested a third time, perhaps one person passes again, and we conclude that such a repeat performance is statistical evidence of genuine ESP.

To see why this is just a different face of survivor bias, consider the following experiment. I believe some people are “heady” when it comes to coin-flipping — getting heads more often than chance alone would suggest. So I put 1,000 people in a room and tell them to flip a coin ten times. Sure enough, a woman named Margaret makes “heads” ten times in a row! The chance of her getting heads ten times in a row is only one in 1,024, so I conclude Margret has special abilities.

Actually, that last statement is true but misleading. The chance that Margaret would flip ten heads in a row is one in 1,024, but that’s not the experiment I ran, was it? I let 1,000 people flip and “found” Margaret in the crowd.

The chance that somebody in a crowd of a thousand would flip heads ten times is a whopping 62 percent! Because so many people are attempting the feat, some normally unlikely events will happen. This isn’t a test of Margaret’s abilities at all!

Doesn’t most business advice suffer from this fallacy? Take me for instance. I’ve done three consecutive successful startup companies, so that’s proof that I know what I’m doing and that you should do everything I say, right? Except maybe I’m just the one in the crowd who guessed right on the Zener cards three times, and there’s no reason to believe I would be successful a fourth time.

Specific examples of survivor bias in business advice

So far I’ve been asking rhetorically whether survivor bias might be severely skewing business advice. Steven Levitt (of Freakonomics fame) investigated this question directly.

He was reading Good to Great by Jim Collins, a book that analyzed eleven companies that were mediocre — just pooping along — but then transformed themselves into stock market sensations. A conclusion was that the common trait was a “culture of discipline.” This book has sold many millions of copies, so it’s a good example of popular writing on business advice.

One of the eleven “great” companies was Fannie Mae, and Levitt was reading this book just as Fannie was collapsing in financial disaster. Hmm, he thought, I wonder how those other “great” companies are doing.

Turns out, had you invested in those eleven companies in 2001 (when the book came out), your portfolio would have underperformed the S&P 500! (Fannie Mae wasn’t even the only case of total disaster — also extolled was the now-bankrupt Circuit City.)

Why didn’t these companies continue to succeed? It turns out Jim started by combing through 1,435 companies looking for good candidates for the book, and picked eleven. It’s the ESP experiment all over again!

On top of that, Jim doesn’t bother asking whether any of the 1,424 other companies also displayed a “culture of discipline.” Maybe that’s something that many public companies have regardless of performance.

Is this book an aberration? Nope, Steven investigated another business book from the 1980s — In Search of Excellence — and found the same effect.

Steven then comes to the same conclusion that I’m coming to:

“These business books are mostly backward-looking: what have companies done that has made them successful? The future is always hard to predict, and understanding the past is valuable; on the other hand, the implicit message of these business books is that the principles that these companies use not only have made them good in the past, but position them for continued success.”

To the extent that this doesn’t actually turn out to be true, it calls into question the basic premise of these books, doesn’t it?

Oops, did I just invalidate my blog?

Lately, I’ve been wondering if a lot of business advice — both mine and others — is really a case of survivor bias. I mean, I didn’t start out at Smart Bear with a load of philosophy and a fixed idea of who the customer was or even what the products would be.

How do I know this post-hoc philosophy and advice isn’t just a case of survivor bias? Am I not like the ESP-savant, successful not by force of nature but by simple chance of surviving?

Or perhaps I’m like Dr. Rhine, the ESP experimenter – convinced I’ve discovered something important with “objective measures of success” — and, yet, I’m actually living in a dream world.

More to the point, how can you, dear reader, ascertain whether my articles or any advice from anywhere suffers from this fallacy?

In the end, of course, you don’t know. But here’s something: Just the fact that you’re aware of survivor bias means you’re less likely to be fooled by it. So, reading this article has helped a little.

Beyond that, prefer advice that makes you think and forces you to answer tough questions of yourself, not advice that simply tells you to march in a certain direction. Use advice as a sounding board rather than gospel.

What do you think? How do you decide which advice to take? Leave a comment  and join the conversation.

This post originally was published Aug. 17, 2009, on Jason Cohen’s blog, “a smart bear” Check there to see comments and more tips from his readers!

Jason Cohen founded Smart Bear Software, maker of Code Collaborator, a tool for peer code review and recent winner of the Jolt Award. He took Smart Bear from start to multiple millions in revenue and 50 percent profit margin without debt or VC, then sold it for cash. He also is a founding member of ITWatchdogs, another bootstrapped startup which became profitable and was sold. He’s also a mentor at Capital Factory (like TechStars or Y-Combinator in Austin). And, he’s the author of Best Kept Secrets of Peer Code Review, the most popular book (35,000 copies) on modern, lightweight methods for doing peer code review effectively without everyone hating life. He blogs at asmartbear Email him: jason (at) asmartbear (dot) com

This post was tagged:

{ 1 comment }

Darrell C. Greenhouse November 7, 2009 at 4:21 pm

Great article… 
On a side note… 
When I was 25 and starting out in my first business, I met a V.C. in all places but “traffic court.”
It was obvious from a number of factors that he was extremely successful and really didn’t need to spend his time in traffic court – “It’s the principle” he replied after explaining to me why he’d “waste” an hour and half of his valuable time. He was right and won his case handily! 
Anyway, we started talking about business and he began probing me about my current situation. I told him about my business activities, lamented the cash-flow issues I was experiencing [hoping of course to interest him in my venture :) ], but all-in-all, I informed him that I thoroughly enjoyed the freedom and independence of having my own business afforded me. 
I believe I came right out and asked if he’d be interested in investing in my company to which he replied – “I like you kid, but I never invest in anyone that hasn’t failed at least twice.” 
Paraphrasing what he went on to say – “someone who hasn’t failed and gotten back up to do it again, isn’t truly ready to succeed. They just don’t know enough yet…” 
I’ve noted his sage advice many times throughout my life, and find in the majority of cases, he was & is correct. 

Comments on this entry are closed.

{ 2 trackbacks }