Jellybeans

In 2007, Michael Mauboussin presented a big jar of jelly beans to his 73 Columbia Business School students. How many beans did they think it contained?

Guesses ranged from 250 to 4,100; the actual number was 1,116. The average error was 700 — a massive 62 percent — demonstrating that the students were awful estimators.

Now here comes the weird part.  Even with all these wildly incorrect guesses, the average guess was 1,151 — just 3 percent off the mark. Not only that, only 2 of the 73 students guessed better than this group average.

So, although individually everyone was woefully inaccurate, collectively the group was incredibly accurate.

Was this a fluke? Hardly. The experiment was made famous in 1987 by Jack Treynor. In his case, it was 850 jelly beans and 56 students. The group estimate was 2.5 percent off; only one student guessed better. The study has been repeated many times since with similar results.

This eerie effect goes beyond jelly beans; it’s also a big help when you’re trying to make money on TV.

The best multiple-choice test, ever

A contestant on the game show Who Wants to be a Millionaire can win a million dollars if she answers 15 consecutive multiple-choice questions. If she’s stumped along the way she has three “life-lines:” (1) eliminate two of the four choices, (2) telephone a friend, or (3) poll the audience. The jelly bean experiments imply that this third choice might be pretty good. Is there as much wisdom in the crowd for pop culture and science as there is in counting jelly beans?

The TV studio audience predicts the correct answer an astonishing 91 percent of the time. Remember, these are questions from all domains of knowledge, all ranges of difficulty, polling a group of people who happened spend the (weekday) afternoon in a TV studio.

To quantify how amazing that is, compare (that pecentage) with the accuracy of the “phone a friend” life-line where the contestant gets 30 seconds with a predetermined person.  This accomplice is probably considered to be “the smartest person I know,” plus undoubtedly has access to the web of lies Google and Wikipedia.

The intelligent friend with broadband access to the entirety of human knowledge gets it right only 65 percent of the time.

Crowd wins again.

Is the rule universal?

There’s seemingly no end to studies like these, all showing that the crowd is smarter than the individual. Is this a universal rule? Should we be leveraging this power more often?

Big companies do use crowd wisdom. You always hear about advertising campaigns being honed by focus groups of “real people.” (I’d like to see the questionnaire that distinguishes “real people” from that elusive other kind of person.)

However, company messaging, product features, advertising layouts, and the other creative aspects of business require innovation, and we know that design-by-committee is the antithesis of innovation. Average products designed for the average consumer is the path to small business failure.

So what should we do? Can we rely on the wisdom of the collective or should we trust a stroke of inspiration?

Analysis of how “crowd wisdom” works

Let’s take another look at Who Wants to be a Millionaire.

Suppose there are 100 people in the audience and only 16 of them know that “A” is the correct answer. Of the rest, none knows the answer and they vote randomly. The result of the vote will be: 37, 21, 21, 21:

Oh, gee, it’s awfully similar to the graphic earlier of a real audience poll.

(For those of you so inclined, it’s fun to try more complex scenarios, although you’ll find the result is always similar.  For instance, what if only 11 know the answer is A, 15 each know that B, C, or D are certainly not the answer (and vote randomly for the other three), and the remaining 44 have no clue and vote randomly.  In this scenario, the vote distribution is the same as in the given example!)

So, we have the interesting result that a mere 16 percent of the voters were able to make choice A the clear winner — nearly double the next closest answer. The reason? The ignorant people vote randomly and their votes cancel out, leaving the rest in control of the result.

The crowd vetoes innovation

Now that we understand how crowds can be right, let’s see why this same process doesn’t work for creative endeavors.

Consider what happens when you’re planning a holiday meal. There’s a range of fantastic things you could cook, but wait: Some people can’t take spicy food. Uncle Bill is allergic to garlic. Aunt Sarah doesn’t eat red meat. Timmy doesn’t eat anything green.

Eventually you realize there’s only one way to please everyone: Cook something bland, mild, and safe, like chicken and rice. But does chicken and rice actually please anyone? Not really, it was just what everyone hated the least.

Votes don’t converge on something wonderful. Rather, votes are vetoes.

Of course, if you’re a catering company for weddings, chicken and rice might be the way to go! After all, no one goes to weddings for the food, so your primary goal is to piss off as few of the 300 guests as possible. Come to think of it, chicken and rice does seem to be popular at those sorts of functions …

But this isn’t a strategy for startups. Little companies need a niche — a market space they can completely, unquestionably own, not some gray middle-ground where your attempt to offend no one also means exciting no one.

There is “wisdom in the crowd” only when errors cancel out, like when estimating jelly beans or answering pop culture questions. In creative work, votes eliminate the interesting edges, leaving only the boring residue that no one hated enough to vote off the island.

That’s not how great products are made.

What do you think? When should you follow consensus and when should you run off and trust your guy? Join the conversation!

Further Reading

• James Surowiecki’s The Wisdom of Crowds, with more stories and implications for Wall Street. He also writes a terrific financial column and blog for The New Yorker.
• Scott Page’s The Difference, explaining how diversity makes a group smarter. The inspiration for my Who Wants to be a Millionaire example.

This post originally was published Jan. 10, 2009, on Jason Cohen’s blog, “a smart bear.” Check here to see comments and more tips from his readers!