Dan Gardner's "Future Babble: Why Expert Predictions Fail -- and Why We Believe Them Anyway"

This is an excellent review of just why we are suckers for doomsday fanatics and inveterate seekers of certainty about the future. He reviews the science, describes why we have a proclivity for prediction, and walks you though a number of examples of famous predictors and their failed predictions. And what is truly mind-boggling, he shows how people continue to believe the predictions in the face of overwhelming evidence of their wrongness.

He divides experts into hedgehogs (who "know" one thing in great depth) and foxes (who know many things, but more importantly know their limits). He shows how hedgehogs dig themselves into a hole with their "predictions" and when caught predicting something that is untrue, they simply continue digging their hole deeper. He explains that foxes are reluctant to predict, hedge their predictions, and are willing to admit to error when their predictions are shown to be untrue.

This book is an excellent antidote to those talking heads that the media loves to put in front of a camera and have them make grand "predictions" about this and that. You will learn why and how you get sucked into their predictions. It isn't simply them bedazzling you. You are built to want the certainty of a "good prediction" about the future. We create these doomsday fanatics and these over-priced sellers of "predictions" about the future.

Here are some interesting bits from the book:

With natural science increasingly aware of the limits of prediction and with prediction even more difficult when people are involved, it would seem obvious that social science -- the study of people -- would follow the lead of natural science and accept that much of what we would like to predict will forever be unpredictable. But that hasn't happened, at least not to the extent it should. In fact, as the Yale University historian John Lewis Gaddis describes in The Landscape of History, "social scientists during the 20th century embraced a Newtonian vision of linear and therefore predictable phenomena even as the natural sciences were abandoning it." Hence, economists issue wonky forecasts, criminologists predict crime trends that don't materialize, and political scientists foresee events that don't happen. And they keep doing it no matter how often they fail.

One of my personal favourites, Paul Ehrlich, founder in the late 1960s of ZPG and author of a number of doomsday books including his most famous The Population Bomb, is featured in this book and covered in several sections. Here is a bit:

Paul Ehrlich was also emboldened by the way things were going and he published a string of jeremiads, each more certain of worse disasters than the last. "In the early 1970s, the leading edge of the age of scarcity arrived," he wrote in 1974's The End of Affluence. "With it came a clearer look at the future, revealing more of the nature of the dark age to come." Of course there would be mass starvation in the 1970s -- "or, at the latest, the 1980s." Shortages "will become more frequent and more severe," he wrote. "We are facing, within the next three decades, the disintegration of nation-states infected with growthmania." Only the abandonment of growth-based economics and other radical changes offered any hope of survival. And some countries were doomed no matter what they did. India was among the walking dead. Ehrlich was sure. "A run of miraculously good weather might delay it -- perhaps for a decade, maybe even to the end of the century -- but the train of events leading to the dissolution of India as a viable nation is already in motion." Japan is almost certainly "a dying giant." Same for Brazil. The united Kingdom was only slightly better off. The mere continuation of current trends will ensure that "by the year 2000 the United Kingdom will simply be a small group of impoverished islands, inhabited by some 70 million hungry people, of little or no concern to the other 5-7 billion people of a sick world," he wrote in an earlier paper. Of course, it could be worse than that: Thermonuclear war or some variety of eco-catastrophe were distinct possibilities. "If I were a gambler, I would take even money that the average Briton would be a distinctly lower than it is today." Not that Americans are in any position to gloat, Ehrlich cautioned. The United States is entering "the most difficult period ever faced by industrialized society." This dark new age may well see the end of civilization. Ehrlich noted that those in the burgeoning "survivalist" movement were stockpiling supplies in wilderness cabins -- "a very intelligent choice for some people" -- but he and his wife had decided to stay put. "We enjoy our friends and our work too much to move to a remote spot and state farming and hoarding." Ehrlich wrote. "If society goes, we will go with it."

He looks into the underlying psychology that fuels our need for predictions and the cognitive biases that make us susceptible:

In contrast, the gloom-mongers have it easy. Their predictions are supported by our intuitive pessimism, so they feel right to us. And that conclusion is bolstered by our attraction to certainty. As strange as it sounds we want to believe the expert predicting a dark future is exactly right, because knowing that the future will be dark is less tormenting than suspecting it. Certainty is always preferable to uncertainty, even when what's certain is disaster.

And here is our real Achilles' heel:

The media superstars who tell us what will and won't happen are an overwhelmingly confident bunch. Talking heads on business shows are particularly cock-sure but the other pundits who engage in prognostication -- whether futurists, newspaper columnists, or the sages on television who tell us the fate of politicians and nations -- share the same fundamental characteristics. They are articulate, enthusiastic, and authoritative. Often, they are charming and funny. Their appearance commonly matches the role they play, whether it's a Wall Street sharpie or a foreign affairs maven. They commonly see things through a single analytical lens, which helps them come up with simple, clear, conclusive, and compelling explanations for what is happening and what will happen.

They do not suffer doubts. They do not acknowledge mistakes. And they never say, "I don't know."

They are, in this book's terms, hedgehogs. Their kind dominates the op-ed pages of newspapers, pundit panels, lecture circuits, and best-seller lists.

Now, if this is true, and if it's also true that the predictions of hedgehogs are even less accurate than those of the average expert -- who does as well as a flipped coin remember -- then a disturbing conclusion should follow. The experts who dominate the media won't be the most accurate. In fact, they will be the least accurate. And that is precisely why to measure the fame of each of his 284 experts, Tetlock found that the more famous the expert, the worse he did.

The result may seem more than a little bizarre. Predictions are a big part of what media experts do, after all. Surely experts who consistently deliver lousy results will be weeded out, while the expert who does better than average will be rewarded with a spot on the talking-head shows and all that goes with it. The cream should rise, and yet, it doesn't. In the world of expert opinions, the cream sinks. What rises is the stuff that should be, but isn't, skimmed off and thrown away.

How is this possible? Very simply, it's what people want. Or to put it in economic terms, its supply and demand: We demand it; they supply.

Gardner is referencing a scientific study by Tetlock that investigated the quality of predictions by experts. He goes on to explain why and how we demand these inaccurate experts.

Here's an example of one psychological reason why we cling to bad predictions: confirmation bias:

After Japan attacked the United States at Pearl Harbor, General John DeWitt was sure American citizens of Japaneses origin would unleash a wave of sabotage. They must be rounded up and imprisoned, he insisted. When time passed and there was no sabotage, DeWitt didn't reconsider. "The very fact that no sabotage has taken place is a disturbing and confirming indication that such action will be taken," he declared. As the reader may realize, DeWitt's reasoning is an extreme example of the "confirmation bias" discussed earlier.

He ends the book by trying to give the reader a guide to deal with the gloomy doomsters that proliferate in the media:

Skepticism is a good idea at all times but when the news is especially tumultuous and nervous references to uncertainty are sprouting like weeds on the roof of an abandoned factory, it is essnetial. The 1970s were one such time. As I write, we are in another.

The crash of 2008 was s shock. The global recession of 2009 was a torment. Unemployment is high, economics weak, and government debt steadily mounts. The media are filled with experts telling us what comes next. We watch, frightened and fascinated, like the audience of a horror movie. We want to know. We must know.

For the moment, what the experts are saying is, in an odd way, reassuring. It's bleak, to be sure. But it's not apocalyptic, which is a big improvement over what they were saying when the crash was accelerating and the gloomier forecasters, such as former Goldman Sachs chairman John Whitehead, were warning it would be "worse than the Great Depression." To date, things aren't worse than the Great Depression, nor are they remotely as bad as the Great Depression. Naturally, the gloomsters would be sure to add "so far." And they would be right. Things change. The situation may get very much worse. Of course it may also go in the other direction, slowly or suddenly, modestly or sharply. The range of possible futures is vast.

This book is a fun read and very informative. It is an excellent prophylactic to the crazies who want to worm their way into your mind and turn you into some end-of-the-world cult member. Read it!

If you want to read about modern day hedgehogs whose "predictions" have big consequences for us, read this bit by Paul Krugman about bond vigilantes and here where he points out those who see any rise in inflation as a sign of hyperinflation and a debasement of the currency. Sadly these "experts" are forcing governments around the world, in the midst of the Great Recession, to take on austerity programs similar to FDR's in 1936 that led to the great 1937 recession within the Great Depression.