Tim Harford's "Adapt: Why Success Always Starts with Failure"

This is an excellent survey of adaptation as a strategy to deal with complex environments in which a top-down rational planning simply can't find solutions. There are many excellent stories, but this one bit speaks to me, a 1960s generation person:

When the US Army faced the 'disruptive innovation' of guerrilla warfare in Vietnam, there was great reluctance to accept that it had changed the nature of the game, making obsolete the Army's hard-won expertise in industrial warfare. As one senior officer said, "I'll be damned if I permit the United States army, its institutions, its doctrine, and its traditions to be destroyed just to win this lousy war."

That ranks up there with the infamous statement by a US Major "It became necessary to destroy the town to save it." as reported by Peter Arnett.

This book is richly illustrated with examples of how an evolutionary strategy can find a solution where managerial dictum or top-down planning fails miserably. Some of the best examples are from business.

He does a very good job of reviewing:

• The disaster of the US invasion of Iraq and how a bottom-up effort by rebellious colonels and captains finally changed the US military tactics off their dangerous track and toward a more successful approach.


• The regulatory failures and coupled risks that led a meltdown in a constrained "sub-prime real estate" market to the globe straddling collapse of financial markets in 2008. He walks the reader through the failure to listen to whistle-blowers and the reluctance to effectively change the banking rules to prevent another catastrophe.


• The mindless simplifications of a greenhouse gas enthusiast compared to the known complexities of trying to identify a proper "green strategy". He walks through a day's choices of "green alternatives" and shows why each and every one was wrong because the underlying reality is far more complex than the simplistic green enthusiast ever could imagine.


• He examines nuclear safety and shows how the various catastrophes were waiting to happen because the systems are designed with too much complexity and coupled failure modes.


• He looks at two big oil rig diasters, the Piper Alpha in July 1988, and the Deepwater Horizon in April 2010. He walks through the failure in design and safety systems. He shows why these were accidents waiting to happen.

I like his discussion of Philip Tetlock who did a scientific study of the ability of pundits, gurus, and consultants to provide accurate predictions of the future. In a previous post I reviewed Dan Gardiner's book "Future Babble" that explores this topic more deeply.

I worked in a company where they paid lip service to the idea that "there is no failure, just a learning opportunity" and that projects, especially in the R&D lab, should expect a high failure rate. But in reality, failures were punished, so creativity was suppressed and lessons really weren't learned. Tim Harford gives a glowing review of Google as a learning environment with adaptive engineering practices, but I'm cynical. It is hard for managers to accept failure. Corporations are always going to get atherosclerosis. The big old successful corporations are always going to fall to the young, rising whippersnappers.

I also worked closely with QA (Quality Assurance) people and watched them play their role. In theory they were the frontline defence against obfuscation and deception on the part of managers and teams that are failing but want to pretend that things are going teckety-boo. The org chart showed them reporting independently right up to the CEO to ensure independent and timely information about project problems. But in reality project managers had a "right" to demand issues first be heard by them and they could muscle most QA auditors into silence. Similarly, I was involved in a ISO 9000 initiative within the company and quickly discovered that most of our "learning organization" capabilities, such as our extensive audited written procedures, were in fact window-dressing. In short: it is hard to build and maintain a truly adaptive organization that uses evolutionary strategies for problem-solving. Humans don't like uncertainty and they love hierarchical organizations. I like the message in Tim Harford's book, but I'm sure it will get more lip service than real implementation.

There is much wisdom in the book, much to learn, I strongly recommend it to everyone. It will open your eyes to the complexity that is out there. It will stun you to realize how badly our engineered "safety systems" have failed. And it will give you an appreciation for a need for more experimentation and a healthy acceptance of failure as the technique for learning.

Dean Baker gives Kudos for a Media Outlet that Gets the News Right

This is rare. Here is Dean Baker in his Beat the Press blog toasting the AP news service for getting a story right:

AP Gets It Right! The Recession Caused the Deficit

AP did what news organizations are supposed to do; it checked the numbers and showed that politicians yelling about "out of control spending," don't know what they are talking about or are making things up. Its FACT CHECK, printed in the NYT, showed that the deficit exploded because of the recession. There was no issue of out of control spending prior to the increases for programs like unemployment insurance and other spending needed to counteract the effect of the downturn. AP gets an "A" for this one.

Now, if only the other news media picked up on this story maybe Americans could get out of the grips of the deadly Republican idiocy about deficits and debts being "the #1 problem" in the US and get back to the real problem: unemployment!

How the Iraq War was Won... sort of

I was reading this post by Tim Harford where he mentioned Travis Patriquin who came up with the strategy in Al Anbar province which turned the tide of war and finally subdued the insurgents. I've bolded the key bits:

My military case study was the turnaround in the War in Iraq. I didn’t want to go over the post-invasion fiasco, the mysteriously elusive weapons of mass destruction, or the timing of the withdrawal: that’s for other writers and other books. But I was horribly fascinated by the initial refusal to learn from what was going on, and later at the military’s ability to learn and to adapt.

The key lesson is that this adaptation was a bottom-up phenomenon. (Perhaps it’s more accurate to call it “middle-management”: some of the key actors were colonels.)

At the time Donald Rumsfeld, then Defense Secretary, was holding press conferences and announcing that he didn’t even want to hear the word “insurgent” being used, troops on the ground were fighting a vicious three-sided insurgency. (To pick one horrifying incident amongst many, police recruits in the town of Tal Afar were murdered when somebody with explosives strapped all over their body walked into their midst. It wasn’t a suicide bomber, but a mentally disabled 13-year old girl, accompanied by a toddler whose hand she had been asked to hold as she walked towards the line of recruits.)

The soldiers on the front line had to figure out their response without calling their superiors’ attention to the fact that they were doing anything of note. Tips were circulated via email or even PowerPoint presentations, such as the wonderful “How to win the war in Al Anbar by CPT. Trav.” which used stick-figure drawings to convey vital information that the men at the top seemed to be ignoring.

All this, of course, was decision-making at far higher stakes than those we economists usually study. The creator of the “CPT. Trav.” slide deck, Captain Travis Patriquin, was killed in 2006 three weeks before Christmas, leaving behind his wife and three children. He was so respected in Al Anbar that the local sheiks turned out in force at his funeral.

So this is partly a story about how quickly good ideas spread when they have to: one British general told me, with an air of resignation, that the junior ranks quickly adapted to new challenges and new lessons because doing so saved lives, but the most senior officers tended to learn very slowly.

It is also a story about the role of technology in decision-making. The U.S. military placed increasing emphasis on the use of “effects based analysis of operations” – using massive amounts of data and computing power to allow commanders at headquarters to absorb information and react quickly, moving units around like chess pieces. Such techniques are hugely useful in some circumstances – a “shock and awe” campaign – but even then they do not always perform as advertised. We’re now discovering that in many campaigns, it is the decision maker on the ground – a captain or even a regular soldier – who has the information that counts and the ability to use it.

We’re also discovering that communications technology can be more effective at distributing information than at collating and centralizing it. This isn’t just a lesson for counterinsurgency campaigns but for businesses. The economists Julie Wulf and Raghuram Rajan have found evidence that businesses are increasingly decentralising their decision making, while Lorin Hitt and Erik Brynjolfsson showed a decade ago that information technology worked much better when combined with this kind of decentralisation.

Meeting some of the soldiers who fought in Iraq and Afghanistan was a sobering experience for me. It was fascinating to discover how the U.S. military adapts. It was awful to read and hear about some of the atrocities committed in Iraq, and wonder about how things might have gone differently. And it was hard to forget the conversations with these soldiers. Experimenting, learning, and adapting is inescapable in a complex world. But rarely has it mattered so much to find the right answers as quickly as possible.

Click here to read Travis Patriquin's PDF containing the PowerPoint slide show with stick figures that explaining how to win the Iraq war. It truly is wonderful. Please read it.

Of course, everybody knows that the wonderful lessons learned by Travis Patriquin were lost when American officials handed over control of Al Anbar to the central government in Iraq and these Shiites didn't like the fact that the sheiks with their local Sunni "policemen" were a threat to their central power, so that was all disbanded and chaos has again returned to Al Anbar. So much for "progress". So much for the illusion that rational people control government or that humans are fundamentally "rational".

Sadly, the Wikipedia account doesn't present it this way. It presents it as a "successful" handover to central authority... and everybody lived happily ever after... amen.

An Inspirational Story

I love stories about underdogs rising up and surprising everybody. Here's a story about a water-logged soccer team and its surprising success:

Worshiping the God of War

Here is a bit from an interesting article by William J. Astore on TomDispatch:

No wonder that George W. Bush, Dick Cheney, and Donald Rumsfeld were so eager to go to war in Iraq in 2003. They saw themselves as the new masters of Blitzkrieg, the new warlords (or “Vulcans” to use a term popular back then), the inheritors of the best methods of German military efficiency.

This belief, this faith, in German-style total victory through relentless military proficiency is best captured in Max Boot’s gushing tribute to the U.S. military, published soon after Bush’s self-congratulatory and self-adulatory “Mission Accomplished” speech in May 2003. For Boot, America’s victory in Iraq had to “rank as one of the signal achievements in military history.” In his words:

"Previously, the gold standard of operational excellence had been the German blitzkrieg through the Low Countries and France in 1940. The Germans managed to conquer France, the Netherlands, and Belgium in just 44 days, at a cost of ‘only’ 27,000 dead soldiers. The United States and Britain took just 26 days to conquer Iraq (a country 80 percent of the size of France), at a cost of 161 dead, making fabled generals such as Erwin Rommel and Heinz Guderian seem positively incompetent by comparison.”

How likely is it that future military historians will celebrate General Tommy Franks and elevate him above the “incompetent” Rommel and Guderian? Such praise, even then, was more than fatuous. It was absurd.

The whole article is well worth reading. It looks at the misguided belief that the US can be so good at war that it will win. The author points out that this ignores luck and changeable facts. Worst, it igores the hard truth of how the effectiveness of the military since 2001 has going steadily downhill as Tom Englehardt's preface makes very clear:

Remember the 100 hours of combat that made up the first Gulf War, the mere weeks it took for Kabul to fall in the 2001 invasion of Afghanistan, or the “shock and awe” wave of air attacks that led off the 2003 invasion of Iraq, followed by the 20-day blitzkrieg-like campaign that left American troops occupying Baghdad? Those were the days when, as retired lieutenant colonel and TomDispatch regular William Astore reminds us, the civilians in the Bush Pentagon thought they were the masters of lightning war. Now, skip almost seven years, and in Afghanistan the U.S. military has just launched the largest campaign since the invasion of 2001. Fifteen thousand U.S., British, and Afghan troops have been dispatched to take Marja, a single, modest-sized, Taliban-controlled city of 80,000 in one of more than 700 districts in Afghanistan, many under some degree of Taliban control or influence. How the time frame for success has changed.

Making it Big

The NY Review of Books has an article by Sue M. Halpern that reviews three books on what it takes to be a success:

• The Snowball: Warren Buffett and the Business of Life
  by Alice Schroeder


• Outliers: The Story of Success
  by Malcolm Gladwell


• Talent Is Overrated: What Really Separates World-Class Performers from Everybody Else
  by Geoff Colvin

These three books have very little in common. The Buffett book is a standard biography. The Gladwell book is a brilliant thought piece in the tradition of his other books. I haven't read the Colvin book but I suspect it is a fairly pedantic book that looks at factors that contribute to success. The first two books are a lot of fun to read. Here's a bit that Halperns includes from the Gladwell book:

Consider, as well, Chris Langan, whose astronomically high IQ—around 195, 45 points higher than Einstein's—does actually make him a statistical anomaly. In Gladwell's estimation, though, the poor fellow—who lives in relative obscurity on a midwestern horse farm—is a great failure:

He'd been working for decades now on a project of enormous sophistication—but almost none of what he had done had ever been published much less read by the physicists and philosophers and mathematicians who might be able to judge its value.

Gladwell writes:

Here he was, a man with a one-in-a-million mind, and he had yet to have any impact on the world. He wasn't holding forth at academic conferences. He wasn't leading a graduate seminar at some prestigious university. He was living on a slightly tumbledown horse farm in northern Missouri, sitting on the back porch in jeans and a cut-off T-shirt. He knew how it looked: it was the great paradox of Chris Langan's genius.

"I have not pursued mainstream publishers as hard as I should have," he conceded. "Going around, querying publishers, trying to find an agent. I haven't done it and I am not interested in doing it."

It was an admission of defeat.

Really—says who?

Gladwell's explanation for what he believes is Langan's epic failure goes to the heart of his main thesis about success—that it cannot be explained by understanding what a person is like but only by where he or she is from. It's not clear, precisely, why Gladwell considers this is a revelation, not a tautology, but he does. The social science bookshelf is filled with studies linking achievement to background. Most recently, the economic mobility project of the Pew Charitable Trusts found, for instance, that nearly half "of those born to parents in the top quintile [income] who have a college degree remain at the top, [which is] nearly triple the percentage of college graduates born to parents at the bottom that make it to the top of the income distribution." In any case, all stories of success or failure are construed after the fact and the same set of circumstances often leads to fundamentally different outcomes, the explanation for which typically invokes those circumstances. (For example, one man's inherited wealth leads to the revolving door of the Betty Ford Clinic, while another's leads to the Pulitzer Prize in Poetry.) To claim, as Gladwell does, that "extraordinary achievement is less about talent than opportunity" overstates the obvious while letting the rest of us, who are not overachievers, off the hook. "People don't rise from nothing," he writes.

We owe something to parentage and patronage. The people who stand before kings may look like they did it all by themselves. But in fact they are invariably the beneficiaries of hidden advantages and extraordinary opportunities and cultural legacies that allow them to learn and work hard and make sense of the world in ways others cannot.

Chris Langan, the oldest son of a woman who had four boys, each with a different father, the last of whom was an abusive alcoholic, was raised with none of the advantages that might have allowed him to prosper. Though he won full scholarships to the University of Chicago and to Reed College, and enrolled at Reed, he left before the end of freshman year, a crew-cut kid among long-hairs whose mother forgot to fill out the financial aid forms. Then he went to Montana State, and his car broke down and a professor wouldn't let him change a morning class for an afternoon class, and Langan left there, too.

If Langan had had what Gladwell, citing the psychologist Robert Sternberg, calls "practical intelligence"—knowing what to say and who to say it to—he might have graduated from college, gone to graduate school, become an academic, written peer-reviewed articles, sat on innumerable committees, and made something recognizable out of his life. But lacking the kind of family background from which, Gladwell says, such knowledge comes, he was doomed to failure.

For some reason, the Langan story makes Gladwell think of the physicist Robert Oppenheimer, another young prodigy ("Ask me a question in Latin and I will answer you in Greek,") who was also a depressive and a miscreant: at Cambridge he tried to poison his tutor. Rather than being sent home or to jail, Oppenheimer was set up with a psychiatrist and allowed to continue his studies, a turn of events that Gladwell attributes to his practical intelligence:
Would Oppenheimer have lost his scholarship at Reed? Would he have been unable to convince his professors to move his classes to the afternoon? Of course not. And that's not because he was smarter than Chris Langan. It's because he possessed the kind of savvy that allowed him to get what he wanted from the world.

The sad tale of Chris Langan resonates with me. It is very hard to rise above what you know no matter how "bright" you are. The story of Oppenheimer is the opposite. It is hard to fall from grace when your parents are affluent and well-connected. That's the friction that means that the "meritocracy" claimed by our society will never really honestly work.