Rational Thinking

Here is an example of using bayesian reasoning to wriggle between the extremes of ignoring a possible problem and giving oneself over to despair. This is from Tim Harford's blog in which he has allowed "Sophy" to post:

You’re a woman in her early fifties. You’re invited to a breast cancer screening unit, and you go along hoping for the all-clear. After all, 99 per cent of women your age do not have breast cancer. But … the scan is positive. The screening process catches 85 per cent of cancers. There is a chance of a false alarm, though: for 10 per cent of healthy women, the screening process wrongly points to cancer. What are the chances that you have breast cancer?

Over 50,000 British women face this awful question each year. I first encountered it – in a less alarming context – as an undergraduate economist. And I was in the audience recently when David Spiegelhalter used it as an example in his Simonyi Lecture, “Working Out the Odds (With the Help of the Reverend Bayes)”. The numbers approximately reflect the odds faced by women who go for breast cancer screening. And the answer – courtesy of the Reverend Bayes in question, who died 250 years ago – is surprising.

Bayes was concerned with how we should understand the notion of “probability”, and how we should update our beliefs in light of new information.

A Bayesian perspective on the apparently grim screening result tells us that things are not as bad as they seem. The two key pieces of information point in different directions. On the one hand, the positive scan substantially worsens the odds that you have cancer. But on the other, the odds are worsening from an extremely favourable starting point: 99 to 1 against. Even after the positive scan, you still probably don’t have cancer.

Imagine 1,000 women in your situation: 990 do not have cancer, which means we can expect 99 false positives, far more than the 10 women who do have cancer. This is why any apparent sign of cancer should be followed up with further tests in the hope of avoiding unnecessary treatments. The chance that you have cancer is 9 per cent – up dramatically from 1 per cent, but with plenty of room for optimism.

None of this proves screening is pointless. It can save lives, but it raises dilemmas. The UK’s breast cancer screening programme is currently under review. A systematic analysis published by the Cochrane Collaboration found that for every woman who had her life extended by early detection and treatment, there would be 10 courses of unnecessary treatment in healthy women, and more than 200 women would experience distress as the result of a false positive.

Bayesian reasoning has implications far beyond cancer screening, and we are not natural Bayesians. Daniel Kahneman, a psychologist who won the Nobel memorial prize in economics, discusses the issue in a new book, Thinking, Fast and Slow. I recently had the opportunity to quiz him in front of an audience at the Royal Institution in London. Kahneman argues that we often ignore baseline information unless it can somehow be made emotionally salient. New information – “possible cancer” – tends to monopolise our attention.

Another example: if somebody reads the Financial Times, should you conclude that they are more likely to be a quantitative analyst in an investment bank, or a public sector worker? Before you leap to conclusions, remember that there are six million public sector workers in the country. Base rates matter.

Sometimes there is no objective base rate and we must use our own judgment instead. I think homeopathy is absurd on theoretical grounds; others find it intrinsically plausible. Bayesian analysis tells us how to combine those prior beliefs – or prejudices – with whatever new evidence may come along.

Whenever you receive a piece of news that challenges your expectations, it’s tempting either to conclude that everything has changed – or that nothing has. Bayes taught us that there’s a rational path between those two extremes.

The value of mathematics, or logic, and rational thinking is that it gives you tools to work through the muddle of real life. Even models can be useful to provide a guide in a murky area. But the trick is to always realize that these tools are idealizations. The actual world isn't mathematical, logical, rational, or capturable in a model.

Math & Art & Science

Here's a nice video that combines humour and math with a little mythical earth history:



I especially enjoy the little bits of realism, like getting eaten by the Pythagasaurus. The problem with nerdy kids is that they don't have a healthy fear of the real world. Their math & science gives them a false sense of power and authority. The nice thing about art is that is can reintroduce these aspiring minds to the fact that math kills, like the Pythagasaurus. A little healthy fear is good for the budding scientist. And only the magic of art can help groom that fear.

Paul Krugman Despairs the Economics is Not a Science

Paul Krugman has devoted his career to economics and has won a Nobel Prize, but he fells that he is living through a "Dark Age" in which economics has unlearned the lessons of the past. He is really despondent. Here is the relevant piece from a post on his NY Times blog:

I’ve never liked the notion of talking about economic “science” — it’s much too raw and imperfect a discipline to be paired casually with things like chemistry or biology, and in general when someone talks about economics as a science I immediately suspect that I’m hearing someone who doesn’t know that models are only models. Still, when I was younger I firmly believed that economics was a field that progressed over time, that every generation knew more than the generation before.

The question now is whether that’s still true. In 1971 it was clear that economists knew a lot that they hadn’t known in 1931. Is that clear when we compare 2011 with 1971? I think you can actually make the case that in important ways the profession knew more in 1971 than it does now.

I’ve written a lot about the Dark Age of macroeconomics, of the way economists are recapitulating 80-year-old fallacies in the belief that they’re profound insights, because they’re ignorant of the hard-won insights of the past.

What I’d add to that is that at this point it seems to me that many economists aren’t even trying to get at the truth. When I look at a lot of what prominent economists have been writing in response to the ongoing economic crisis, I see no sign of intellectual discomfort, no sense that a disaster their models made no allowance for is troubling them; I see only blithe invention of stories to rationalize the disaster in a way that supports their side of the partisan divide. And no, it’s not symmetric: liberal economists by and large do seem to be genuinely wrestling with what has happened, but conservative economists don’t.

And all this makes me wonder what kind of an enterprise I’ve devoted my life to.

I'm stunned that the field has shown itself that incompetent in the face of the 2008 financial crisis. But the academics fell in love with their models, the math, and the simplifying assumptions required to make the math and the models work. They took their eye off what is critical to any real science: the facts. They've turned economics into a branch of theology where the divines of the field debate the number of angels dancing on the head of a pin. They've gotten away from the hard insights of Keynes from the Great Depression. Tragic.

Math Reality Rises Up and Bites You

Here is a talk about how the bloodless world of math has crept into the real world and now has risen up and is biting us, not just biting, gashing, chomping, and devouring use.

From a TED talk in July 2011...



The bit around 09:00 in the video where he talks about "the physics of culture" and worries about those algorithms crashing is really, really scary.

The future belongs to the algorithms... and we are hanging on tight hoping we are lost as the future races away.

Math Lesson

I taught high school math nearly 40 years ago. This video brings back vague memories of the experience...



By the way, I lunched at a Burger King today and had four big burly guys sit down next to me. They were in the forties and early fifties. They discussed their high school experience, the good teachers, the bad teachers. It seems that schooling is one of the most mind-searing experiences we undergo. Tragic.

The good news, at least I took it as good news, is that none of them bad mouthed a math teacher. But I would size them up as the kind of guys who took their last math course in tenth grade and haven't had a thought about math sense then. That's too bad. Math is actually a lot of fun if you can get into it. But too many people are browbeat in school and decide they "hate" math. I used to bring math games into class to give the kids a wider perspective. They certainly loved the games. But most were never keen on the math. They just didn't get the joy of factoring polynomials. They kept asking "what is the use of this?" The idea that it was isometric exercises for the mind didn't impress them.

Now who wouldn't say that topology is fun...

An Interesting Tale of Statistics

I never realized that most babies are born later than expected. I foolishly thought that they would be dispersed normally about the expected 9 month date. But I got a lesson in statistics today from Tim Harford talking to his young daughter:

“I thought you wanted to talk about babies, my dear. By the way, did I ever tell you that you have more than the average number of arms?”

“But I have two arms.”

“Precisely. The average number of arms is less than two.”

“Don’t you mean fewer?”

“No. I don’t.”

“But everyone has two arms.”

“Not everyone has two arms. Most people do. The most common number of arms is certainly two. And more than 50 per cent of people have two arms, so the median number of arms is also two.

“But if you add up all the arms and divide by the number of people – which is what we tend to mean when we say “average” – you’ll find that the result is slightly less than two.”

“That’s silly.”

“No – that’s statistics.”

“Statistics are silly.”

“Statistics are not silly. But many people use them in a silly way.”

“They seem silly to me. So babies are a bit like arms?”

“A little bit. Some babies come months early, but no baby comes months late, and in any case the doctors have a tendency to get twitchy and whip the little tykes out when they’re a couple of weeks overdue.”

“But doesn’t that mean babies are likely to be early rather than late?”

“No: it means that a few babies are very early but no babies are very late. And this baby is not going to be very early, since it’s almost July and there’s been no sign of it yet.

“Anyway – if a few babies are very early and none are very late, then if you are trying to figure out a due date by adding up all the lengths of all these pregnancies, the very early babies will pull the average to the early side.”

I'm always delighted when people surprise me and show me that I didn't know something, especially when I thought I knew something!

Teaching Math

For a few years I taught math in high school many, many years ago. I wish I had been half... no, even a smidgen as inventive as Vi Hart is in teaching math concepts.

Here's a wonderful video to show deep connections between various techniques to multiply:



Go spend some time wandering around the amazing and beautiful things she has on her web site.

A Little Math Expression

Truth through mathematics...



Who said beauty is truth and truth is beauty? Truth is math and math is beauty.

Fun with Math

Here is a short entertaining but educational video on how mathematicians can turn a sphere inside out without tearing or pinching it:

The Collapse and Failure of Modern "Economics"

Here is a bit from an article in Foreign Affairs magazine which is reprinted in the blog Crooked Timber. The article looks at how monetary and economic policy in Europe is failing as that continent falls under the sway of "austerity" and the incompetence of "modern" economics:

The EU is now drifting toward a thinly disguised version of the gold standard, which wreaked economic havoc in the 1920s and led to a toxic political fallout. Under that system, European states had fixed exchange rates. During economic crises, they refused to increase government spending because of a failure to either understand or care that monetary disturbances and shocks to demand could lead to joblessness. The result was generalized misery. Governments responded to economic crises by allowing unemployment to go up and cutting back wages, leaving workers to bear the pain of adjustment. As Golden Fetters, Barry Eichengreen’s classic history of the period, shows, the gold standard began to collapse when workers in Europe gained the power to vote out of office the parties that supported austerity.

...

This approach cannot be sustained for long. The EU has never had much popular legitimacy: many voters have gone along with it so far only out of the belief that their politicians knew best. Today, they are more suspicious. And if they come to think that further European integration is causing more economic hardship, their suspicion could harden into bitterness and perhaps even xenophobia. Ireland’s new finance minister, Michael Noonan, has told voters that the EU is a game rigged in Germany’s favor; editorials in major Irish newspapers warn of Germany’s return to racist imperialism. As economic shocks hit other EU countries, politicians in those states will also look for someone to blame.

If the EU is to survive, it will have to craft a solution to the eurozone crisis that is politically as well as economically sustainable. It will need to create long-term institutions that both minimize the risk of future economic crises and refrain from adopting politically unsustainable forms of austerity when crises do hit. They must offer the EU countries that are the worst hit a viable path to economic stability while reassuring Germany, the state currently driving economic debates within the union, that it will not be asked to bail out weaker states indefinitely.

...

Contrary to the beliefs of nearly all anti-Keynesians—and, regrettably, some Keynesians, too—Keynesianism demands more, not less, fiscal rectitude in normal times than does the orthodox theory of balanced budgets that underpins the EU. John Maynard Keynes argued that surpluses should be accumulated during good years so that they could be spent to stimulate demand during bad ones. This lesson was well understood during the golden age of Keynesian social democracy, after World War II, when, aided by moderate inflation, the governments of the countries in the Organization for Economic Cooperation and Development greatly reduced their ratios of public debt to GDP. This approach should not be confused with the opportunistic support for large budget deficits evident, for example, among advocates of supply-side economics. If anything, “hard” Keynesianism suggests that the problem with the macroeconomic rules governing the euro is not that they are too tough and too detailed but that they are not tough or detailed enough. States in the eurozone should not be allowed to run moderate budget deficits in boom years, the Keynesian argument goes; instead, they should be compelled to run budget surpluses. The surpluses could then be saved in rainy-day funds or used to pay down government debt or, if the country had reached a satisfactory debt-to-GDP ratio, spent as a fiscal stimulus in the event of a crisis. Unlike the kind of budget management advocated by the German government, this approach does not seek to eliminate or minimize governments’ leeway to conduct fiscal policy. It gives governments up-front the means to manage demand whenever they might need to.

The call for "hard Keynesianism" is the right call. It is the call that points out that the last 80 years of "economics" has been a disaster and a mistake built around lovely mathematics but an idiotic understanding of markets and humans.

I think it is funny. The call for "hard Keynesianism" is the call to go back to Joseph in Egypt in 1000 BC when during the 7 fat years the Pharaoh stored up surpluses from the land to feed the people during the 7 lean years. That "modern" economics doesn't understand this need to lean against the wind says volumes about the incompetence of "modern economics".

James Gleick's "The Information: A History, A Theory, A Flood"

This is a monster of a book. It is a vast landscape in which there are sections of pure delight and others where the writer appears to have gone a bit too far and ended up beached and lost. The bits about "information flood" and especially the many occasions on which Gleick turns to Jorge Luis Borges’s story "The Library of Babel" is a bit much for my taste. I found nothing useful on the musings about infinite random texts or the historial worries about "too much information". On the whole, however, the book is well worth reading.

For me, the meat-and-potatoes of this book are the sections on logic, cryptology, information theory, and quantum information. The other historical bits of fun to read but distracting. The real value in this book is that it offers a nice historical review of the rise of understanding about symbol manipulation and the mathematics of "information". The vignettes about key historical figures adds an excellent layer to the technical account.

The good news is that this book is accessible to anyone with a modest technical background. It doesn't try to teach you theories or delve into the mathematics, but it gives you a good appreciation of this technical side of things. The weaving of a story through several technical displines and their key personalities gives you good insight into how the "science of information" has developed and its trajectory into the future via the great 20th century physicist John Archibald Wheeler's maxim of "it from bit".

An excellent review of this book by Freeman Dyson can be found here as well as links to subsidiary information about Gleick.

Update 2011aug22: A less useful review is available via the NY Times Sunday Book Review from Geoffrey Nunberg:

Gleick ranges over the scientific landscape in a looping itinerary that takes the reader from Maxwell’s demon to Godel’s theorem, from black holes to selfish genes. Some of the concepts are challenging, but as in previous books like “Chaos” and “Genius,” his biography of Richard Feynman, Gleick provides lucid expositions for readers who are up to following the science and suggestive analogies for those who are just reading for the plot. And there are anecdotes that every reader can enjoy: Shannon building a machine called Throbac I that did arithmetic with Roman numerals; the Victorian polymath Charles Babbage writing to Tennyson to take exception to the arithmetic in “Every minute dies a man / Every minute one is born.”

Nunberg provides a literate review but not a particularly insightful.

Compare the above to Freeman Dyson's succinct summary:

According to Gleick, the impact of information on human affairs came in three installments: first the history, the thousands of years during which people created and exchanged information without the concept of measuring it; second the theory, first formulated by Shannon; third the flood, in which we now live. The flood began quietly. The event that made the flood plainly visible occurred in 1965, when Gordon Moore stated Moore’s Law. Moore was an electrical engineer, founder of the Intel Corporation, a company that manufactured components for computers and other electronic gadgets. His law said that the price of electronic components would decrease and their numbers would increase by a factor of two every eighteen months. This implied that the price would decrease and the numbers would increase by a factor of a hundred every decade. Moore’s prediction of continued growth has turned out to be astonishingly accurate during the forty-five years since he announced it. In these four and a half decades, the price has decreased and the numbers have increased by a factor of a billion, nine powers of ten. Nine powers of ten are enough to turn a trickle into a flood.

Nunberg is for artistic dilettantes, Dyson is for the scientific types.

Is the Glass Half Full or Half Empty?

Here's a good post by Paul Krugman showing that perspective matters. First, from the blogger Calculated Risk's viewpoint, America is now "half recovered" from the big losses of the 2008 financial meltdown:

Calculated Risk has gone through the latest Flow of Funds report, which shows that households have made up about half of the loss in net worth they suffered in the crisis. We’re almost home!

Or not.

The central insight of deleveraging models is that the distribution of wealth matters — specifically, that what is weighing down the economy in the aftermath of a Minsky moment is the debt of highly-indebted agents, which is not offset by the assets of creditors, because the debtors are forced to spend less while the creditors aren’t forced to spend more.

So, whose net worth has improved? Most of the gain reflects the recovery of the stock market — and highly indebted households are not major stock investors. The main asset price affecting debtors’ financial position, the price of housing, has not improved.

That’s not to say that rising stock prices have no effect; they do encourage higher spending — but not remotely to the extent that an equal-value fall in debt would. So the news on net worth isn’t nearly as positive as it seems. Full recovery is still a long, long way off.

The fact that most people's "assets" are tied up in homes and not stocks and home prices continue to fall means that the financial position of most people is at best bottomed out and at worst still falling.

This shows that statistics can be very deceptive. I like the story of a bunch of down at their luck guys at a bar in Seattle grousing about their financial situation. In wall Bill Gates and suddenly the average net worth of the 20 guys there jumps from $100,000 to $2 billion! Sure, statistically this is true. But the fact is that 20 guys only have $100,000 and one guy as $40 billion. The "average" is a figment of statistics. There is nothing real about the improvement of "average net worth" for the 20 guys.

Freeman Dyson's Review of a James Gleick Book

The NY Review of Books has a wonderful review of James Gleick's new book The Information: A History, a Theory, a Flood written by one of my favourite mathematicians, Freeman Dyson. Here are some key bits:

The consequences of the information flood are not all bad. One of the creative enterprises made possible by the flood is Wikipedia, started ten years ago by Jimmy Wales. Among my friends and acquaintances, everybody distrusts Wikipedia and everybody uses it. Distrust and productive use are not incompatible. Wikipedia is the ultimate open source repository of information. Everyone is free to read it and everyone is free to write it. It contains articles in 262 languages written by several million authors. The information that it contains is totally unreliable and surprisingly accurate. It is often unreliable because many of the authors are ignorant or careless. It is often accurate because the articles are edited and corrected by readers who are better informed than the authors.

Jimmy Wales hoped when he started Wikipedia that the combination of enthusiastic volunteer writers with open source information technology would cause a revolution in human access to knowledge. The rate of growth of Wikipedia exceeded his wildest dreams. Within ten years it has become the biggest storehouse of information on the planet and the noisiest battleground of conflicting opinions. It illustrates Shannon’s law of reliable communication. Shannon’s law says that accurate transmission of information is possible in a communication system with a high level of noise. Even in the noisiest system, errors can be reliably corrected and accurate information transmitted, provided that the transmission is sufficiently redundant. That is, in a nutshell, how Wikipedia works.

I love the idea of bottom-up knowledge. Too much of our life is controlled by top-down autocrats busy trying to mold us to fit their version of what we should be ("proper" citizens, "dutiful" servants, patient and loyal consumers).

And Dyson catches the joys of science:

The information flood has also brought enormous benefits to science. The public has a distorted view of science, because children are taught in school that science is a collection of firmly established truths. In fact, science is not a collection of truths. It is a continuing exploration of mysteries. Wherever we go exploring in the world around us, we find mysteries. Our planet is covered by continents and oceans whose origin we cannot explain. Our atmosphere is constantly stirred by poorly understood disturbances that we call weather and climate. The visible matter in the universe is outweighed by a much larger quantity of dark invisible matter that we do not understand at all. The origin of life is a total mystery, and so is the existence of human consciousness. We have no clear idea how the electrical discharges occurring in nerve cells in our brains are connected with our feelings and desires and actions.

Even physics, the most exact and most firmly established branch of science, is still full of mysteries. We do not know how much of Shannon’s theory of information will remain valid when quantum devices replace classical electric circuits as the carriers of information. Quantum devices may be made of single atoms or microscopic magnetic circuits. All that we know for sure is that they can theoretically do certain jobs that are beyond the reach of classical devices. Quantum computing is still an unexplored mystery on the frontier of information theory. Science is the sum total of a great multitude of mysteries. It is an unending argument between a great multitude of voices. It resembles Wikipedia much more than it resembles the Encyclopaedia Britannica.

I love the great "democratic" thrusts of our age: science, democratic governance, popular culture, Wikipedia, etc. These are expressions of individuals that rise up and catch wind if enough people are attracted, or they can fail -- like democracy in Germany in 1933 -- if not enough people are willing to stand up for individual rights and individual beliefs.

And this bit shows Dyson's great optimism:

The explosive growth of information in our human society is a part of the slower growth of ordered structures in the evolution of life as a whole. Life has for billions of years been evolving with organisms and ecosystems embodying increasing amounts of information. The evolution of life is a part of the evolution of the universe, which also evolves with increasing amounts of information embodied in ordered structures, galaxies and stars and planetary systems. In the living and in the nonliving world, we see a growth of order, starting from the featureless and uniform gas of the early universe and producing the magnificent diversity of weird objects that we see in the sky and in the rain forest. Everywhere around us, wherever we look, we see evidence of increasing order and increasing information. The technology arising from Shannon’s discoveries is only a local acceleration of the natural growth of information.

The visible growth of ordered structures in the universe seemed paradoxical to nineteenth-century scientists and philosophers, who believed in a dismal doctrine called the heat death. Lord Kelvin, one of the leading physicists of that time, promoted the heat death dogma, predicting that the flow of heat from warmer to cooler objects will result in a decrease of temperature differences everywhere, until all temperatures ultimately become equal. Life needs temperature differences, to avoid being stifled by its waste heat. So life will disappear.

This dismal view of the future was in startling contrast to the ebullient growth of life that we see around us. Thanks to the discoveries of astronomers in the twentieth century, we now know that the heat death is a myth. The heat death can never happen, and there is no paradox. The best popular account of the disappearance of the paradox is a chapter, “How Order Was Born of Chaos,” in the book Creation of the Universe, by Fang Lizhi and his wife Li Shuxian. Fang Lizhi is doubly famous as a leading Chinese astronomer and a leading political dissident. He is now pursuing his double career at the University of Arizona.

The belief in a heat death was based on an idea that I call the cooking rule. The cooking rule says that a piece of steak gets warmer when we put it on a hot grill. More generally, the rule says that any object gets warmer when it gains energy, and gets cooler when it loses energy. Humans have been cooking steaks for thousands of years, and nobody ever saw a steak get colder while cooking on a fire. The cooking rule is true for objects small enough for us to handle. If the cooking rule is always true, then Lord Kelvin’s argument for the heat death is correct.

We now know that the cooking rule is not true for objects of astronomical size, for which gravitation is the dominant form of energy. The sun is a familiar example. As the sun loses energy by radiation, it becomes hotter and not cooler. Since the sun is made of compressible gas squeezed by its own gravitation, loss of energy causes it to become smaller and denser, and the compression causes it to become hotter. For almost all astronomical objects, gravitation dominates, and they have the same unexpected behavior. Gravitation reverses the usual relation between energy and temperature. In the domain of astronomy, when heat flows from hotter to cooler objects, the hot objects get hotter and the cool objects get cooler. As a result, temperature differences in the astronomical universe tend to increase rather than decrease as time goes on. There is no final state of uniform temperature, and there is no heat death. Gravitation gives us a universe hospitable to life. Information and order can continue to grow for billions of years in the future, as they have evidently grown in the past.

I thoroughly enjoyed Freeman Dyson's review. Now I can look forward to reading James Gleick's book. I've got myself signed up for the first copy at the local library!

This review points at an article by James Gleick that was published too late to be included in his book. It was published as the article "The Information Palace" in the NY Review of Books:

The word “information” has grown urgent and problematic—a signpost seen everywhere, freighted with new meaning and import. We hardly need the lexicographers of the Oxford English Dictionary to tell us that, but after all, this is what they live for. It is a word, they tell us, “exhibiting significant linguistic productivity,” a word that “both reflects and embodies major cultural and technological change,” therefore a word crying out for their attention. In their latest quarterly revision, December 2010, just posted, the entry for “information” is utterly overhauled. (The OED, in case you hadn’t noticed, has evolved into an enterprise of cyberspace, rather than a mere book.)

The renovation has turned a cottage into a palace. Information, n., now runs 9,400 words, the length of a novella. It is a sort of masterpiece—an adventure in cultural history. A century ago “information” did not have much resonance. It was a nothing word. “An item of training; an instruction.” Now (as people have been saying for fifty years) we are in the Information Age. Which, by the way, the OED defines for us in its dry-as-chili-powder prose: “the era in which the retrieval, management, and transmission of information, esp. by using computer technology, is a principal (commercial) activity.”

...

Originally—and by originally I mean in the fourteenth century, when the written record begins, as far as the OED can tell—the word had a sinister flavor. It budged its way into the old gruff Anglo-Saxon as part of the Norman invasion. It meant something like “accusation” or “incrimination.” The earliest citation comes from the Rolls of Parliament for 1386: “Thanne were such proclamacions made‥bi suggestion & informacion of suche that wolde nought her falsnesse had be knowen to owre lige Lorde.” For centuries thereafter, informations were filed, or recorded, or laid, against people.

From then to now the word takes a twisty path, and the OED‘s lexicographers hold our hand around every corner. Information can be “a teaching; an instruction.” It can be “divine influence or direction; inspiration, esp. through the Holy spirit.” It can be “that of which one is apprised or told; intelligence, news.”

And this bit of sleuthing in the OED gives Gleick:

By the way, how do we know people have been talking about the Information Age for fifty years? The OED tells us. The first recorded usage is attributed to “R. S. Leghorn in H. B. Maynard Top Managem. Handbk. xlvii. 1024,” 1960. He turns out to have been Richard Leghorn, founder of Itek Corporation, which made aerospace spy cameras, and later Chief of Intelligence and Reconnaissance Systems Development at the Pentagon. In a single sentence Leghorn invented the phrase and predicted it would not catch on:

Present and anticipated spectacular informational achievements will usher in public recognition of the “information age,” probably under a more symbolic title.

No better title has come along. Along with information age, the OED now recognizes information storage, information transfer, information processing, information retrieval, information architecture, information superhighway, plus (the bad news) information explosion, gap, warfare, overload, and fatigue.

Unique and Interesting People

My strategy in life is to travel with the herd using their numbers to protect me. But I've always been fascinated with characters who take risks, insist on "doing their thing", make their own path through life, and end up doing astounding things. Here's an interesting person pointed out by Barry Ritholtz in his The Big Picture blog. This is the fascinating story of James Simons:

Don’t expect to glean any market tips or trading secrets from James Simons, who steadfastly refuses to disclose the method behind his remarkable record in investing. Instead, listen to this mathematician, hedge fund manager and philanthropist sum up a remarkably varied and rich career, and offer some “guiding principles” distilled along the way.

Simons drew a bead on studying math at MIT from an early age, which some acquaintances found surprising. As a 14-year-old, he was demoted in a temporary job from stockroom worker to floor-sweeper, because he “couldn’t remember where in hell everything went.” This switch suited him fine, since he had “lots of time to think.” When he told his employers he hoped to attend MIT, “they thought it was the funniest thing.” Ultimately, Simons had no choice about it: After Wesleyan recruited, then rejected him, there was only MIT. “I was destined for this place,” he says.

The idea of a math career was “clinched” for Simons after a typical late night of poker and sandwiches with MIT classmates. At 1 a.m. in a Brookline restaurant, Simons saw MIT math legends Isadore Singer and Warren Ambrose “doing math over coffee and cigarettes,” which he “thought was the coolest thing.” After a motor scooter trip to Bogota with Colombian friends — in whose business he fatefully invested — Simons leapt into the math phase of his career, writing a famous Ph.D. thesis, teaching at MIT, solving prickly geometry problems and helping build bridges between math and physics. During this phase, he managed to get fired as a cryptanalyst at a Defense Department think tank, after criticizing the pro-Vietnam War stance of his boss, General Maxwell Taylor.

While at Stony Brook’s math department, Simons “got really stuck, very frustrated,” trying “to prove a certain number was irrational.” Meanwhile, he had begun investing dividends generated by his South American business venture and “found out I was not bad at it.” In 1978 at age 38, with 20 years behind him as a mathematician, he concluded it was time for a change. He began an investment business, Renaissance Technologies, that deployed sophisticated, proprietary models to generate astonishing returns (and business envy) over many years. “We have a lot of smart guys,” he comments.

After his retirement in 2009, Simons got “busy as hell” with his third career. The Simons Foundation supports basic math and physics as well as autism research. Simons also wants to improve math at the high school level, by pumping money into teaching jobs so talented people don’t drift to “Google or Goldman Sachs.”

Simons says he is “always doing something new,” and doesn’t like to run with the pack. This approach, which he recommends, “gives you a chance.” Some other parting tips: collaborate with the best people you possibly can; try at problems “for a hell of a long time;” be guided by beauty; and “hope for some good luck.”

I have the book The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed it by Scott Patterson which apparantly has much to say about James Simons, but I have not yet read it.

Book Banned in China

This is just too much...

I received the following in email from our publisher:

I write with regards to the project to publish a China Edition of your book "Data Analysis Using Regression and Multilevel/Hierarchical Models" (ISBN-13: 9780521686891) for the mainland Chinese market. I regret to inform you that we have been notified by our partner in China, Posts & Telecommunications Press (PTP), that due to various politically sensitive materials in the text, the China Edition has not met with the approval of the publishing authorities in China, and as such PTP will not be able to proceed with the publication of this edition. We will therefore have to cancel plans for the China Edition of your book. Please accept my apologies for this unforeseen development. If you have any queries regarding this, do feel free to let me know.

Oooh, it makes me feel so . . . subversive. It reminds me how, in Sunday school, they told us that if we were ever visiting Russia, we should smuggle Bibles in our luggage because the people there weren't allowed to worship.

Xiao-Li Meng told me once that in China they didn't teach Bayesian statistics because the idea of a prior distribution was contrary to Communism (since the "prior" represented the overthrown traditions, I suppose).

And then there's this.

I think that the next printing of our book should have "Banned in China" slapped on the cover. That should be good for sales, right?

I'm thinking of rushing out to buy a copy so that I can prop it up in a position of honour in my house to allow me to regale visitors with stories of the heroic stance I'm taking against Communist Chinese oppression of the free flow of ideas. Everybody needs to be aware that Bayesian statistics is deeply subversive and therefore should be embraced!

I'm thinking of buying a hundred and handing them out as Christmas presents. That way I can form a clandestine cell with my hundred closest friends dedicated to freeing the oppressed Chinese from their tyrannical overlords, the CCP. I can already sense the politburo in Beijing quaking in their boots!

Mathematicians of the world, rise up! All we have to lose are our prior, conditional, and marginal probabilities and the chance to gain a new world with our posterior probabilities!

I am convinced we can defeat Communism if we all arm ourselves with one of these...



This is the new secret weapon system that America is planning to roll out by the millions all along its Pacific border to protect citizens from the Yellow Peril!

Charles Seife's "Proofiness: The Dark Arts of Mathematical Deception"

This is an excellent look at how statistics is being mangled and manipulated by people who know better to achieve results that border on or go beyond the criminal. The first couple of chapters are dedicated to giving you a quick background in math and statistics so that you can see how these tools can be used to manipulate you. The really interesting stuff comes later.

I really sat up and started paying attention in Chapter 4 which looks at how polls are used to manipulate us. He has a lot of interesting background and historical examples of polls that mispredicted elections and he explains why this occurs and why it is bound to occur. He goes into fair depth to explain why a 1936 poll by Literary Digest which got responses from millions of Americans could so badly predict the FDR vs Alf Landon presidential vote. He looks at the 1948 Truman vs. Dewey polling that "predicted" the election of Thomas Dewey.

In Chapter 5 he delves into even more fascinating material by looking at the 2008 Al Franken vs. Norm Coleman senatorial race in Minnesota. I bet you haven't heard of the "lizard people" but you really must read this chapter to understand their crucial role in that election. He also looks at the 2000 "hanging chad" controversy of the 2000 Bush vs. Gore presidential election. You might think you "know" about these elections and what went wrong, but I assure you, until you read Seife's book you won't truly understand them. Just this chapter alone is worth getting the book. Excellent stuff!

Chapter 6 delves into the history of gerrymandering and how that is undermining democracy in the US. Again, this chapter alone justifies getting this book and reading it!

Proofiness has undermined the very foundations of our democracy -- the mechanisms that we use to count our citizens and ensure that they are represented in the Republic. Gerrymandering for political gain is deemed acceptable, even though it clearly dilutes the votes of some of our citizens. Statistical sampling is deemed unacceptable, even though rejecting it forces the government to use numbers that it knows are inaccurate. No matter how many intellectual backflips legislators and judges go through to justify their positions, the fact remains: bad mathematics is being used to deny our citizens -- mostly our minorities -- their rightful vote. In a democracy, there can be no graver sin.

In Chapter 7 he looks at some of the more egregious attempts by legislatures to use laws to redefine reality. In 1897 the Indiana state legislature passed a bill to define the mathematical constant pi to be 3.2. This was speeding toward senate approval and the governor's signature until the Purdue University's math department explained that passing this law wouldn't change mathematical reality but it sure would make the state of Indiana a laughingstock around the world.

Indiana's abortive attempt to rewrite the laws of nature is absurd, but it's not an isolated incident. Our governments misue mathematics in subtler ways all the time, trying to banish facts that are embarrassing and inconvenient. And the most mind-bending denials of mathematical reality come from the branch of government that's supposed to be the guardian of truth: the judiciary.

He then goes on to document incredible judicial decisions -- life and death decisions -- taken by US courts, all the way up to the Surpreme Court, and not just in the 19th century but the 20th and now the 21st century -- that are just as incredibly wrong-headed at that Indiana attempt to redefine pi.

He looks at the infamous 1944 paternity suit brought against Charlie Chaplin. It was a "he said, she said" case until they did a blood test. The result? Chaplin was type O. The child was type B. The Mother type A. There was absolutely no way that Chaplin could be the father of the baby. Did that scientific fact stop the judicial process. Did hard reality force the judge to face facts? No!

It didn't matter. The prosecutor -- and the courts -- rejected the notion that the blood test was conclusive and pressed on with the trials, using such scientific evidence as a "resemblance" test, where jurors gazed intently at the baby to determine whether she looked like Chaplin. The Mann Act charges at first ended in a mistrial and then in an acquittal; however, in his paternity trial, Chaplin was found to be the father. The court's truth -- the legal truth -- was that Chaplin was the father of Joan Berry's baby, even thought the Truth, with nearly absolute mathematical certainty, was otherwise. And the prosecutors and the judges knew it.

Shades of the O.J. Simpson trial. (In fact, the book has a good discussion of that trial as well and the misuse of math by the defense to trick the jury into finding Simpson innocent.)

In Chapter 8 he looks at how the "Right to Life" crowd cynically uses misinformation and untruths to push its ideological agenda. They have seized on a published "research" article that linked abortion with an increased risk of breast cancer:

The link between abortion and breast cancer is randumbness; people saw a pattern when there really wasn't one. In 2003, the National Cancer Institute held a conference on the subject -- at the end of the meeting, the report stated that it was well established, scientifically, that abortions did not increase the risk of breast cancer. (The report was endorsed by all the conference except for one: Joel Brind [the original "researcher"].) The scientific consensus was as clear as it could be; the link was fictional. Yet a few months later, Texas passed the Worman's Right to Know Act. Abortion doctors around the state were forced to put a phony doubt into their patients' minds -- doubt about whether they'll get breast cancer because of an abortion.

This is a blatant misuse of statistics to advance a religious/political viewpoint. Here's the truth behind the phony "research" and its bad math:

According to a 2005 Finnish study, women who terminate their pregnancies are about six times more likely to kill themselves. However, the same study also found that women who have abortions have a higher risk of being murdered -- ten times more likely than those who went through with the birth. They were also 4.4 times more likely to die in an accident. Thse statistics hint at what's really going on here: causuistry. It's not that abortions are causing people to commit suicide, any more than abortions are causing people to get murdered or have accidents. Women who are in a high-risk category are almost certainly more likely to get into a situation where they need an abortion than those in the general population. Mentally ill people, risk-takers, women who run with a wild crowd -- they are more likely to get into trouble of all sorts, including having an unwanted pregnancy. Conversely, women who have a child are more likely to be more stable and responsible, and less likely to take needless risks. Yet antiabortion lawmakers turn that correlation into causation, forcing doctors to scare their patients with the idea that going through with the abortion might cause them to commit suicide later on. The "right to know" laws have nothing to do with knowledge. They are vehicles for whatever bits of proofiness can be deployed to support the lawmakers' opinions.

He finishes with a final chapter on propaganda and points out:

Proofiness is toxic to a democracy, because numbers have a hold on us. They are powerful -- almost mystical. Because we think that numbers represent truth, it's hard for us to imagine that a number can be made to lie. Even the oafish Joe McCarthy knew this; when he declared that 205 communists had infiltrated the State Department, his outrageous falsehood was given the appearance of absolute fact. But proofiness is not merely a tool for propaganda as as it was for McCarthy -- it is much more dangerous than that. Democracy is a system of government based upon numbers, and rotten numbers are eroding the entire edifice from within. Proofiness in America is used to disenfranchise voters and to bias elections. It's used to drain money from our treasury and put it in the pockets of unscrupulous businessmen. Proofiness tilts the scales of justice and helps condemn the innocent. It puts lies in the mouths of reporters. There's no institution in a democracy that's immune.

How to Really Know the World

Here is an fine video by Hans Rosling that looks at how statistics can help us understand the world. More strictly, it is about how data visualization techniques can help us understand the world:



I love the bit where he states that "almost every Swede has more legs than the average Swede". That's his way of introducing you into more sophisticated statistics than simple averages that try to summarize phenomena in a single number.

You can think of the above as an hour long infomercial for Rosling's wonderful site that lets you explore data: Gapminder.org.

The Joy of Stumbling Upon Fun Math

Here's an interesting web site by a young woman, Vi Hart, interesting in many things including mathematics.

Here are some doodles with lines and loops and interesting mathetmatical properties...



Go check out her web site. She has many other interesting videos, photos, papers, and stuff!

Stephen Baker's "The Numerati"

This book is a fun read. It discusses the trend toward data collection and data mining to understand humans. The author gives examples of this technology applied to the workplace, to shopping, to politics, terrorist hunting, medicine, and using a online dating service. It will acquaint you with the technology, its possibilities, and the dangers. It won't give you enough information to understand how the technology works or to really appreciate where, how, and to what extent this technology can be applied.

The writers style is breezy. He intrudes his own persona into the narrative. I especially loved the bit about how he cajoled his wife to sign up with him for a dating service to see if that service could spot them as "potential mates". He sheepishly had to admit it didn't because he put his preference down for a younger woman. Only after he correct this did his wife show up in the list of candidates. Whoa! I'm a little shocked that he would admit to this. I can only imagine how much boxing around the ears he got from his wife. Especially when she discovers that the whole world now knows this about him.

The term "numerati" is used by him to cover those who collect and analyze data, build models, and predict human behaviour. Here's a bit from the book:

... today's Numerati are plowing forward, with an eye on us. They're already stitching bits of our data into predictive models, and they're just getting warmed up. In the coming decade, each of us will spawn, often unwittingly, models of ourselves in nearly every walk of life. We'll be modeled as workers, patients, soldiers, lovers, shoppers, and voters. In these early days, many of the models are still primitive, making us look like stick figures. The ultimate goal though, is to build versions of humans that are just as complex as we are -- each one unique.

The book builds the story from chapter to chapter of how we are being mapped, reduced to numbers, and turned predictable. But almost as a throwaway line at the very end of the book emerges an alternative view. One that I endorse from my experience in computer modeling. Here is a bit of a conversation he had with a friend with a doctorate in computer science on this vision of the Numerati:

...he explains that once he too dreamed of modeling the world but has since concluded that math, while powerful, is flawed.

"Why?"

"Ever heard of garbage in, garbage out?" His point is that mathematicians model misunderstandings of our world, often using the data at hand instead of chasing down the hidden facts.

There is a nugget of truth in this statement, but it is mangled. The data isn't garbage. It us simply unstructured, unqualified, and never completely understood. The "interpretation" that the Numerati are taking is purely statistical. So they will never be able to filter out the misleading and untrue. They simply hope that with large enough piles of data their analysis will approximate ever more closely the real underlying person. But people are maddenly unreliable. Humans learn. Once they know they are being "inspected" they change behaviour. Fads are a good example. Once the trendsetters realize that the "trend" is becoming too popular, they abandon it and start a new one. The data is not "static" because people change. So the data is unreliable, ambiguous, misleading, changeable, and sometimes just wrong. At best you get a crude approximation from this.

The techniques of data analysis, data mining, etc. work best for things that are dumb and static. They will be very useful in medicine, e.g. looking at diseases and genetics, but they will never be definitive in creating a model of a "shopper" or a "voter" or anybody else who can change their mind. The techniques will be useful, but they won't be definitive. You just can't pin people down.

I do recommend that you read the book. It is enjoyable. It will teach you some things. But take it with a grain of salt.

Benoit Mandelbrot Joins the Imaginary Numbers

The mathematician Benoit Mandelbrot has died. Here is the NY Times obituary.

Here's a lecture he gave at a TED conference in Feb 2010:



I remember attending a lecture of his at the University of British Columbia in the 1980s. What I remember most was his enthusiasm and his curious and interesting examples from his mathematics.

Update 2010oct17: Here is an article Remembering Benoit Mandelbrot by the mathematician Rudy Rucker. It is a reminiscence of going to Mandelbrot's house in 2001 to talk about helping put together an IMAX film on fractals. It has some nice graphics and a video clip touring a quintic Mandelbrot set.