Blog: Numeracy

But everything we know isn't always true when you analyze statistics properly. Matt Bailey of SiteLogic uses the red-shirt phenomenon as an example to illustrate good statistical analysis, discovering that almost as many red-shirts died on the Enterprise as on planets, and most of those deaths occurred in groups. Most of all, he discovered that when Captain Kirk becomes romantically involved with a local alien woman, the red shirt survival rate skyrockets.

The article even includes a funny parody of classic lame PowerPoint statistical presentation.

Episodes take weeks or more to write and produce, so the writers likely based this one on the book, which came out six weeks ago. But it's surely just randomness that I saw the show days after hearing Levy speak. After all, if the two events hadn't coincided in time, I probably wouldn't have written a blog post about this. The fact that I did makes it noticed, and noticing coincidence makes us see the clusters all the more clearly even when they occur randomly.

Those with a good understanding of statistics will immediately realize that this is an inevitable human reaction to randomness. As Levy came to realize, "the pattern is not in the iPod, but in ourselves." Human brains are wired to recognize patterns, and they even see them where there are none. This is why some people are sure that they have a "feeling" before a disaster, or believe in ESP, or luck (as Levy points out, the gambling industry is built on people's irrational response to randomness). Not having clusters where one artist's song appears four times in a row is much more unusual than having some such clusters.

This is a great example of innumeracy among people in general, and why teaching statistics is so important.

One extremely common fallacy is confusing correlation with causation. If an alien were to come to earth and notice that on days when people wear raincoats it rains more often than on other days, that alien might conclude that raincoats cause rain. Of course, in reality the causation runs the other way. The only way to prove a causation is through a controlled experiment, where a scientist creates two groups of subjects at random, alters one attribute about one group, and then measures whether an observable result appears different in the experimental group versus a control group. This is how drug trials work, or Google user interface experiments. In social research, it's often not possible to, for example, ask half your subjects to go through a divorce just to see what happens to children, so researchers are limited to analyzing data and finding correlations without the ability to prove causation.

Many studies that find a correlation between two things either do a sloppy job of making it clear that they've only proven correlation, or journalists unfamiliar with the distinction confuse the two in articles. Just today I saw these two articles that confused correlation with causation:

First, BoingBoing cited a BBC story of a study that tracked subjects for two weeks and how much sex they had. At the end of the time period, they were put into stressful situations like public speaking. Those who engaged in penetrative sex did better, while those engaged in no sex or other types showed no change. But astute BoingBoing reader added this comment:

Just because people who have sex are less stressed doesn't mean that the sex causes them be that way. At least as plausible is that they have more sex _because_ they are less stressed and recover from stress more quickly— stress and staying steamed being a turn-off, in general, not to mention physiological effects of stress on performance, for men.

Second, University of Minnesota researchers claim that "having too many meetings and spending too much time in meetings per day may have negative effects". Again they collected data by having participants keep diaries, and again there doesn't appear to be much accounting for the preexisting biases that lead one participant to attend more meetings.

Did the data show that people who had more meetings overall were more stressed? The article mentions how managers spend more time in meetings. Maybe managers are more likely to be stressed than other people. Or did the diaries reveal that on days when people had more meetings, they were more stressed? Depending on the jobs they did (we don't know), it could be that people had a fairly consistent amount of work, but some days also had meetings layered on top, making those days more stressful.

Now, both these studies were conducted by professional social science researchers. And it's entirely possible that they carefully tried to account for all variables that would bias the results. But just from reading the articles, it's clear that either the article glosses over important details, or the researchers did.