King Tut, Statistics, and a Pet Peeve
I recently came across a National Geographic article describing the results from DNA analysis of a number of mummies, including King Tut. It was an interesting article, but there was one thing in it that annoyed me. In describing their results, the author said that the DNA analysis showed that there was a 99.9% probability of a particular relationship between two of the mummies.
That statement was false. I know it was false because that sort of analysis cannot produce that sort of result. Like many other people who use statistics without understanding it, the author was confusing the information the statistical analysis produced with the information he wanted it to produce.
A confidence result in classical statistics tells you how likely it is that you would get the result you got if the assumption you were testing was false—more precisely, if a particular alternative assumption, called the null hypothesis, was true. His 99.9% means that if the null hypothesis (presumably that the two mummies were not closely related—the article doesn't say) was true there is no more than a .1% chance that the genetic evidence that they were related would be as good as it is.
Unfortunately for the author of that article and many others, the probability of getting their result if their assumption is false is not the same thing as the probability that their assumption is false, given that they got their result. The latter is what they want, and what the assertion of a 99.9% probability for the relationship claimed. But it wasn't what they got.
To see the difference, consider a much simpler experiment. I pull a coin out of my pocket without looking at it. My theory is that it is a two headed coin; the null hypothesis is that it is a fair coin.
I flip it twice and it comes up heads both times. If it is a fair coin, the probability of that outcome is only 25%. If it is a two headed coin, it's 100%. If the probability of the result given the null assumption was the same thing as the probability of the null assumption given the result, that would mean that the odds were now three to one—75% probability—that the coin was double headed. I don't think so.
Readers interested in what it takes to actually generate a probability estimate for an assumption being true are invited to read up on Bayesian probability.
After writing this post, I discovered that I had mentioned the same point some time back in a different context.