Prediction vs Explanation
A commenter on an earlier post raised the question of whether predicting something in advance is different in some important way from explaining it after the fact. I think it is, I think the reason is interesting, hence this post.
Suppose someone does ten experiments and comes up with a theory that is consistent with the results of those ten and predicts the outcome of ten more experiments, none of which has been done yet. The experiments are done and the predictions are correct.
Someone else looks at the results of the experiments and creates a theory consistent with all twenty.
We now do one more experiment, for which the two theories give different predictions. Are they equally likely to be correct, and if not why?
Let me start with the obvious argument to show that the two theories are equally good. There are lots of possible theories to deal with the subject of the experiments. All we know about the two candidates theories is that each is consistent with the first twenty experiments. Hence they are equally likely to be correct.
Imagine that each possible theory is written on a piece of paper, and pieces of paper are sorted into barrels according to the results they predict for the various experiments. The first theorist restricted himself to the barrels containing theories consistent with the first ten experiments, drew one theory from one of those barrels, and it happened to be from the barrel containing theories also consistent with the next ten experiments. The second experimenter went straight to that barrel and drew a theory from it.
What is wrong with this model is the implicit assumption that experimenters are drawing theories at random. Suppose we assume instead, as I think much more plausible, that some people are better at coming up with correct theories, at least on this subject, than others. Only a small fraction of the barrels contained theories consistent with the second ten experiments, so it would be very unlikely for the first experimenter to have chosen one of those barrels by chance. It's much more likely if he is someone good at coming up with correct theories. Hence his coming up with a theory in that barrel is evidence that he is such a person—increases the probability of it. We have no similar evidence for the second person, since he looked at the results of all twenty experiments before choosing a barrel.
Since we have more reason to believe that the first theorist is good at creating correct theories, or at least more nearly correct theories, than that the second one is, we have more reason to believe his theory and so more reason to trust his prediction for the next experiment.
Statisticians may recognize the argument as a version of spurious contagion. Picking the right barrel doesn't make the theorist any better, but the fact that he did pick the right barrel increases the probability that he was (even before picking it) a good theorist.