How to Test Tests
A recent news story reports on tests of the cameras of four high end smart phones, giving high marks to the new iPhone. Reading it, it occurred to me that would be nice to have some measure of the reliability of reports of this sort.
In this case, there is a simple way to do it. Each image was evaluated by five independent judges; their conclusions were combined for the final result. It would be straightforward to calculate the standard deviation of the scores they produced and from that how likely it was that the difference between the phone that won and the phone that came in second was due to chance.