Loaded Dice: How to bias research
In a recent Usenet thread, the question was raised of whether authoritarian personality types were more likely to be on the political right than on the political left. One contributor offered a link to a webbed book by Professor Robert Altmeyer which, the poster claimed, described scientific research that showed that the right was much more authoritarian than the left. I read the beginning of the book and concluded that it was indeed interesting--as an example of how to load the dice in order to get the results you want out of supposedly objective research.
The book starts off by defining "right wing authoritarian" (RWA) in a way which purports to be politically neutral; the author offers an implausible explanation of his entirely non-political reasons for labelling it "right wing." There follows the set of twenty questions (plus two that don't get scored) used to test subjects to see how RWA they are. On each question, the responder is supposed to express a view from "strongly agree" to "strongly disagree." The grading is simple--on some questions you count as more authoritarian the more strongly you agree, on the rest you count as more authoritarian the more strongly you disagree.
What is almost immediately obvious if you read the questions is that they aren't testing for RWA as the author defines it but for a combination of that and right/left political views. When the question is of the form "people who campaigned for unpopular causes X, Y and Z were good," X, Y and Z just happen to be causes more popular on the left than on the right. When the question is of the form "We should follow authority X," X just happens to be a source of authority, such as the church, more popular on the right than on the left. No questions about people who campaigned for unpopular right wing causes or about deferring to sources of authority popular on the left.
Perhaps the worst question of all was:
6. Atheists and others who have rebelled against the established religions are no doubt every bit as good and virtuous as those who attend church regularly.
Almost nobody taking the test--my guess is literally nobody--has the data needed to know whether atheists are less virtuous, more virtuous, or just as virtuous as churchgoers. The only reason I can see why someone would respond with "strongly agree" is that he is an authoritarian accepting the authority of a particular subgroup within his society--one that is skeptical of religion. But that is supposed to be the extreme non-authoritarian answer to the question.
Interested readers can find a more detailed critique by me on usenet; the thread, including responses, is here.