Authenticity, Status and SCA Culture

Also spelling bees, student testing, and other examples of Goodhart's law

[This post refers to the Society for Creative Anachronism, a  historical recreation group in which I have long been active.]

Some years ago I put  a post on one of the SCA Facebook groups asking whether "authenticity police," people who go up to a stranger to upbraid him for the supposed lack of authenticity of his garb, were mythical. I asked because, although stories of such incidents are common in the Society, neither I, my wife, nor my daughter have ever seen one — and the three of us have been in the Society for a combined span of about a century.

Reading the long thread that post spawned, I reached the following conclusions:

1. They are not mythical — such incidents sometimes occur.

2. I am not the only long time SCA participant who has never observed one.

3. Often the attack is bogus — the critic’s claim about what is or is not authentic is false.

4. A lot of people enjoy hearing and telling such stories, making them more common than the frequency of such incidents would explain.

All of which, I think, fits an unfortunate pattern.

The only required authenticity for an SCA event is some attempt at pre-17th century garb, a very low standard. But events frequently have contests in which entries are judged in part on how historically authentic they are and the Order of the Laurel, a high status rank in the Society, is awarded primarily for researching, demonstrating, and teaching period arts. The result is to associate the knowledge and practice of historical authenticity with status, which I think explains my four points.

The aggressor in an authenticity police story is simultaneously putting down his victim and pretending to expert knowledge. He chooses vulnerable targets, avoiding individuals who appear experienced and self confident and circles in which historical authenticity is taken seriously, which helps explain why some of us have never observed such an incident.

The SCA contains a minority of people who find it fun and interesting to try to figure out how things were done in our period and do them — make armor, cook from medieval recipes, compose music and poetry in period styles. It contains a larger number whose interest in authenticity is an attempt to do things the way they believe they are supposed to do them, an arbitrary rule of the game they are playing.

Suppose I am an active and productive participant who does not happen to have any interest in researching historical arts or using more of such research than necessary. I would like to defend my status against the feeling that I am failing at an important part of what the SCA is about. One way to do so is to convince myself that people who act interested in such things are only doing it as an excuse to put other people down or to gain rank. Repeating and elaborating authenticity police stories is one way of doing so. I can tell myself that people who are doing what I half feel I ought to be doing only do it to curry favor or as an excuse to push other people around and I would not want to be a person like that.

Another consequence of a culture that views the study and practice of authentic arts as largely a status game is to encourage people to pretend to an interest they do not have in the hope of being rewarded with status. Having gotten it in the form of a peerage, they are likely to reduce or terminate  activities that have now served their purpose.

How might one reduce the problem? One way is to replace contests with displays and classes. Arts contests treat period arts as a competitive game, encouraging the idea that historical authenticity only matters for contests. That point struck me long ago reading an SCA publication on a particular art that devoted a couple of pages to what was or was not period — preceded by the comment that this information would be needed by those entering contests. The clear implication was that it would otherwise not matter. And arts contests make little sense except as a competitive game; what does it mean to say that one entrant's sonnet is better or worse than another entrant's dress?

The Problem Generalized

Goodhart’s law: "When a measure becomes a target, it ceases to be a good measure"

To encourage something you create a measure of it and reward doing well by that measure. If the activity is learning to spell you organize spelling bees and give prizes for winning them. If it is good teaching you design standardized tests to measure what students have learned, require all students to take them, and reward teachers and school administrators according to how well their students do. If it is interest and skill in medieval arts you have arts contests, praise the winners, give them awards.

Rewarding the measure is an incentive to produce the measure, not the thing it is measuring. Good spelling is much more important for words people use routinely than for words used rarely but it is much easier to test the ability to spell difficult and obscure words than common ones. So spelling bees ask competitors to spell Ahuehuete, Ahura Mazda, ailette, aistopod, Aitutakian, none of which is likely to come up in anything they will ever write (Scripps National Spelling Bee: Words of the Champions). Knowing the spelling of hundreds of such words is not a useful skill, except for winning spelling bees, but testing the ability to spell words people actually use would require a much longer test since you cannot eliminate a contestant until he gets one wrong and most contestants will get most of them right.

Rewarding teachers and administrators by how well their pupils do on standardized tests encourages spending class time teaching those and only those things the teacher expects to be on the test, teaching pupils to memorize answers instead of learning history or math. If the pressure to get good scores is high enough teachers and administrators have an incentive to ignore cheating, encourage it, even cheat themselves by altering the answers on student test papers.

If you look at a used textbook for a freshman course you will probably see a lot of highlighted words and phrases. Those are the parts of the course the owner of the book expected to be tested on.

For the SCA case Campbell’s law may be even more relevant.

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.