Optimizing On A Single Variable

why not to

The current standards for dishwashers took effect in 2013. The standards, which were based on a consensus agreement between manufacturers and efficiency advocates, specify minimum energy and water efficiency levels. The standards require that standard-size dishwashers use no more than 307 kWh per year and 5.0 gallons of water per cycle.

In 2024, DOE finalized amended standards for dishwashers based on a joint recommendation from manufacturers and efficiency advocates. The new standards for dishwashers will cost-effectively reduce energy consumption by 15% relative to the current standards while also cutting water waste. Dishwashers

It is a general problem, but what started me thinking about it was being told by my dishwasher that it would take three and a half hours to wash the dishes. That seems, judging by a quick search online, to be longer than average but still within the normal range. I have not been able to find figures online for how long dishwashers took twenty or thirty years ago but, by what I remember, it was substantially less — and the dishes ended up dry, which ours don’t.

The explanation is in the final word of the quote above, “waste.” The owners of dishwashers pay for water and power, so if making them more efficient in those dimensions was costless, did not require giving up something else, there would be no need for the Department of Energy to make the manufacturers do it. I conclude that it was not costless, that it either made dishwashers cost more or do their job less well — take longer, not dry the dishes as well, not clean them as well. Using more power or water to do a better job is not waste.

Covid provides a much larger example of the same mistake. Regulations were tailored to minimize infections from Covid, whatever the cost in other values. That cost, as we now know and could have predicted, included children missing a year or more of school, people confined to their homes for extended periods, old people dying alone, lost production and jobs, people dying of other causes while waiting for medical services committed to Covid.

An earlier example was the push to minimize consumption of saturated fats, in large part by persuading people to switch from butter to margarine — without worrying about the effect of the transfats in the margarine.

After the Fukushima accident the German government concluded that the optimal number of reactors was zero and achieved it by shutting down all of its reactors. One effect was to raise the cost of electric power, with negative effects on both consumers and industry. Another was to substitute power from fossil fuels for power from reactors, increasing the output of CO₂, which the German government was also trying to minimize. Another was to make Germany dependent on Russian natural gas and increase the income of the Russian government, some of which could be devoted to invading Ukraine.

Goodhart’s Law

When a measure becomes a target, it ceases to be a good measure.

My examples so far involve optimizing on one variable while ignoring the effect of doing so on other variables that you also care about. For a different version of the problem, consider the academic world. Universities want to hire and promote productive scholars. One thing a productive scholar does is to write articles and get them published in journals. The number of published articles is an easy measure of how productive a scholar is. Hence “publish or perish.”

It is not a very good measure. Productive scholars publish but “publish or perish” gives unproductive scholars an incentive to publish too, even if they have nothing worth publishing. One way to do so is literal fraud, reporting interesting results you didn’t get, but there are others. It is easier to get published if your theory is new, false theories are more likely than true ones to be new, and there are ways short of fraud to produce evidence for a false theory. Run enough versions of an experiment or run one version and analyze it in enough different ways and you are likely by sheer chance to get something that looks significant, hence worth publication. Journal editors and peer reviewers are supposed to filter out bad articles but there are multiple ways of getting around them. You can select your reviewers by whose articles you cite, since that is one way editors find reviewers, so cite your friends or people who will like the conclusion of your article. Submit to a journal that is not too picky in what it accepts; there are a lot of journals out there and they are funded, at least in part, by submission fees.¹ An elite university can limit submission credit to elite journals, may have faculty members willing to read and evaluate the candidate’s work, but most universities are not elite.

Judging an academic by the number of his publications corrupts academia in two different ways: the effect on the quality of academics and the effect on the quality of publications. The latter was strikingly illustrated by the replication crisis, when it turned out that a lot of striking experimental results did not happen if someone else did the experiment.

George Stigler, in the first chapter of The Intellectual and the Marketplace, tells the story of an academic reformer, a young academic who found himself rector of a South American university due to his father having bankrolled a successful revolution. His first plan to improve the faculty was to let a graduate student or faculty member challenge his immediate superior for his place and salary, the challenge being judged by an exam graded by an American professor. The results at first looked good, but:

The library experienced an unprecedented rush. Learned journals — especially American — came out of dusty stacks, and hot disputes raged over the attempts of some men to draw out all of the modern treatises in the field. Few were willing to discuss their field except with better informed people, and the exceptions were attributed to deceit as often as the arrogance. … the graduate students suffered most. Filippo devoted his year course in the advanced theory of functions to a review of Euclid. Donto succeeded in getting many economics students to read Alfred instead of Adam Smith; and Ricard reviewed the Baconian theory in painstaking detail in his course on Shakespeare.

… so he amended the regulations to grant five points in 100 to a teacher for each of his students who won a challenge. This new rule led to careful calculation; would 5 points outweigh a superior performance by the student in the examinations? The general belief was that professors gained, and assistant professors lost, by careful instruction. …

By the next autumn, another unanticipated result of the reform became apparent. There was a precipitous fall in the enrollment of graduate students, and it was soon discovered that all could afford it had gone to the United States for graduate work ….

… Storeo, the brilliant young astronomer, was defeated by a mediocre graduate student who — with the examiner — had spent the year studying some obscure variable stars. And Birni fell because his magisterial command of political theory did not extend to the details of the New England Township.

…

And as the scheme entered its third year, a further effect could no longer be overlooked. Research had almost stopped. Once observed of course it was easy to explain. A man was likely to pass the examination with high marks if he knew what others were doing. It did not help his chances materially to do something himself. The faculty was becoming extremely well read, and extremely unenterprising. …

… Once persuaded he issued still another amendment: a man was to receive two points for each article and seven for each book. …

… Cimoor, whose father owned a publishing house, succeeded in getting out two books within the first year, and so influential was his father that many of the reviews were neutral. The political scientist Broze withdrew a book already in page proof, and published the 19 chapters as 19 articles.

I considered retelling Stigler’s story in my own words but he was a better writer than I am. If you can find a copy of the book, read it.

Goodhart’s law famously existed in the planned economy of the Soviet Union, where how a factory’s output was measured for deciding whether its administrators should be rewarded or punished determined what it was in their interest to maximize. If a nail factory was rewarded by number of nails it was in their interest to produce lots of small ones, if by weight …

(From the humor magazine Krokodil)

---

Past posts, sorted by topic

My web page, with the full text of multiple books and articles and much else

A search bar for past posts and much of my other writing

1. Promotion/Tenure Decision in the Fall? Submit your Paper Now to IAER! (seen online)