39 Comments
User's avatar
Eric Darwin's avatar

Climate change claims seem to me to be frequently based on very short periods of accurate data collection, or smallish areas which are then extrapolated, thus not accounting for natural variability over longer time and place. Where I live we tend to get heavy downpours in August, so when we had a heavy rain on Thursday July 31st the headlines were all about the record breaking rainfall and climate change, even though we often got the same rainfall on other days of the week and frequently one calendar day later in August. This is the media that tell me it's "twice as hot" as some other period...they don't realize that 30 degrees C is not twice as hot as 15C. Just try expressing that in degrees Fahrenheit! One year in the 1970s had abnormally very few forest fires, so guess where "record setting" accounts usually start counting. Media and others who should know better seem to confuse number of forest fires (now labeled the more dramatic wild fires) with the area burned individually or collectively. Anything to make a headline.

Expand full comment
CarlW's avatar
Nov 5Edited

Here's one (small sample problem + Bayesian angle): A majority of rural communities have rates of kidney cancer that are lower than urban centers. Many are much lower. Is there something about rural communities that makes them less prone to the disease?

Expand full comment
Doctor Hammer's avatar

I could believe that, or I could believe that kidney cancer is very rare, such that rural areas (with very low population density) are likely to have below whole numbers of people with kidney cancer per capita. That is to say, since you can't have 1/5 of a cancer, if the rate is 1 in 5000 a rural region with a population of 6,000 might have 1 person with kidney cancer and a rate of 1/6000, while an urban region of 60,000 might have 12 people for a rate of 1/5000. If you draw small enough circles, very rare binary things start to get very spotty in their rates.

Of course, maybe kidney cancer requires constant treatment (dialysis maybe?) so people with it tend to move closer to urban centers so they don't need to commute 2 hours to get treatment a few times a week. I honestly have no idea.

Expand full comment
CarlW's avatar

You put your finger on the Bayesian angle. Note that nearly a majority of rural areas can have a higher incidence. Taken as a whole, rural areas and urban areas have similar rates, but for a rare condition, some rural areas have a higher rate and some have lower.

Expand full comment
Kevin S. Van Horn's avatar

Small sample sizes leading to a false conclusion are quite common. They arise when there are reports of "no statistical significance," and this is taken to mean no effect, when what it really means is that the study was too underpowered to detect the effect.

I don't remember the details, but there was one case where 1) first study finds a statistically significant effect, but 2) second study does not, and this is treated as a failure to replicate. In reality, the effect estimate in the second study was even larger than in the first, but the second study had a smaller sample size, leading to a conclusion of "no statistical significance."

Reporting the confidence interval, or doing a Bayesian analysis reporting a credible interval, would help avoid this sort of false conclusion. A very broad CI tells you that a practically significant effect is not ruled out.

Expand full comment
omar's avatar

In evaluating the efficacy of seat belt use, would it be more accurate to note that unrestrained occupants are more than twice as likely to be killed in a crash as restrained occupants? Although this statistic may overlook correlated causes, it is far more probative than deaths per passenger mile, which mix seat-belt effects with broader improvements in vehicle design, road safety, emergency response, and driving habits.

Expand full comment
David Friedman's avatar

There is a classic article by Peltzman that criticizes that approach, I think in the context of earlier legal rules to mandate safer cars. He points out that making cars safer reduces the cost to the driver of behavior that makes accidents more likely, such as driving faster or driving when tired. He found that requirements that reduced deaths per accident also increased accidents, roughly cancelling the effect.

https://www.journals.uchicago.edu/doi/abs/10.1086/260352

Expand full comment
omar's avatar
Nov 7Edited

In the case of seat belt regulation, are you using Peltzman’s article to suggest that mandatory seat belts do not actually reduce fatalities? If the reduction disappears because drivers adjust their behavior to keep perceived risk constant, doesn’t that imply a gain in utility, for example, by allowing them to drive faster or more comfortably while maintaining the same subjective level of danger?

However, that risk compensation adjustment still doesn’t explain why unrestrained occupants face more than twice the fatality rate of restrained ones. If Peltzman’s model holds, we would expect the self-selected unrestrained group to take fewer risks, bringing their mortality rate closer to that of restrained occupants rather than diverging so sharply.

Expand full comment
David Friedman's avatar

The argument does imply that they are getting utility from the behavior that raises the accident rate.

I took the claim on restrained/unrestrained mortality to be about what happens if there is an accident, not on mortality rates for people who do or do not use seatbelts.

Expand full comment
Paul Brinkley's avatar

Another common error I see involves statistics reported in terms of relative size instead of absolute size, or vice versa, depending on which one will have the desired effect.

Just today, I saw a chart of deaths per passenger mile of various modes of transport. The chart is dominated by motorcycles - 212.57 vs. 7.28 for cars, 3.17 for ferries, and less than one for trains, subways, buses, and planes. While that 30x figure for motorcycles looks immense, the numbers are all deaths per 1 billion passenger miles - a motorcyclist will ride an average of 4 million miles before dying. If a cyclist commuted 50 miles to work each way, every weekday, for 50 out of 52 weeks per year, it would take him 160 years to reach that distance.

--

Now that LLMs are commonplace, I again wonder if there's a way to automate checks on statistics, so that a reported figure could automatically be cast in different ways as easily as the reported figure can be read.

Expand full comment
Paul Brinkley's avatar

For other "Small Sample Size" problems, another example centers on deaths from mass shootings. Assuming for the moment that everyone agrees on a standard definition of a mass shooting, one obvious measure to care about is the total number of lives lost to them. According to WorldPopulationReview ( https://worldpopulationreview.com/country-rankings/mass-shootings-by-country ), the #1 country in lives lost is the US, which is probably not surprising to most people. What is surprising is the #3 country: Norway, with 69. All of them happened during one incident, committed by Anders Breivik. It's the deadliest mass shooting in the world (accompanied by a bomb Breivik set off two hours earlier, that killed 8 more), and singlehandedly defines virtually all of Norway's stats with respect to mass murder.

I had once encountered a plot of countries by mean death per incident, showing Norway at #1 with that 69. France was #2 with over 15. The US had a bit under 2, and didn't even rank in the top 10.

Expand full comment
Paul Brinkley's avatar

Either Substack popped a gear, or you have a cut-and-paste error in the OP. Search for "Past posts" in the middle of a sentence that looks like it's supposed to say:

"That assumes that causation runs from law to crime. The obvious alternative is that causation runs the other direction, that a high homicide rate is a reason for a state to have capital punishment."

Expand full comment
David Friedman's avatar

Thanks. Fixed.

Expand full comment
omar's avatar

You have clearly demonstrated how stating facts can promote false conclusions. In your vision for an anarcho-capitalist society without governmental "protections" (in quotes because I recognize you may argue that these protections are a net negative), how will people, who are less skilled at interpreting data, navigate this? If private companies pursue profit rather than accurate contextual truth or the utility of those they serve, how would market incentives attenuate the gap between what is profitable to say and what is contextually true?

Expand full comment
Wasserschweinchen's avatar

Traffic fatalities per capita dropped by 54% in the US in that time period. I would assume that that was what the mayor was referring to.

Expand full comment
Arqiduka's avatar

Your seatbelt story is due to not only cherry picking but small sample: extrapolation form two years only, instead of 40.

Expand full comment
Gian's avatar

A pet peeve is how cross-country comparisons such as per capita GDP vs national IQ or economic freedom are presented without any indication of population size.

The correlations are apparently made without adjusting for population as if Andorra equals India.

Expand full comment
David Friedman's avatar

Is your objection based on the small sample size problem? Andorra has a population of about 87,000, which is a pretty large sample size. Or is your point that some problems are harder (or easier) to deal with in a large population?

Expand full comment
Gian's avatar

When one is trying to correlate per capita GDP with national IQ, is the fact that India is 16000 times Andorra irrelevant? Isn't it just a matter of historical contingency that Andorra is an independent country and India is not 10 or 100 independent countries? Don't these historical contingencies skew whatever correlation one is trying to derive?

Expand full comment
Wasserschweinchen's avatar

But if you're trying to explain why some countries are richer than others, it seems natural to look at country-level data? People have also looked at the differences between Indian states though, e.g. https://emilkirkegaard.dk/en/wp-content/uploads/Differences-in-cognitive-ability-per-capita-income-infant-mortality-fertility-and-latitude-across-the-states-of-India.pdf – and, obviously, lots of people have looked at the differences between individuals.

Expand full comment
Gian's avatar

The country level data ought to be weighted by population size.

Expand full comment
Wasserschweinchen's avatar

Have you tried doing that? I think you'd just end up with a model that explains less of the country-level data.

Expand full comment
Gian's avatar

The country-level data itself is misleading as it clubs vastly different countries together.

If you have hypothesis--economic freedom --> economic growth then if you weigh China and India same as Andorra you probably get a positive answer.

But if you weight China/India by population, the hypothesis is likely destroyed.

Expand full comment
Roman's Attic's avatar

Law of large/small numbers: the schools with the highest average test scores are very small, and the schools with the lowest average test scores are very small. These facts, when presented independently, can (and have) caused people to assume that the school size causes a much larger impact on student education than it does.

Expand full comment
Joy Schwabach's avatar

I think you're saying that sodium is not a major cause of death worldwide because it's balanced by other causes. Of course it is. Countries vary. For example, there's a tribe in Africa that has more of the Alzheimer's gene than anywhere on the planet, but because of their diet, they have extremely low levels of Alzheimer's. Every place is different. Heart disease is huge in some areas of the world mainly because of high levels of saturated fat, but excess sodium is also a culprit. You can definitely conclude that most people eat too much salt. This is from Joel Fuhrman:

For links to references, Google "sodium dr. fuhrman."

High-sodium diets estimated to cause 3 million deaths per year worldwide

The Global Burden of Disease Study, which tracks health metrics in 195 countries, estimated in 2019 that 22 percent of deaths around the world are caused from poor diet. Of the dietary factors they studied, high sodium intake was responsible for the greatest number of deaths: 3 million deaths per year worldwide.2

According to the WHO report, the average sodium intake globally is about 4310 mg/day (10.78 grams of salt), more than twice the WHO’s recommended maximum of 2000 mg sodium/day.1 Much evidence demonstrates that even this 2000 mg/day figure contributes to disease and premature death.3,4

Sources:

GBD Diet Collaborators. Health effects of dietary risks in 195 countries, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017

Blood Pressure Effects of Sodium Reduction: Dose-Response Meta-Analysis of Experimental Studies

Global sodium consumption and death from cardiovascular causes

Why is high sodium intake dangerous?

The most well-known consequence of a long-term high-sodium diet is elevated blood pressure. Many people believe they don’t need to be concerned about sodium intake if their blood pressure isn’t already high. But if you’re eating the standard American diet, and your blood pressure isn’t high, it’s probably only a matter of time. In the U.S., the lifetime risk of hypertension is estimated at 70-85%.5 In the Framingham Heart Study, among men and women with normal blood pressure levels at age 55-65, 90% of those who survived to age 80-85 had developed hypertension.6

Also, there detrimental health effects of high sodium intake that are unrelated to blood pressure.

Sources:

Lifetime Risks for Hypertension by Contemporary Guidelines in African American and White Men and Women

Residual lifetime risk for developing hypertension in middle-aged women and men: The Framingham Heart Study

Heart disease and stroke:

Excess sodium increases heart disease and stroke risk, through both elevation of blood pressure and blood pressure-independent pathways. These include production of reactive oxygen species, arterial stiffening, impairing endothelial function, reducing the release of nitric oxide, and effects on the water and salt-balancing hormone aldosterone.7-14

Sources:

Vascular consequences of dietary salt intake

Salt Reduction to Prevent Hypertension and Cardiovascular Disease: JACC State-of-the-Art Review

Negative effects on the kidneys: High sodium intake increases urinary calcium excretion, which can lead to kidney stones. Also elevated blood pressure increases the risk of chronic kidney disease.10,12,15,16

Sources:

Harmful effects of dietary salt in addition to hypertension

Sodium Intake and Blood Pressure in Patients with Chronic Kidney Disease: A Salty Relationship

Autoimmune diseases:

There is evidence that high sodium intake promotes the production of pro-inflammatory immune cells that contribute to autoimmune diseases.12,17,18

Sources:

Sodium chloride drives autoimmune disease by the induction of pathogenic T17 cells

Induction of pathogenic T17 cells by inducible salt-sensing kinase SGK1

Gastric cancer:

A high-salt diet is associated with a greater risk of gastric cancer, which is thought to be because sodium supports the growth of the ulcer-promoting bacteria H. pylori in the stomach.10,19-21

Source:

Habitual salt intake and risk of gastric cancer: A meta-analysis of prospective studies

Obesity:

Recent evidence shows a link between a high-salt diet and greater risk of obesity.22 High-sodium foods are usually high-calorie foods; in addition, salt stimulates thirst and often intake of sugar-sweetened beverages. High sodium intake may also affect satiety signals.23 But higher sodium intake is associated with greater body fat mass independent of calorie intake, and the reasons aren’t completely clear yet.24

Sources:

Salt Promotes Passive Overconsumption of Dietary Fat in Humans

High salt intake: independent risk factor for obesity?

Most salt is already in the food

The high salt intake in the American diet doesn’t come from adding salt from the shaker at the table. Most of it – over 70% – comes from packaged and restaurant foods.25 The way to consume less sodium is to consume less packaged and restaurant food, and carefully read labels to limit your sodium intake from any packaged foods you do consume.

The U.S. requires sodium to be listed on the Nutrition Facts label of packaged foods, and there is a mandatory sodium limit for foods served in schools. But all other sodium reduction policies are voluntary for food manufacturers.1 This means that, in the U.S., it’s up to the individual to monitor and reduce their own sodium intake.

Source:

U.S. Centers for Disease Control and Prevention: Sodium and Food Sources

Expand full comment
David Friedman's avatar

"I think you're saying that sodium is not a major cause of death worldwide because it's balanced by other causes."

That is not what I am saying.

What I am saying is that if sodium increases deaths from some causes, such as high blood pressure, but increases life expectancy, which is the claimed empirical result, it must decrease deaths from other causes.

Expand full comment
Joy Schwabach's avatar

I don't think you would claim that sodium increases life expectancy if you'd read as many books on nutrition as you have on economics. Isn't that one of the things that bothers economists most, that everyone has an opinion without having studied it? Your opinion is an outlier among nutritional scientists, just as anti-free trade positions are an outlier among professional economists. I just asked Google AI: "Analyze this statement using the consensus of nutritional scientists: "If sodium increases deaths from some causes, such as high blood pressure, but increases life expectancy, which is the claimed empirical result, it must decrease deaths from other causes." Here is its answer: "The statement is a logically sound premise, but it is fundamentally inconsistent with the scientific consensus regarding sodium intake and health outcomes.

Logical Analysis

The statement, 'if sodium increases deaths from some causes, such as high blood pressure, but increases life expectancy, which is the claimed empirical result, it must decrease deaths from other causes,' is logically correct. If an overall positive outcome (increased life expectancy) occurs despite a negative impact in one area (more high blood pressure deaths), there must be a compensatory decrease in deaths from other causes.

Analysis using the Consensus of Nutritional Scientists

The core premise of the statement, that increased sodium intake increases life expectancy or decreases all-cause mortality, contradicts the vast majority of nutritional science consensus and high-quality evidence.

Sodium and Mortality: The scientific consensus, supported by numerous large-scale studies and health organizations (including the CDC, WHO, AMA, AHA, and FDA), is that high sodium intake is associated with increased risks of high blood pressure, cardiovascular diseases (heart attack, stroke), and all-cause mortality.

The "Claimed Empirical Result": The idea that sodium increases life expectancy or has an inverse relationship with all-cause mortality is a finding from a few specific, often observational, studies that have been subject to significant criticism due to potential methodological issues and confounding variables. The consensus view largely dismisses these findings as inconclusive or flawed when weighed against the robust evidence from randomized controlled trials and large meta-analyses showing the opposite.

Conclusion: The scientific community widely agrees that reducing sodium intake is a cost-effective way to improve public health and reduce the burden of cardiovascular disease. Therefore, the statement is based on a 'claimed empirical result' that is not accepted as accurate by the consensus of nutritional scientists."

Expand full comment
David Friedman's avatar

I don't know whether sodium decreases mortality. My point was that there is some evidence it does, and your source argues confidently for the opposite conclusion without mentioning that evidence, hence should not be trusted.

Expand full comment
Joy Schwabach's avatar

I have sources that are hundreds of pages long. It's hard to fit them in. I wish you would start reading nutrition books like "How Not to Age," by Michael Greger, M.D. or "How Not to Die," by the same author. Or "The End of Heart Disease," by Joel Fuhrman, M.D. Or the books of Dean Ornish or countless other MDs who use science plus experience with hundreds of in-person patients and thousands of online ones. Over and out.

Expand full comment
Andrew Currall's avatar

I don't think I actually buy this. The NIH study suggesting the opposite conclusion is simply an observational comparison of countries, noting that countries with lower salt intake tend to have lower life expectancy.

But that obviously heavily confounded! Most poor countries have relatively low salt intake. Most rich ones have relatively high. That proves nothing about salt, which I suspect is generally bad beyond very low levels (well below what most people in developed countries actually consume). There might be an argument for increased salt consumption if you have dangerously low blood pressure, but otherwise, I wouldn't start eating more salt.

Expand full comment
David Friedman's avatar

From the abstract:

"after adjusting for potential confounders such as gross domestic product per capita"

"In a sensitivity analysis restricted to 46 countries in the highest income class, sodium intake continued to correlate positively with healthy life expectancy at birth "

Expand full comment
Andrew Currall's avatar

OK, well I didn't actually read the abstract in full. That's a start.

But I'm still skeptical. I doubt salt has *that* much influence on life expectancy in either direction- national salt consumption will have a tiny (though I expect negative) influence on life expectancy.

And... well you can never control for counfounders enough. Restricting the analysis to the top 50 countries by income is not enough even just to control for a wealth effect alone.

Even fully controlling for GDP/capita (did they do that?) almost certainly won't be enough- there will definitely be other counfounders and they will have substantial effects, larger than the effect you're looking for.

It's actually extraordinarily hard to prove anything at all credibly with by-country analysis like this; your sample size is so small, and there are massive correlations all over the place.

Expand full comment
David Friedman's avatar

I agree that their conclusion might be wrong. My point was that it is evidence against the standard recommendation and that a source that argues with confidence for a conclusion and does not mention evidence against it should not be trusted.

Also, in the post, that evidence that something increases one cause of mortality does not necessarily imply a reduction in overall mortality, and similarly for other things.

Expand full comment
Deepa's avatar

Yes (the point about salt and evolutionary pressure), but also, evolution does not care about us after the age of 25 or so. It doesn't care about preserving genes that keep us healthy after that.

Expand full comment
David Friedman's avatar

Women remain fertile well past 25.

Expand full comment
CarlW's avatar

It cares a little if the hosts of those old genes help their offspring survive and reproduce.

Expand full comment
Deepa's avatar

You could say grandparents help with that too.

Expand full comment
CarlW's avatar

Yes "descendants" would have been better than "offspring," but really anyone carrying copies of their genes, because genes are the what evolution selects.

Expand full comment