Checking the Fact Checkers

Some years back I came across a link on Facebook to a claim that "Over Half of all Statements Made on Fox News are False," based on a story in the Tampa Bay Times' PunditFact. My first reaction was that I had no more reason to trust the Tampa Bay Times than to trust Fox, making the story  pure partisan assertion, so I followed the link to see what support it offered. To their credit, they listed the statements on Fox that they based their claim on and provided the basis for their conclusions. But looking at them in detail, their evaluation was clearly biased in favor of what they wanted to believe. 

The clearest case was one where they said a claim they liked was mostly true when, on their evidence, it was  clearly false. The claim was:

The people who want President Barack Obama impeached "are all white, they're all older, and guess what, they're in the far right wing of the Republican Party.

According to the poll information they cited, about 33 percent of the public wanted Obama impeached. There is no way that older whites in the far right wing of the Republican party could make up anything close to 33 percent of the public. According to the poll information, about 17% of non-whites wanted Obama impeached—a lower percentage than of whites but quite a lot more than zero. The statement should have been rated as false.

Another claim was that

Over the last 40 years, real wage growth has been flatlined because of the policies of the Federal Reserve ... It was all driven by the Federal Reserve.

PunditFact found the flatlined claim "largely correct" but rated the overall statement "mostly false" on the grounds that economists they consulted did not think the Fed had any responsibility for the long term trend in wages. Given the state of macroeconomics — I like to describe a course in macro as a tour of either a cemetery or a construction site — I do not think it's legitimate to rate such a statement as either true or false. The claim that the Fed was largely responsible for the Great Depression, after all, was accepted by, among others, a previous chairman of the Fed and the Fed has repeatedly made it clear that it believes that its policies affect the overall health of the economy.

Another claim they rated as false, by Lou Dobbs, was reported as:

Because of President Barack Obama’s failure to "push job creation," the black unemployment rate in Ferguson, Mo., is three times higher than the white unemployment rate.

I rate PunditFact's statement of what Dobbs said as itself false, since if you follow the link you discover that what he actually said was

These are the results of policies on the part of the state government, the local community, and the president of the United States,

which did not imply that all, or even most, of the problem was due to Obama's policies.

PunditFact's explanation of why what they claim Dobbs said is false includes:
Experts have consistently told us and our colleagues at PolitiFact that although government policies can affect employment figures, other economic factors are typically at play; in this case, the Great Recession and subsequent recovery.

Considering that Obama himself claimed his policies would bring down unemployment and so terminate the Great Recession it makes no sense to take it as an entirely independent cause.

PunditFact points out, correctly, that the higher black unemployment rate long predates Obama. But that does not tell us whether different policies by Obama would have changed it. Obama and his supporters clearly  believed that there were policies he could and should follow that would solve long standing problems.

In summary ...

To their credit, PunditFact took advantage of the opportunity provided by hypertext to provide detailed explanations of their ratings. Looking at those explanations, I conclude that:

1. PunditFact interpreted disagreement with them and the experts they choose to consult as either false or mostly false.

2. PunditFact judged statements they liked not by whether they were true but by whether a much weaker claim along similar lines was. Thus they implicitly convert

they're all white, they're all older, and guess what, they're in the far right wing of the Republican Party,

a claim that is clearly false, into

people who are white, older, and right wing Republicans are more likely to want Obama impeached than people who are not,

and then judge it mostly true.

They take the opposite approach to statements they don't like. Converting what Dobbs actually said to "Because of President Barack Obama’s failure to ‘push job creation’" I would rate as "pants on fire"—a flat lie by PunditFact.

Those examples of biased fact checking were ones I found nine years ago and reported on my blog, so I have been looking around the web for more recent examples. Here are some:

Politifact

In September of 2020, commenting on a guest on Tucker Carlson’s show who argued that Covid originated in a lab leak, Politifact wrote:

Scientists say the coronavirus emerged from bats and later jumped to humans.

• The genetic structure of the novel coronavirus rules out laboratory manipulation. Public health authorities have repeatedly said the coronavirus was not derived from a lab.

In a Sept. 15 interview, the most-watched program on cable network television aired a conspiracy theory that has been debunked since the beginning of the coronavirus pandemic.

"Give us, for a non-scientific audience, a summary of why you believe this virus came from a lab in Wuhan," said Fox News host Tucker Carlson during his self-titled primetime show.

…

Scientists worldwide have publicly shared the genetic makeup of the coronavirus thousands of times. If the virus had been altered, there would be evidence in its genome data.

But there isn’t. In March, several microbiology, infectious disease and evolutionary biology experts wrote in Nature — a respected scientific journal — that the genetic makeup of the coronavirus does not indicate it was altered. 

On May 17^th, 2021, however, Politifact retracted that claim, writing:

When this fact-check was first published in September 2020, PolitiFact’s sources included researchers who asserted the SARS-CoV-2 virus could not have been manipulated. That assertion is now more widely disputed. For that reason, we are removing this fact-check from our database pending a more thorough review. Currently, we consider the claim to be unsupported by evidence and in dispute.

It is true that the lab leak theory was and is in dispute but it is supported by one very strong piece of evidence known from the very beginning. So far as we know, there were at the time only two labs in the world doing the sort of research on bat viruses that could have produced Covid, one in the U.S. and one in Wuhan. There were thousands of live markets where the virus could have originated as an animal to animal transmission. The fact that Covid first appeared in Wuhan is by itself strong evidence for the lab leak theory.

It is to the credit of PolitiFact that they eventually retracted their claim, replaced it with a detailed account of the arguments on both sides, and left up an archived copy of the original claim as evidence of their past error. It is to their discredit that it apparently did not occur to them, either when writing the first article or the second, that neither virologists nor people responsible for funding the sort of research done at the WIV are unbiased sources on whether that sort of research was responsible for the pandemic. It is now known that the Nature article they cite was written by researchers pressured by, among others, Fauci, to overstate the evidence against a lab leak origin.

In February of 2020 The Lancet published a “Statement in support of the scientists, public health professionals, and medical professionals of China combatting COVID-19” which included:

We stand together to strongly condemn conspiracy theories suggesting that COVID-19 does not have a natural origin.

The statement ended:

We declare no competing interests.

One of the signers was Peter Daszak, the President of EcoHealth Alliance, the organization that had funded research on bat viruses at the Wuhan Institute of Virology with money from a government grant.

All of that was public information six months before PolitiFact published the first article. That article reported what someone who got his information from the New York Times and similar sources would believe but not what a competent reporter evaluating the evidence for himself would have concluded.

Looking for more recent fact checks from PolitiFact, I found:

Vice President Kamala Harris drew wide attention when she called a section about slavery in Florida’s new middle school social studies standards an "insult."

"They push forward revisionist history," Harris said July 20 at a national convention of Delta Sigma Theta Sorority Inc. "Just yesterday, in the state of Florida, they decided middle school students will be taught that enslaved people benefited from slavery.”

The relevant passage in the standards was:

"Instruction includes how slaves developed skills which, in some instances, could be applied for their personal benefit."

If “benefited from slavery” means “got some benefits from slavery” it is obviously true — slave owners fed their slaves and being fed is a benefit. If it means, as Harris obviously intended it to mean, “were better off as a result of slavery than they would have been if they were not slaves,” then it did not follow from the quoted passage.

PolitiFact rated Kamala Harris’ statement “Mostly True.”

After the first Republican candidate debates, Politifact factchecked the candidates. Commenting on Ramaswamy’s climate skepticism, it wrote:

More than 97% of the world’s climate scientists and a majority of domestic and international scientific organizations agree that human activity is causing the Earth to warm,

It is almost certainly true that the Earth is warming and human activities are at least in part responsible, but I discussed why the 97% factoid based on Cook et. al. 2013 is a bogus figure in an earlier post.

Responding to Ramaswamy’s claim that:

And so the reality is more people are dying of bad climate change policies than they are of actual climate change."

Politifact wrote:

Ramaswamy didn’t explain his evidence, and it was unclear what he was referencing — his campaign didn’t immediately respond to us. But data from a number of credible sources shows the human toll from climate change is significant.

Worldwide, extreme weather disasters made worse by climate change caused more than 2 million deaths between 1970 and 2021, the World Meteorological Organization said in a May 2023 report. The World Health Organization reported in November that at least 15,000 people died due to the heat in 2022. And the U.S. Centers for Disease Control and Prevention says that 600 people a year die from heat-related illnesses, although experts told PolitiFact that number is likely an undercount.

Summer 2023 has set record temperatures in the U.S. and across the globe, with July the hottest month on record. Extreme temperatures that month were made at least three times likelier because of human-caused climate change, one research group found.

Extreme weather disasters may be made worse by climate change — or, in the case of extreme cold, made less bad — but the piece linked to says nothing at all about how many of those deaths would have happened without climate change. Similarly for the figure on deaths from heat.

Further, while global warming makes extreme heat more common it makes extreme cold less common, as can easily be seen in the relevant tables in the IPCC report. Total deaths from cold, globally and in the US, are much larger than deaths from heat, so if the relevant figure were total deaths, as Politifact implies,

¹

the effect would be to reduce them.

"I don’t know of any climate policies that are killing people," said Andrew Dessler, professor of atmospheric sciences and director of the Texas Center for Climate Studies at Texas A&M University. "Fossil fuels, however, kill millions of people every year from air pollution."  

Maize, which Americans call corn, is a major food crop, especially in poor countries, producing nearly 20% of the total calories consumed by humans, at least 30% in Sub-Saharan Africa. The US biofuels program converts more than ten percent of the world’s supply of maize to alcohol, pushing up its price. I do not know how many people die as a result but it is surely a substantial number. If I were willing to copy Politifact’s logic I would imply that all deaths from hunger, nine million a year by one estimate, are due to the biofuel program and hence to bad climate change policies.

Snopes

Snopes is a well known fact checking site. Looking over its posts I found many I agreed with, mostly debunking easy targets, but when they got into issues of serious controversy the quality went down.

Under the headline “Does FBI Have Evidence That Joe Biden Received $5M Bribe from Burisma Exec?” Alex Kasprak, on June 22, 2023, wrote:

GOP investigators and Rudy Giuliani have both claimed to know the identity of a whistleblower with information on an alleged bribe to the Bidens.

He goes into considerable detail on Giuliani’s version of the story, arguing, I think correctly, that its details are inconsistent with known facts. The question at issue, however, is not whether Giuliani is a reliable source of information but whether the Bidens accepted a bribe.

Kasprak:

“Claims that the FBI may have a document that asserts an informant has audio recordings showing then-Vice President Joe Biden and his son Hunter Biden discussing a "bribe" have brought a deluge of largely rehashed conspiratorial claims.”

Ten days before Kasprak published his article the FBI provided a retracted text of the document to Congress, after which “may” was no longer appropriate. Whether what the document reported was true was, and is, an open question, but the existence of the document was not.

Kasprak:

“These claims are problematic for several reasons. First, Zlochevsky has explicitly denied that such a conversation ever occurred. In 2020, Politico asked Zlochevsky to detail any contacts he had with Joe Biden from 2013 to 2019. He replied that, "No one from Burisma ever had any contacts with VP Biden or people working for him during Hunter Biden's engagement." “

From a news story on Hunter Biden’s laptop:

Joe Biden met with Ukrainian, Russian and Kazakhstani business associates of his son’s at a dinner in Washington, DC, while he was vice president, records on Hunter Biden’s abandoned laptop show.

The dinner, on April 16, 2015, was held in the private “Garden Room” at Café Milano, a Georgetown institution whose catchphrase is: “Where the world’s most powerful people go.”

The next day, Hunter received an email from Vadym Pozharskyi, an executive of the Ukrainian energy company Burisma, to thank him for introducing him to his father.

“Dear Hunter, thank you for inviting me to DC and giving an opportunity to meet your father and spent [sic] some time together,” Pozharskyi wrote on April 17, 2015.

“It’s realty [sic] an honor and pleasure.”

Kasprak was quoting as evidence a statement he knew was false — unless he believed that the laptop was a fraud, a position that, by the time he wrote, had become difficult to defend.

His conclusion:

There is no evidence to suggest that Joe Biden, or the Biden family, received money in exchange for political actions.

There is no proof that Joe Biden delivered political actions in exchange for money. There is massive evidence that the Biden family, principally Hunter, received large amounts of money from foreign actors at times when Joe Biden, as VP, was in a position to do things for those actors, a fact not mentioned in the Snopes article. It is possible that Hunter managed to fool Ukrainians, Chinese, and Russians into paying him millions of dollars for imaginary favors from his father but it is not the most likely explanation.

Another Snopes piece, this time by Izz Scott LaMagdeleine, is headlined “Has There Ever Been a School Shooting Where Staff Could Carry Guns?” It references a quotes a tweet by U.S. Rep. Thomas Massie: "There's never been a school shooting in the hundreds of schools that allow staff to carry." 

When we reached out to Massie's office for evidence to support his claim about school shootings and armed staff, a spokesperson cited an April 2019 study by economist John Lott. Lott has been described by publications like The New Yorker and The Trace as the most influential pro-gun researcher in the country. Many academics have refuted his findings.

The link to “refuted” references Lott and Mustard’s 1997 journal article “More Guns, Less Crime,” which found that laws permitting concealed carry reduced confrontational crime, and reports that later work by John Donohue incorporating an additional decade of data and using different statistical techniques got the opposite result.  It adds:

These new findings are strong. But there's rarely such a thing as a slam-dunk in social science research. Donohue notes that "different statistical models can yield different estimated effects, and our ability to ascertain the best model is imperfect." Teasing out cause from effect in social science research is often a fraught proposition.

“A later researcher using additional data and different statistical methods got a different result” is well short of “Many academics have refuted his findings.”

The Snopes piece offers no evidence against Lott’s claim, which looked at schools where teachers were armed. It  points out that Massie’s “staff” could include school police and that there is at least one case, the Uvalde shooting, of students being killed in a school with armed school police. But, in the debate between Massie and Representative Bowman that the article cites just before quoting Massie’s tweet, Massie told Bowman, "You know, there's never been a school shooting in a school that allows teachers to carry," which avoids the ambiguity of “staff” in the later tweet — a fact LaMagdeleine does not mention.

Nobody, so far as I know, is arguing about whether school police should be armed; the controversy is over whether teachers should be. The article quotes various people who think they shouldn’t be but the only evidence mentioned, Lott’s article, is on the other side. LaMagdeleine tries to obscure that fact with the unbacked claim that Lott has been refuted by many academics and the irrelevant issue of armed police.

Conclusion

After looking through quite a lot of fact checking articles from a variety of sites I conclude that most of them are true, non-controversial, and uninteresting, variants on “this picture claimed to show person A in context B is actually about unrelated person X in context Y.” Social media, where anyone can post, provide a target-rich environment.

The rare fact-check that deals with issues associated with political polarization tends to be biased, in the cases described here towards the left, not surprising given the pattern of political views among journalists. There may well be right wing fact-checking sites with the opposite pattern but my searches have not found them. Perhaps readers can offer some.

In the interest of honesty I should report that John Lott was a student of mine at UCLA many years ago and that the Lott and Mustard article on concealed carry provided empirical support for a theoretical point in my Price Theory. I followed the controversy over the article for a while, abandoned it when it reached a level that required more effort and more statistical expertise than I was prepared to offer.

P.S. After posting this I came across a site that classfies fact checking sites by bias. Now I have to look at the ones it labels right.

1. The relevant figure would be not total but marginal, not total deaths from heat or cold but by how much deaths from heat increased and from cold decreased due to global warming.