A widely-covered study that claimed ExxonMobil misled the public on climate change is fatally flawed, according to a bombshell rebuttal published today by one of the foremost experts in content analysis. The original study, authored by Harvard University’s Geoffrey Supran and Naomi Oreskes, relies on data analysis that is “unreliable, invalid, biased, not generalizable, and not replicable,” according to the expert, whose methods were cited by Supran and Oreskes.

The academic rebuttal is the biggest blow yet to the activists-turned-researchers, and follows similar critiques by Energy In Depth and others.

The Supran/Oreskes study alleged that ExxonMobil had obscured the facts on climate change by producing internal research that conflicted with the company’s advertisements. But as Energy In Depth revealed at the time, Supran and Oreskes relied on an incomplete collection of “advertorials” compiled by Greenpeace; mischaracterized documents that acknowledged climate change as being an example of climate denial; and compared the research of Exxon with the “advertorials” of Mobil before the companies merged.

The latest rebuttal was written by Kimberly A. Neuendorf, Ph.D., a Cleveland State University professor with over 40 years’ experience in the field and who literally wrote the book on content analysis. Neuendorf’s textbook was actually cited by Supran and Oreskes as the source for their information on content analysis. ExxonMobil asked Neuendorf to review the Supran/Oreskes paper based on the authors’ reliance on Neuendorf’s methodology, although the company had no editorial control over Neuendorf’s conclusions.

Tom Johnson, a University of Texas journalism professor, told the Daily Caller that “Neuendorf is one of the preeminent scholars in the field of document analysis.”

“’She literally wrote the book on content analysis and has an invaluable website connected to the page,’ Johnson said without weighing in on the report she conducted for Exxon. ‘She has conducted numerous content analyses. I would put her among the top three trusted sources on this topic.’”

Before delving into what Neuendorf said in her rebuttal, it’s worth recalling a few key facts about Supran and Oreskes.

The New York Times has labeled Naomi Oreskes as one of the ringleaders of the #ExxonKnew campaign, in part due to her work co-organizing and participating in the infamous 2012 La Jolla conference, where anti-fossil fuel activists developed their playbook for attacking the energy industry. Oreskes admitted under questioning from Members of Congress that she met with state attorneys general on several occasions to discuss potential investigations of ExxonMobil. She is also closely tied to Matt Pawa, the plaintiffs’ lawyer leading climate lawsuits against ExxonMobil in New York and California.

Supran, meanwhile, is heavily involved in the fossil fuel divestment movement and tried (unsuccessfully) to pressure the American Geophysical Union to disassociate itself with ExxonMobil. In May 2017 Supran suggested in an op-ed on Mashable that the homeless should be doing more to lower their carbon footprints.

Both Supran and Oreskes had publicly established their bias against ExxonMobil well before they published their study. As Energy in Depth wrote last summer:

“In one tweet from 2015, Oreskes wrote: ‘Did Exxon deliberately mislead the public on climate change? Hello. Of course they did!’

“The fact that Oreskes was declaring ExxonMobil was ‘misleading the public’ two years before her study was finished suggests she worked backwards from a conclusion.

“Oreskes has also publicly cheered on the Massachusetts Attorney General for her costly legal investigation into ExxonMobil’s statements on climate change.

“Supran, meanwhile, has tweeted that it’s a ‘sane’ position to suggest that ‘Exxon’s actions may have imperiled all of humanity…It’s time to divest.'”

The Supran/Oreskes study was funded by the Rockefeller Family Fund, the anti-fossil fuel group that has funded every step of the #ExxonKnew campaign, including the series of articles written by InsideClimate News and the Columbia School of Journalism that underpin this study. The study also relied on materials prepared by Rockefeller-funded groups, including Greenpeace and the Union of Concerned Scientists.

In her analysis of the Supran/Oreskes study, Neuendorf pulls no punches, writing:

“After a detailed review of the study, its supplementary information (“SI”), and the documents S&O [Supran and Oreskes] analyzed for their study, I have concluded that S&O’s content analysis does not support the study’s conclusions because of a variety of fundamental errors in their analysis.” (emphasis added)

Neuendorf identifies seven “fundamental flaws” that undermine the study, including non-representative sampling, inappropriate coding, and a heavy reliance on dubious consensus measurement. Adding to criticisms previously raised by Energy In Depth regarding Supran’s and Oreskes’ reliance on documents compiled by Greenpeace, Neuendorf writes:

“In general, there is no explanation of how documents were collected objectively via comparable techniques across document types, and, upon inspection, it is clear that numerous documents were hand-picked and not part of a comprehensive sampling strategy. Indeed, S&O admit that there are ‘countless additional climate change communications from ExxonMobil that could be included in future work’ (p. 2), thus undercutting the integrity of their own sampling.” (emphasis added)

She also had sharp critiques of Supran’s and Oreskes’ reliance on themselves and a fellow anti-fossil fuel activist Ploy Achakulwisut as the sole coders for their study:

“Content analysis coding ought to be conducted with coders who are at arm’s-length with regard to the research, in order to maximize objectivity. Optimally, coders should be blind to the research questions or goals. In the S&O study, the coders were not blind. In fact, they were as non-blind as could be imagined. They were the investigators themselves, as well as an affiliated graduate student. In this particular case, the problematic nature of informed coders is magnified by the coders’ long-time and intensive involvement in the popular communication of climate change. Further, two of the coders have publicly demonstrated particular biases that existed before the execution of the S&O study (Oreskes, 2004; Oreskes, 2015a, 2015b; Oreskes & Conway, 2010; Supran, 2016). (emphasis added)

Neuendorf also took issue with how the documents included in the Supran/Oreskes study were coded:

S&O employ a coding scheme that includes bias, is complex and requires specialized expertise beyond the scheme, and instructs coders to engage in activities that are unacceptable for content analysis, including skimming, looking beyond the documents for context, and resolving coding ambiguities through discussion.” (emphasis added)

The Supran/Oreskes study also violates one of the most basic tenets of the scientific method, writes Neuendorf:

“Content analysis is optimally conducted within the framework of the scientific method. It is therefore important that the research be guided by specific hypotheses derived from broadly generalizable theory, or more general research questions optimally derived from theory or past scholarship. In this way, the relative objectivity of the research endeavor is supported. The S&O journal article does not present hypotheses or research questions.

“The S&O SI does indicate that ‘research questions’ were created (but not actually presented in the article or the SI) ‘in order to determine whether the corporation misled consumers and/or shareholders by making public statements that cast doubt on climate science and its implications, and which were at odds with available scientific information and with what the company knew’ (p. 1). This phrasing is more consistent with presuppositions than queries, revealing assumptions made by the researchers, and indicating a bias that precludes the study’s objectivity.

“Even more definitive, a public statement of the study’s major conclusion was made by one of the authors prior to the execution of the study. As noted above, Oreskes issued the following statement on social media on October 21, 2015: ‘Did Exxon deliberately mislead the public on climate change? Hello. Of course they did!’ (Oreskes, 2015b). This pre-study conclusion violates basic tenets of scientific research.” (emphasis added)

And finally, Neuendorf critiques Supran’s and Oreskes’ reliance on consensus measurement, “a method that does not appear to qualify as an accepted, scientific method:”

“Consensus measurement, in comparison to content analysis, is not a standard, time-honored research technique…In a sense, it seems to be a conclusion in search of a method, as S&O note it has been used to ‘quantify the consensus on AGW’ (p. 2)…

“As referred to in the S&O piece, consensus measurement seems to be located within the purview of a specific group of researchers. The investigators using consensus measurement seem to be a relatively small group, with inter-citation and self-citation notable (e.g., Anderegg & Goldsmith, 2014; Cook, 2016; Cook & Jacobs, 2014; Cook et al., 2013; Cook et al., 2016; Maibach & van der Linden, 2016; Oreskes, 2004). This type of interdependence has the potential to create an ‘echo chamber’ of reinforcing ideas, without critique and correction (see, e.g., Jankó, Vancsó, & Móricz, 2017).” (emphasis added)

Neuendorf’s findings raise serious questions about the quality of research that Supran and Oreskes published. As Neuendorf concludes in her paper, Supran and Oreskes “provide no scientific support for either a discrepancy among ExxonMobil’s climate change communications, or a claim that ExxonMobil misled the public.” Indeed, it is clearer now more than ever that it was Supran and Oreskes that misled the public on ExxonMobil.