It’s only been a week since Naomi Oreskes and Geoffrey Supran entered back into the chat with a new “Exxon Knew” study, and, like their past work as part of this campaign, the cracks are already beginning to show. A deeper look at the latest study has led to questions regarding the methodology used by the two researchers who have a long track record of engaging in paid and coordinated activism against American energy companies to support a litigation campaign they helped create.
Back At It Again
As Energy In Depth pointed out last week immediately after the study was published, the authors of the study are far from objective researchers. The primary author is University of Miami professor Geoffrey Supran, a self-described “scientist and an activist” who in 2016 endorsed “engineer[ing] Exxon out of business” and more recently advised the U.S. House Oversight and Reform Committee during its nothingburger investigation of the energy industry.
Harvard professor Naomi Oreskes, who “conceived” the infamous 2012 La Jolla conference where the entire climate litigation campaign began, also contributed to the study. Oreskes was also a paid consultant for the plaintiffs’ law firm Sher Edling – a fact that was finally acknowledged in this latest study and had gone undisclosed in her previous research. Not to mention that she had the outcome of her studies determined in 2015 – prior to their publishing.
Notably, Supran and Oreskes have faced fierce criticism for their 2017 analysis of internal industry documents, which was found to be “unreliable, invalid, biased, not generalizable and not replicable” by Professor Kimberly Neuendorf, Ph.D., an expert in content analysis whose method was cited by Supran and Oreskes in a previous report. Neuendorf also noted that the pair’s long record of a very public bias against ExxonMobil and fossil fuel companies “violates basic tenets of scientific research.”
The same bias is evident in this latest study, which is also co-authored by Stefan Rahmstorf, a German climatologist who leads the research department at the Potsdam Institute for Climate Impact Research. The authors ascribe intent to the company’s climate modeling and communications, a conclusion that goes beyond the “scientific” findings of the article.
Now, after a first review of the study, Energy In Depth has identified several glaring flaws with the research that massively undercut the conclusions pushed by the authors.
Convenient Methodology
Following an initial review of the study last week, University of Colorado Boulder professor Roger Pielke Jr. tweeted out a few thoughts, concluding “there is sufficient ‘play’ and complexity in the methods that anyone getting expertise is [sic] this work probably stands to make some bank in the court cases for which this research is designed.”
The study’s findings are based upon a method of data collection that introduces added uncertainty into their research.
First, the study’s ultimate conclusion – that ExxonMobil “knew” about climate change and should therefore be held liable in court – is derived from a sample size of 12 climate models attributed to the company. To arrive at these twelve models, the authors reviewed “32 internal documents produced in-house by ExxonMobil scientists and managers between 1977 and 2002, and 72 peer-reviewed scientific publications authored or coauthored by ExxonMobil scientists between 1982 and 2014.”
Of these 12 models, several were published in peer-reviewed articles and involved many collaborating authors representing different organizations. For example, climate model #10, which Oreskes, et al. describe as a global warming projection “reported by ExxonMobil scientists,” is pulled from the Working Group 1 contribution to the Third IPCC Assessment. This report was compiled by 122 lead authors and 515 contributing ones. It was reviewed by a separate group of 420 other experts, and widely distributed to the scientific community and to the public.
The fact that Oreskes, et al. rely on research that wasn’t authored exclusively – or, in some cases, even substantially – by employees at the company undermines the entire theory of the paper and the “Exxon Knew” campaign, which absurdly claims that only the company knew about climate change decades ago.
In another instance, for two out of the twelve models, the authors treat low, nominal, and high models as separate observations. The decision to treat separate estimates from the same paper as three distinct models assigns a higher level of certainty than the original authors’ predictions. Four out of the twelve models cited – Black/Mastracchio, Shaw/Glaser (fig. 9), Kheshgi & Jain (7c), Kheshgi & Jain (8c) – also have large confidence intervals relative to the other eight models included in their study. In other words, the models’ projections are less certain, and therefore more likely to overlap with actual, measured climate effects.
Even more concerning is how the authors obtained the data behind the 12 models attributed to the company’s scientists. For most of the models evaluated, the authors inferred the projected CO2 estimates from the graphs in the cited papers rather than replicating the graphs using the actual raw data. This is particularly true for the earlier published papers and internal documents, many of which are small, blurry scans of graphs contained in decades-old academic work.
This method runs the risk of replication error – particularly when discussing statistical significance of models, the inferred projections being off by very small margins could change the final conclusions of the regression, which was calculated against the projected data and real-life outcomes.
Fundamentally, the study is the epitome of “hindsight is 20/20” – it seeks to hold scientists accountable for forward-looking projections. Back in the 1970s and 80s, the company might have conducted research which turned out to track closely with real-life observations decades later, but the scientists would have no way of knowing that at the time.
However, science is an iterative and evolutionary process. The core principle of why science is the gold standard of inquiry is that researchers are allowed to be wrong, uncertain, imperfect, and correct, as long as future researchers build on and improve previous work.
While it isn’t disingenuous for the study’s authors to review and evaluate prior work, it is disingenuous to hold previous work to the standards of today’s knowledge while misrepresenting the level of confidence the original authors had in their models.
What The Source Documents Really Said
The authors also conveniently leave out essential context the original studies’ authors used to frame the climate models cited. Oreskes and Supran claim that, for years, the company’s public affairs strategy revolved around emphasizing uncertainty in climate projections, and that these communications were in direct contradiction to the company’s internal models. But no fair reading of the actual company reports would indicate anything like that at all. In the end, they don’t really show that “Exxon Knew.” What they show is that “Exxon Wasn’t Really Sure.”
Nearly every document containing a model evaluated by the authors includes language that qualifies, hedges, and contextualizes the accuracy of climate modeling at that point in time.
The earliest climate models evaluated by the authors – pulled from an internal memo on the “greenhouse effect” presented to the company’s management team in 1978 – claims that the climate modeling techniques used at the time were very limited in their predictive ability:
“Predictions on the significances of increases in atmospheric CO2 must be based on climate modeling. Modeling climatic effects is currently handicapped by an inability to handle all the complicated interactions which are important to predicting the climate.”
Similarly, a 1984 internal company memo cited by the authors presents forward-looking climate models but acknowledges their limitations:
“Validity of models not established. Complexity of carbon cycle and climate system require many approximations and parameterizations. Geological and historical data are inadequate for validation of models.”
As it turns out, company scientists never intended for their work to be used in the way Oreskes et al. evaluated it in their review. A public affairs briefing on climate modeling cited by the authors makes this clear:
Conclusion
This study has been marketed as “Exxon Knew 2.0”, but, as a reminder – the several iterations of the original “Exxon Knew” theory died on the vine during the New York Attorney General’s failed case against the company.
In 2015, former New York Attorney General Eric Schneiderman launched a years-long investigation into ExxonMobil in November 2015, pressured by activists behind the original “Exxon Knew” campaign. While Schneiderman alleged that the company’s public statements on climate change were at odds with its internal research on the topic, millions of subpoenaed internal documents never backed up that claim, and AG Schneiderman was forced to change his justification for the investigation several times.
When the Attorney General ultimately filed suit against the company in 2019, the state’s argument had been scaled down to a niche question of accounting in a case that was ultimately defeated both at trial and on appeal to the New York State Supreme Court. Ironically, some of the internal company documents evaluated by the study’s authors are the same ones that failed to justify the “Exxon Knew” hypothesis after years of legal review.
What’s more, projections are forward-looking estimates; they are not known facts. Put in layman’s terms: you may accurately project the final score of the Super Bowl based on everything you know about the teams and how the season has gone before the game starts (say, the Dallas Cowboys beat the Kansas City Chiefs 28-21), but you wouldn’t know that you were right until after the game is over.
Still, Oreskes, et al. make the flawed attempt to use modern knowledge to hold historical projections accountable. The study – tailor-made for a litigation campaign – downplays the uncertainty of decades-old projections and disregards the fact that the research was a complement to government and academic scientists’ existing work, throwing cold water on the claim that the company “knew” about climate change when no one else did.
