Despite efforts by the industry to reduce emissions, a new report claims methane emissions from oil and natural gas production in New Mexico are five times higher than the U.S. Environmental Protection Agency estimates.

In fact, the analysis from the Environmental Defense Fund (EDF) claims these methane emissions are about “twice as high” as the group’s previous estimates in the state. Jumping from estimates of 570,000 metric tons per year to about 1 million metric tons annually is alarming, but should also – at the very least – warrant serious scrutiny, especially since EPA data show the state’s methane emissions to be only 19 percent of what EDF is claiming.

A closer examination of EDF’s analysis shows that questionable choices in methodology, use of disputed studies and reliance on vaguely defined “abnormal emissions” all raise considerable doubts about the research team’s claims about methane in New Mexico.

#1: EDF’s Permian Basin methodology is highly questionable.

EDF used differing methods of determining the emissions in the Permian than well sites in other parts of New Mexico.

Methane emission rates were measured at 93 production sites in the Permian – in both New Mexico and West Texas – from July to August 2018. From these sites, EDF determined site level emission factors (EF) – or the rate at which emissions were released from the site – and used them to quantify total emissions by multiplying these EFs by the total number of sites in the region. While this is seemingly straightforward, aspects of these calculations do raise some concerns.

The first issue is how emission rates at each of these sites were measured.

According to the analysis, the measurement of emissions rates was not actually taken onsite, but instead up to 200 meters away. Sites were selected based on the availability of a public road. There are several factors that could impact the accuracy of measurements taken more than 600 feet away from the well sites, including other potential emissions sources.

Additionally, the determination of how far away measurements were taken and at which sites was based on the use of a forward looking infrared (FLIR) camera. As the analysis states:

“Selected sites were quantified when there was public road access allowing the mobile laboratory to be parked approximately 50 – 200 meters downwind from potential emission sources. The research team used a FLIR optical gas-imaging camera to identify major emission sources such as storage tanks and to facilitate positioning of the vehicle within the plume.” (emphasis added)

EDF concedes that relying on public roads may have skewed the sampling:

Although limited public road access required some deviation from stratified random sampling, measured sites were generally representative of the population and selected without bias towards their potential emissions.” (emphasis added)

The measurement method used at these sites, EPA Other Test Method 33A (OTM 33A), also poses an issue, as it is one of the least reliable methods – a fact EDF admits.

The second issue is how these site-level measurements were then used to extrapolate basin-level emissions for oil and gas operations.

According to EDF’s methodology:

“Basin-level emissions were extrapolated from site-level measurements by a statistical bootstrapping approach that randomly sampled emission rates generated from the log-normal function with the best fit to measurement data. Below detection limit sites are censored by the analysis to fit the lower end of the distribution to a log-normal function. This approach better accounts for the fat-tail of the distribution than simple bootstrapping of measurements by including potential emission rates greater than those sampled with limited measurements.” (emphasis added)

Since the report is not peer-reviewed, much less published (yet), it is difficult to scrutinize how these “below detection limit sites” were captured. Did the “fat-tail” assumption lead researchers to “censor” lower-emitting sites as abnormal, regardless of how prevalent they may actually be? It is difficult to say either way, but this is also yet another example of why deeper scrutiny is needed for this kind of analysis – far more than what the headlines suggested it received.

Additionally, the study does not elaborate on the length of emissions measurement at each site, only stating that “measurements were performed July – August 2018.” This is important, as a study from 2017 notes, “emissions from production facilities are not constant and include non-trivial temporal complexities” such as liquids unloading. These activities are short lived, happening maybe a few times a year, meaning that if measurements taken over a short period at a site coincided with one of these activities, extrapolating annual emissions from that data would overstate the real emissions levels – perhaps severalfold.

This aligns with another critique of the so-called “fat tail” theory, published last year by the Proceedings of the National Academy of Sciences. That study indicated how “top-down” measurements could inflate emissions by improperly extrapolating normal operating conditions from a snapshot in time. Notably, EDF’s 2018 study – on which they relied heavily for this New Mexico analysis – relied heavily on “top down” measurements, once again suggesting that EDF’s numbers could be inflated.

The third issue is the small sample size of well sites.

According to EDF, a total of 93 sites were sampled and broken into two groups: “simple” sites which only had wellheads and/or pumpjacks, and “complex” sites which also included infrastructure such as storage tanks and compressors. Of the 93 production sites measured, just 42 were complex sites, with the remaining 51 being simple sites.

Emissions rates drawn from these 42 complex sites and 51 simple sites were then applied to about 8,600 complex sites and roughly 17,200 simple sites in the basin. That means EDF based their emissions assumptions on a sampling of just 0.5 percent and 0.3 percent of the overall sites, respectively.

#2: EDF did not use measurements from New Mexico to analyze emissions from the state’s San Juan and Raton basins.

As potentially troublesome as EDF’s Permian analysis is, the group did at least take site-level samples in the region. The same cannot be said for EDF’s emissions estimates in New Mexico’s San Juan and Raton basins, however.

According to the analysis methodology, EFs for those regions were determined using data from a previous EDF analysis, Alvarez et al 2018 – one which EID previously raised concerns over because of its claims of an abnormally high leakage rate. In short, these EFs were based on older measurements, rather than being sampled specifically for this analysis like EDF did for the Permian.

“For the San Juan and Raton basins, EFs were calculated with a gas production dependent, log-normal equation based on the methods outlined in Alvarez et al 2018, section S.1.1. The underlying data are from >400 site-level measurements from six U.S. basins (Barnett, Fayetteville, Marcellus, Uintah, Upper Green River, Denver-Julesburg). (emphasis added)

But the 2018 report used to determine EFs for the San Juan and Raton basins doesn’t even include measurements from those basins. In fact, it doesn’t include measurements from any basins in New Mexico. As EDF notes:

“Although the underlying data do not include site-level measurements from New Mexico, Alvarez et al 2018 reports that analogous estimates of San Juan Basin 2015 emissions agree closely with basin-level, aircraft-based measurements from the region (Smith et al 2017), suggesting that this combined dataset is broadly representative of the region.” (emphasis added)

#3: EDF placed most emissions into the vague category of “abnormal emissions.”

In addition to claiming yearly methane emissions in New Mexico increased by about 400,000 metric tons since its last report in 2017, EDF also notes that its latest estimate is about 800,000 metric tons larger than EPA’s most recent data.

To account for such a large jump, the analysis broke out the emissions levels for each source within the oil and gas development process. Confusingly, while the three largest specific emission sources listed – gathering stations, malfunctioning pneumatic controllers and leaks – were reported to account for 83,900 metric tons, 83,800 metric tons and 53,300 metric tons, respectively, the largest overall source EDF lists is “abnormal emissions.”

That category includes 648,700 metric tons, or about twice the amount of all other emissions sources combined.

Resting your claim of substantially higher emission levels on a vague, catch-all term like “abnormal emissions,” is baffling enough, but EDF’s explanation of how these abnormal emissions were derived is even more head scratching.

According to the analysis, the abnormal emissions total is the “difference between site-level and aggregate source-level emissions.” The site-level emissions were determined by the EDF team using the methodology outlined above, while the source-level emissions came from “a combination of data sources, including the EPA Greenhouse Gas Reporting Program (GHGRP) and previously published measurement studies.” EDF attributes this resulting difference to “emissions from malfunctions and otherwise avoidable conditions that may be associated with source categories.”

In other words, site-level emission estimates quantified in this analysis are so high relative to source-level emissions derived from EPA data and other studies, the only explanation, according to EDF, is that there’s a massive amount of “abnormal emissions” that these other studies and agencies missed. This raises an important question: If EPA data and other studies were able to account for much smaller emission sources in production like those from produced water, how could they all fail to catch such a substantial level of emissions?

EDF’s contention is that EPA is underestimating emissions by overlooking or omitted certain sources. But the fact that the majority of emissions that EDF “identified” (based largely on calculations and modeling, including from operating areas outside of New Mexico) could not be categorized should raise doubts about whether emissions really are this high, or if it’s actually a considerable overestimate.

Conclusion

Considering the number of assumptions made in this analysis, along with the use of site-level data that didn’t even come from New Mexico and reliance on a vague bucket of “abnormal emissions” to account for a high site-level emissions total, EDF’s latest findings in New Mexico raise more questions than answers – especially when recent state analyses have demonstrated considerable progress in reducing emissions across the state.