20 Things politicians should understand ... (Part 3)

Continuing the previous two postings here is the third set of 5 more "Things politicians need to know about shale gas science", inspired by the recent Guardian article entitled "Top 20 things politicians need to know about science" from an original article in Nature.  

It is not just politicians that need to know this stuff - without it the whole debate is not possible.

11. Seek replication, not pseudoreplication

Results consistent across many studies, replicated on independent populations, are more likely to be solid. There is nothing better than good quality data and lots of it from different locations. Unfortunately data is often man-power intensive and hence expensive. However, government and companies must be prepared to spend money on collecting that data if the general public are to trust their operations (and here).

Moreover, data from different scenarios or locations can often be combined in a systematic review or a meta-analysis to provide an overarching view of the topic with potentially much greater statistical power than any of the individual studies. This requires that data is made freely available between companies and to the general public as well as academics.

Since data is expensive and represents a commercial advantage, companies are not likely to share it or make it available on their own, however enlightened they are. Interestingly Cuadrilla have released a large amount of water quality testing data here and here because they recognise that it represents part of the community patrimony. It is hoped that this will continue. The government should take a central role in coordinating the archiving and publication of all shale gas data through, for example, the British GeologicalSurvey, but is currently avoiding it.

12. Scientists are human

It is not a case of companies bad, politicians bad, activists bad, scientists good – scientists are human too. Although most scientists take extreme care in balancing evidence and following a scientific rationale, a few are less than candid. One must always remember that scientists have a vested interest in promoting their work, often for status and further research funding, and occasionally for direct financial gain. This can lead to selective reporting of results and occasionally, exaggeration. Peer review is not infallible: journal editors might favour positive findings and newsworthiness.

All this adds up to the statement that scientists should not be believed blindly nor their statements regarded dogmatically. If shale gas extraction is to be carried out successfully, it needs the informed consent of the local communities – informed consent means listening to the statements of a range of scientists and others to form a balanced evidence-driven view upon which solid decisions can be made.

13. Significance is significant

Opinion is not important. The only way of testing data is by using valid statistical tests.

One of the most common ways of stating whether an effect, such as whether hydraulic fracturing has contaminated an aquifer, is real is the statistical significance or P-value. The P-value is a measure of how likely a result is to occur by chance. Thus P = 0.01 means there is a 1-in-100 probability that what looks like a link (say fracking and aquifer contamination) actually occurred randomly. We would call P=0.01 very significant as it also indicates that there is a 99-in-100 probability that the link is real. Usually P<0.05 is taken as the limit where a link is considered to be proven.

So far we have no data in the UK that can be used to carry out a test like this because there has been no fracturing where back-ground data is available (no fracking was carried out at Balcombe). Similarly, no background data is available in the USA and so proper statistical tests cannot be carried out there either. However, such tests will be common in future, in the UK at least, because companies are committed to carrying out before and after water quality tests on aquifers.

14. Separate no effect from non-significance

The lack of a statistically significant result (say a P-value > 0.05) does not mean that there was no underlying effect: it means that no effect was detected. A small study may not have the power to detect a real difference. For example, tests of local wild-life around the Balcombe drilling site may suggest that it suffered no adverse effects from disturbance by the drilling operation. Yet if the tests sampled too few animals it would not have the power to detect impacts had there been any. Even then, it would be extremely difficult to distinguish between disturbance by the drilling operations and disturbance by the large number of protestors.

15. Effect size matters

Small responses are less likely to be detected and may fall below the measurement sensitivity of whatever instrument is being used. However, a study with many replicates might result in a statistically significant result but have a small effect size (and so, perhaps, be unimportant).

Let’s take drilling or fracturing induced earthquakes. When hydraulic fracturing is carried out it results in thousands of tiny earth tremors by definition – the whole process is designed to make fractures in the rock and each fracture formation is an earthquake, however small. These earthquakes are mapped in the sub-surface by microseismic methods, and it is possible to see where each one occurs and to delineate the fracture network that forms. If one correlated these earth tremors with the hydraulic fracturing process, there would be, not surprisingly, an extremely significant result -  an apparent smoking gun! However, all of these earthquakes have such a small magnitude that they are never felt at the surface, and are hence unimportant – in fact, a smoking pop-gun!

However, occasionally one earthquake might be big enough to be felt at the surface, but it would not materially alter the significance of the correlation. We must try to correlate problem earthquakes with hydraulic fracturing, but so far there are just too few for this to be possible (only two in the UK, and few in the USA where most of the bigger earthquakes associated with shale gas are not due to hydraulic fracturing, but the irresponsible and thankfully obsolescent habit of disposing of old fracking fluid by deep underground injection).

20 Things politicians should understand ... (Part 2)

Continuing the previous posting here are 5 more "Things politicians need to know about shale gas science", inspired by the recent Guardian article entitled "Top 20 things politicians need to know about science" from an original article in Nature.  

It is not just politicians that need to know this stuff - without it the whole debate is not possible.

6. Regression to the mean can mislead

Extreme patterns in data are likely to be, at least in part, anomalies attributable to chance or error. The next count is likely to be less extreme. There is the tendency in any debate where passions run high and positions are entrenched for either side to grab hold of extreme data and either plug it or lambast it depending on whether it supports their position or not. This is not a rational scientific approach.

The Cuadrilla drilling in Lancashire caused, it is generally agreed, two small earthquakes (magnitudes 2.3 and 1.5). It would not be reasonable to take this observation as being typical of what will happen in all cases of drilling and hydraulic fracturing. In fact, it is likely that most hydraulic fracturing will not cause even earthquakes of this magnitude. On the other hand, if sufficient hydraulic fracturing operations were to be carried out there would be rare occasions when larger earthquakes will be triggered. That is the reason why the government has instituted a traffic light system which sets the threshhold for the freezing of operations at a low magnitude (M=0.5), which is an earthquake that is 32 times smaller than the smaller of the two Lancashire earthquakes and 58 times smaller than the larger.

7. Extrapolating beyond the data is risky

Patterns found within a given range do not necessarily apply outside that range. The range maybe a measurement or may be a location.

In the first case it may simply be that if we calculate that a well corrodes at a certain rate over 1 year, it is not necessarily the case that it will corrode five times as much over 5 years. It might be significantly less! It might me significantly more!!

In the second case, and it should be reasonably obvious, it is not possible to apply observations made in the USA with predicted causes in the UK or elsewhere in Europe, or in fact anywhere other than close to where the original observations were made: The rocks are different, their properties are different, the temperature and pressure is different, the working practices are different and so on.

8. Beware the base-rate fallacy

This is a more technical point. The ability of an imperfect test to identify something depends upon the likelihood of its occurrence (the base rate). For example, an image log (one of the measurement tools that is placed in a well) might be able to identify fractures in the rock with a 99% accuracy, and might identify active fractures in this way, yet it might still be unlikely that the fractures will reactivate when hydraulically fractured.

9. Controls are important

A control group is dealt with in exactly the same way as the experimental group, except that the treatment is not applied. Without a control, it is difficult to determine whether a given treatment really had an effect.

This is really important in understanding most of the existing studies related to hydraulic fracturing. For example, there are no studies of aquifer contamination from the USA where the aquifer water was systematically measured before the hydraulic fracturing started. Hence, it is impossible to say whether any measured contamination after hydraulic fracturing is due to or related to that drilling or whether it pre-existed. That is a fundamental fact of science. Without the initial background levels acting as a control, the post fracturing measurements are meaningless from the point of view of attributing the source of the contamination.

10. Randomisation avoids bias

Experiments should, wherever possible, allocate individuals or groups to interventions randomly. This is a statistical ideal that is difficult to apply in geoscience. We cannot randomly chose a location to do the drilling or hydraulic fracturing because we need to drill where we think there is some chance of success.  It would not be useful, for example, to drill in Brent, where there is no shale gas, despite Brent's recent political posturing.

However, when tests are done, it is important that the companies take account of any reason why their location might not be 'typical' such that it gives odd or extreme values. For example, companies should not drill where there is a known set of major faults, whether they are seismically active or not. In fact it is in the companies' interest not to do this for a whole raft of practical, financial, safety and public relations reasons.

20 Things politicians should understand about shale gas science (Part 1)

In the light of the recent Guardian article entitled "Top 20 things politicians need to know about science" from an original article in Nature, and inspired by it, here are the first five of their points but with particular emphasis on shale gas extraction. But its not just politicians that need to know this stuff - without it the whole debate is not possible.

 

1. Differences and chance cause variation

The real world varies unpredictably. For some branches of science such as physics, the questions may be reduced to very simple experiments whose results are more straightforward to interpret. In Earth Sciences, as in Life Sciences, we cannot simplify the complexity of Nature, and hence scientific results may seem more open to interpretation. The important thing is to recognize that there is a natural complexity and variability and take that into account in the interpretation of scientific observations. There is, for example, a variation of the amount of natural methane in aquifer waters. We need to understand that before we can attribute methane in drinking water to shale gas extraction.

 

2. No measurement is exact

Practically all measurements have some error; and let's be candid here, errors are not bad things but a recognition that there is a limit to what we can do.

Some things cannot be measured very well. Imagine you want to weigh 10 grammes of salt but you only have a 5 kg kitchen scale, the chances are that you might weigh out anything between 1 gram and 50 grammes even if you are careful as can be.

Some things can be measured with incredible accuracy: Arsenic in drinking water can be measured accurately to about 10 parts per trillion! (The US EPA sets its safety threshhold for arsenic in drinking water at 10 parts per billion because it believes that arsenic is cummulatively dangerous at higher levels and it knows it can accurately measure these amounts.)

The good news is that measurement errors can be quantified and quoted easily. You should NOT trust any measurement unless it has an associated measurement error especially if the argument rests on the value of the measurement.

 

3. Bias is rife

Experimental design or measuring devices may produce atypical results in certain circumstances. The corollary is that it is not sufficient to just take the results of a study, but to understand how it was carried out.

For example, a study of gas in drinking water may show that there is more methane within 100 m of a shale gas well. An interpreter (politician, activist, scientist) might then say "shale gas is leaking into the aquifer and contaminating it." This is wrong. There has been no distinction made between thermogenic methane (shale gas, formed at depth by heat) and biogenic methane (naturally occurring methane in aquifers formed at shallow layers by bacteria).

Perhaps if the analysis showed that most of the gas was biogenic (which is actually the case), the interpreter may then say "As the gas is biogenic it was not caused by shale gas extraction." This may also be wrong because the drilling, though not contaminating the aquifer with shale gas from a deep provenance, has disturbed shallow biogenic gas in a way that it has entered the aquifer temporarily.

The results of studies should be carefully studied for interpretation bias.

 

4. Bigger is usually better for sample size

The average taken from a large number of observations will usually be more informative than the average taken from a smaller number of observations. That is, as we accumulate evidence, our knowledge improves. The problem with shale gas is two-fold:

• Most of the concerns come from the practice of shale gas extraction in the USA.
• Almost no scientific studies have been carried out there, although the situation is slowly improving, most opinion is not based on evidence.

It is extremely irresponsible to extrapolate the US situation to the UK and other parts of Europe for a number of reasons (population density, different shales, regulation and data gathering). There is already more high quality independent publicly available scientific data from the few wells drilled in the UK than for all the thousands of wells in the States. That is a result of the responsible attitude of the companies and government protection agencies. We need public overview to ensure that it continues.

 

5. Correlation does not imply causation

It is tempting to assume that one pattern causes another. However, the correlation might be coincidental, or it might be a result of both patterns being caused by a third factor – a “confounding” or 'lurking' variable. For example, it is tempting to believe that methane exists in aquifers because of shale gas drilling, and it is important to find out if that is true. However, we have already seen that an inability to discriminate between two types of gas (thermogenic and biogenic) can lead to misinterpretation, and acts as a 'lurking' variable.

Early studies in the US were not very good because they had not measured the methane in aquifers before shale gas extraction started and hence could not be sure that what they were measuring was natural or as a result of the drilling. These studies relied on the association of a rise in groundwater methane close to wells (other more recent studies have also found the opposite).

However, even if it were true that there is a correlation between well position and high levels of groundwater methane it does not imply that the drilling caused the groundwater gas concentrations. It may simply be that the wells were placed to extract shale gas at a position where gas has been reaching the surface naturally for millions of years. In other words, a well placed well.

In this example correlation does not imply causation, though:

• causation may exist too - more study needs to be carried out if this is suspected, and
• if gas has been reaching the surface naturally (not caused by the drilling), how is this the case? Are there natural pathways, fractures and faults that ease the transport of the gas? A responsible producer would be using science to have the best solution to these questions to ensure that the drilling operations did not exacerbate the effect.