It is not just politicians that need to know this stuff - without it the whole debate is not possible.
11. Seek replication, not pseudoreplication
Results consistent across many studies, replicated on
independent populations, are more likely to be solid. There is nothing better
than good quality data and lots of it from different locations. Unfortunately
data is often man-power intensive and hence expensive. However, government and companies must be prepared to spend money on collecting that data if the general public are to trust their operations (and here).
Moreover, data from different scenarios or locations can
often be combined in a systematic review or a meta-analysis to provide an
overarching view of the topic with potentially much greater statistical power
than any of the individual studies. This requires that data is made freely available
between companies and to the general public as well as academics.
Since data is expensive and represents a commercial
advantage, companies are not likely to share it or make it available on their
own, however enlightened they are. Interestingly
Cuadrilla have released a large amount of water quality testing data here and here because they recognise that it represents part of the community patrimony. It
is hoped that this will continue. The government should take a central role in coordinating the archiving and publication of all shale gas data through, for example, the British GeologicalSurvey, but is currently avoiding it.
12. Scientists are human
It is not a case of companies bad, politicians bad,
activists bad, scientists good – scientists are human too. Although most
scientists take extreme care in balancing evidence and following a scientific
rationale, a few are less than candid. One must always remember that scientists
have a vested interest in promoting their work, often for status and further
research funding, and occasionally for direct financial gain. This can lead to
selective reporting of results and occasionally, exaggeration. Peer review is
not infallible: journal editors might favour positive findings and
newsworthiness.
All this adds up to the statement that scientists should not
be believed blindly nor their statements regarded dogmatically. If shale gas
extraction is to be carried out successfully, it needs the informed consent of
the local communities – informed consent means listening to the statements of a
range of scientists and others to form a balanced evidence-driven view upon
which solid decisions can be made.
13. Significance is significant
Opinion is not important. The only way of testing data is by
using valid statistical tests.
One of the most common ways of stating whether an effect,
such as whether hydraulic fracturing has contaminated an aquifer, is real is
the statistical significance or
P-value. The P-value is a measure of how likely a result is to occur by chance.
Thus P = 0.01 means there is a 1-in-100 probability that what looks like a link
(say fracking and aquifer contamination) actually occurred randomly. We would
call P=0.01 very significant as it also indicates that there is a 99-in-100
probability that the link is real. Usually P<0.05 is taken as the limit
where a link is considered to be proven.
So far we have no data in the UK that can be used to carry
out a test like this because there has been no fracturing where back-ground
data is available (no fracking was carried out at Balcombe). Similarly, no
background data is available in the USA and so proper statistical tests cannot
be carried out there either. However, such tests will be common in future, in
the UK at least, because companies are committed to carrying out before and
after water quality tests on aquifers.
14. Separate no effect from non-significance
The lack of a statistically significant result (say a
P-value > 0.05) does not mean that there was no underlying effect: it means
that no effect was detected. A small study may not have the power to detect a
real difference. For example, tests of local wild-life around the Balcombe
drilling site may suggest that it suffered no adverse effects from disturbance
by the drilling operation. Yet if the tests sampled too few animals it would
not have the power to detect impacts had there been any. Even then, it would be
extremely difficult to distinguish between disturbance by the drilling
operations and disturbance by the large number of protestors.
15. Effect size matters
Small responses are less likely to be detected and may fall
below the measurement sensitivity of whatever instrument is being used. However,
a study with many replicates might result in a statistically significant result
but have a small effect size (and so, perhaps, be unimportant).
Let’s take drilling or fracturing induced earthquakes. When
hydraulic fracturing is carried out it results in thousands of tiny earth
tremors by definition – the whole process is designed to make fractures in the
rock and each fracture formation is an earthquake, however small. These
earthquakes are mapped in the sub-surface by microseismic methods, and it is possible to see where each one occurs and to
delineate the fracture network that forms. If one correlated these earth
tremors with the hydraulic fracturing process, there would be, not
surprisingly, an extremely significant result -
an apparent smoking gun! However, all of these earthquakes have such a
small magnitude that they are never felt at the surface, and are hence unimportant
– in fact, a smoking pop-gun!
However, occasionally one earthquake might be big enough to
be felt at the surface, but it would not materially alter the significance of
the correlation. We must try to correlate problem earthquakes with hydraulic
fracturing, but so far there are just too few for this to be possible (only two
in the UK, and few in the USA where most of the bigger earthquakes associated
with shale gas are not due to hydraulic fracturing, but the irresponsible and
thankfully obsolescent habit of disposing of old fracking fluid by deep underground injection).
