Continuing the previous three postings here is the last set of 5 more "Things politicians need to know about shale gas science", inspired by the recent Guardian article entitled "Top 20 things politicians need to know about science" from an original article in Nature.
It is not just politicians that need to know this stuff - without it the whole debate is not possible.
16. Data can be dredged or cherry picked
Scientists are trained to keep an open and unbiased mind, but even they have a duty to be constantly mindful of what the evidence says, and not to interpret the evidence beyond its limits. One of the best tests of such a commentator is to ask whether all of the evidence supports his or her main point.
The authors of the article “Top 20 things politicians need to know about science” concern themselves with the needing to know whether the authors set out to test a sole hypothesis, or happening across a finding in a huge data set. Data that one happens across when not looking for it can be extremely good, useful and relevant. However, often it is not applicable to the argument because it applies to a different group/location/problem/population etc., or was obtained using inapplicable assumptions or using different premises.
The question one should ask is whether the study was designed to answer the particular question it is being used on, and if not whether there are differences that make its use inapplicable.
- natural variation between locations (due to different geological histories),
- plus sampling (sampling may be atypical because it is done in areas where problems are suspected),
- plus bias (the concentration of methane may depend on some other unknown factor),
- plus measurement errors (different testers using different methodologies in different locations, or simply using erroneous methods, inaccurate tools or uncalibrated tools).
Difference, even extreme ones, may be due to a combination of other factors than that in which you are interested.
For example, there are limits to the generalisations that one can make from US data when trying to predict the effect in the UK or Europe.
Similarly, many people fail to protect themselves adequately from the sun, in part because the sun is natural and because, for some of us, the benefit of a healthy glowing tan outweighs the risks of solar exposure. However, solar radiation is widely believed to be the leading cause of melanoma, which will kill an estimated 7,910 Americans this year (American Cancer Society (2004) Cancer Facts & Figures 2004. Atlanta, GA, USA: American Cancer Society).
Most disasters that damage the environment and take lives in Europe are due, in the last analysis, to more than one factor, which exacerbate each other. That is the reason why all new and unusual processes have to be considered extremely carefully. Shale gas operations qualify for special care simply because we have not carried many of them out in Europe.