Why do we need a new approach?
A sad site in an Oxford graveyard.
(Photo: InfluentialPoints )
This highlights one danger of the P < 0.05 syndrome - where findings are classed as 'significant' or 'not significant' purely on the basis of their P-value. Important findings may be dismissed because P = 0.051, whereas a P-value of 0.049 may be taken to confirm someone's pet hypothesis - even if there is no apparent biological mechanism.
As Jacob Cohen (1994) put it in his paper entitled 'The earth is round (p < .05)' - "after 4 decades of severe criticism, the ritual of null hypothesis significance testing - mechanical dichotomous decisions around a sacred .05 criterion - still persists."
In recent years many have commented on the dire state of statistical analysis of biological research work. Altman (1994) noted that "huge sums of money are spent annually on [medical] research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis and faulty interpretation." Reviewing subsequent developments Bland (2010) concluded that, although publications in major [medical] journals had improved, this was not true of many specialist clinical journals.
In one of the most widely quoted papers of all time, Hurlbert (1984) highlighted the prevalence of pseudoreplication in the ecological literature. Depressingly Hurlbert (2009) noted the 'black art of pseudoreplication' was still widely practised. Such concerns have recently surfaced in the 'popular' scientific press (Siegfried, 2010 )
We suggest the reasons for the current rather dire situation include:
- Most biologists are taught how to do statistics (nowadays using a particular package) whilst the underlying reasoning is ignored.
- Simplistic and often erroneous interpretations of P-values (such as the P < 0.05 syndrome) saturate the literature making inference unsafe.
- Formulaic approaches are used to estimate variation rather than making full use of resampling Monte-Carlo approaches.
- Little attention is paid to ongoing, but fundamental, schisms among statisticians - and between biological disciplines.
- Issues such as study design, measurement error and confirmation bias (not to mention outright fraud) are commonly ignored.
- The 'publish or perish' attitude in academic institutions has led to an exponential growth in science publications, which is overwhelming the peer-review process - and critical appraisal is eschewed in order to get sufficient papers to publish.
Whilst 'publish or perish' is an intractable evil, improvements in the way biostatistics is taught could go some way to deal with the other factors...
- Altman, D.G. (1994). The scandal of poor medical research. BMJ 308, 283-284 (29 January). Full text
- Bland, J.M. (2010). Improving statistical quality in published research: the clinical experience. Talk presented at "Statistical Methods for Pharmaceutical Research and Early Development", Lyon, France, September 27-29, 2010. Full text
- Cohen, J. (1994). The earth is round (p < 0.05). American Psychologist 49 (12), 997-1003. Full text
- Hurlbert, S.H. (1984). Pseudoreplication and the design of ecological field experiments. Ecological Monographs 54 (2), 187-211. Abstract Full text
- Hurlbert, S. (2009). The ancient black art and transdisciplinary extent of pseudoreplication. Journal of Comparative Psychology 123 (4), 434-443. Abstract Full text
- Siegfried, T. (2010). Odds are, it's wrong. Science fails to face the shortcomings of statistics. Science News 177 (7), 26-29. Abstract Full text