InfluentialPoints.com
Biology, images, analysis, design...
Use/Abuse Principles How To Related
"It has long been an axiom of mine that the little things are infinitely the most important" (Sherlock Holmes)

 

 

Statistical Mistakes in Research

Use and misuse of statistics in biology

Statistics courses, especially for biologists, assume formulae = understanding and teach how to do  statistics, but largely ignore what those procedures assume,  and how their results mislead when those assumptions are unreasonable.

The resulting misuse is, shall we say, predictable...

 Note: These pages are excerpts from a biostatistics course for biologists.

For full details, including examples, see:

Dransfield R.D., Brightwell R. (2012)   Avoiding and detecting statistical malpractice: Design & Analysis for Biologists, with R.

InfluentialPoints, UK. [Access online] 

Use and Misuse of Statistics

Statisticians suggest that at least half the published papers in biology contain serious statistical mistakes. If you wish to draw conclusions from such material, or intend to base your research upon it, you may wish to identify those mistakes.

The pages (listed left)These pages  are a series of reviews on the use and misuse of various statistical methods, ranging from simple summary statistics, statistical tests & intervals, to use of appropriate study designs, and generalized linear modelling. These reviews are based upon a large non-random sample of published papers.

These pages summarise about 1000 examples of the several thousand papers we examined. In some cases there was insufficent information to judge the quality of the conclusions, analysis, or design. By far the most common misuses and mistakes we identified were also the most basic! This is not to say that complex techiques were used more sensibly - they were simply that much harder to check.

We did not attempt a representative survey (many of those are reported below), but merely wanted to illustrate good and poor practice in scientific publications. Our criteria were 1. were they fairly recent publications on medical, veterinary, ecological or wildlife research 2. was it possible to work out what analyses were done (this ruled out quite a few papers) 3. were they freely available on line or likely to be in any good university library. Whilst we admit to some bias towards papers in our respective specialities, we did not target particular authors or journals as being of good, or poor, quality.

We apologise to anyone who finds our comments unfair, we do not intend them to be so. If you think we have been too critical of some published work, bear in mind we are often only criticizing (or sometimes praising) a small part of the paper - other parts may contain some real gems of wisdom (or the converse). We leave it you to decide how unrepresentative or representative our selection has turned out to be.

 

What the statisticians say

(At least) seventy years of quotes complaining about messed-up statistical analysis ... quotes given in date order from most recent.

"We reviewed 513 behavioural, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling." Nieuwenhuis et al. (2011)

It's science's dirtiest secret: The "scientific method" of testing hypotheses by statistical analysis stands on a flimsy foundation.... Even when performed correctly statistical tests are widely misunderstood and frequently misinterpreted." Siegfried, T. (2010)

"Blaming statistics for misused statistics is like blaming medical science because of incompetent doctors. And suggesting Bayesian methods will make things better is like suggesting homeopathy should replace medicine." "My comment about homeopathy was meant to be a joke... I didn't mean to offend anyone." Larry Wasserman (2010) Mar. 20, 11:21am in response to Siegfried, T. (2010)

Most of the papers surveyed did not use randomisation (87%) or blinding (86%), to reduce bias in animal selection and outcome assessment. Only 70% of the publications that used statistical methods described their methods and presented the results with a measure of error or variability. Kilkenny C, et al. (2009)

"Standards in the use of statistics in medical research are generally low. A growing body of literature points to persistent statistical mistakes, flaws and deficiencies in most medical journals" Strasak et al. (2007)

The problem of poor statistical reporting is, in fact, longstanding, widespread, potentially serious, and not well known, despite the fact that most mistakes concern basic statistical concepts and can be easily avoided by following a few guidelines. Tom Lang (2004)

"It is often easier to get a paper published if one uses erroneous statistical analysis than if one uses no statistical analysis at all." Stuart Hurlbert & Celia Lombardi (2003)

"Almost every student of probability and statistics simply memorizes the rules. Most ... select their methods blindly, understanding little or nothing of the basis for choosing one method rather than another. This often leads to wildly inappropriate practices, and contributes to the damnation of statistics." Julian Simon and Peter Bruce (1999)

"We need much more explicit criticism of ecological and biological ideas, models, theories. We need much more consistent criticism of the experimental data used to support these ideas. Only by critical evaluation can we proceed to throw out those components of our thinking that are clearly wrong. Criticism of ideas is crucial. Criticism of our experimental results is the first step." Tony Underwood (1997)

"Like many men of my age, I mostly grouse. My harangue today is on testing or statistical significance, about which Bill Rozeboom (1960) wrote 33 years ago, "The statistical folkways of a more primitive past continue to dominate the local scene." And today they continue to continue." Jacob Cohen (1994)

"Amazingly, it is widely considered acceptable for medical researchers to be ignorant of statistics. Many are not ashamed (and some seem proud) to admit that they 'don't know anything about statistics'. "Huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis and faulty interpretation." Douglas Altman (1994)

The practice [of significance testing] seems to be promoted on the theory that scientists need to protect themselves and others against the dangers of thinking. After all, as Fleiss notes with alarm, if we allowed to think we might arrive at different interpretations... To the contrary, the notion that it is hazardous for scientists to think is itself an exceedingly dangerous myth. Charles Poole (1987)

"The percentage of statistical analyses in the ecological literature that are incorrect is high. Thesis advisors, journal editors, textbook writers and statistics professors have not done their job well." Stuart Hurlbert (1984)

"Most scientists today are devoid of ideas, full of fear, intent on producing some paltry result so that they can contribute to the flood of inane papers that now constitutes 'scientific progress' in many areas." Paul Feyerabend (1981)

"Biologists all too often act as if mathematical techniques possess magical powers, transforming incomprehensible raw data into clear and precise scientific conclusions. Nothing could be further from the truth." Michael Begon (1979)

The secret language of statistics, so appealing in a fact-minded culture, is employed to sensationalize, inflate, confuse, and oversimplify." Darrell Huff (1954)