Biology, images, analysis, design...
Use/Abuse Principles How To Related
"It has long been an axiom of mine that the little things are infinitely the most important" (Sherlock Holmes)



Confidence intervals of risk ratio, odds ratio, and rate ratio: Use & misuse

Wald interval, exact intervals, independence of outcome

Statistics courses, especially for biologists, assume formulae = understanding and teach how to do  statistics, but largely ignore what those procedures assume,  and how their results mislead when those assumptions are unreasonable. The resulting misuse is, shall we say, predictable...

Use and Misuse

The large sample normal approximation confidence interval (known as a Wald interval) for each of these ratios is obtained by estimating the standard error of the log transformed ratio, multiplying it by 1.96 to obtain the upper and lower limits, and then exponentiating these limits to obtain the limits of the ratio. Small sample or (sometimes) exact intervals are also available. For the risk ratio the best approach is to invert a single two-sided test using the Wilson score statistic. For the odds ratio an exact interval can be attached to the conditional maximum likelihood estimate of the odds ratio, although a mid-P exact interval is preferable. Various methods are also available to approximate this exact interval. For the rate ratio an exact interval can be obtained by making results conditional on a fixed total of cases.

The risk ratio is appropriate for both cross-sectional and prospective studies, whilst the odds ratio is best restricted to case control designs. Rate ratios are commonly used for cohort designs and randomized trials. All these ratios are used heavily in medical and veterinary research, but rarely in other disciplines, and then usually when logistic regression is used for multivariate analysis. The latter factor contributes to over-use of the odds ratio for cross-sectional studies at the expense of the risk ratio.

We give  several examples of the use of the odds ratios in high prevalence situations where regarding the odds ratio as equivalent to a risk ratio can be highly misleading. For example one researcher claimed that open access increased the chance that an article was cited at least once by 3 times, when the risk ratio was only about 1.1! There is sometimes confusion over whether significance is best assessed by the P-value or by whether the confidence interval overlaps one. We feel it is unwise to use the confidence interval of a ratio as a surrogate for a statistical test.  If there is an appropriate null hypothesis to be tested, then the result of the appropriate statistical test should be given as a P-value in addition to the ratio estimate and interval. Usually (but not always) the two will agree, but whether they do or not has nothing to do with significance versus causality - as believed by one of our authors!

As with Pearson's  and Fisher's  tests, lack of independence of outcome is the commonest factor invalidating the intervals. A common cause of non-independence of outcome is the use of cluster sampling/randomization. We give some examples where the standard error was adjusted to allow for more complex sampling, but more often the 'herd effect' is ignored as in a mastitis study of cows, and a cysticercosis study in pigs. We also give an example of a trial of an insect growth regulator for controlling mosquitoes where randomization was by village and not by individual. Such an approach gives erroneously small standard errors and confidence intervals. The problem is of course worse if convenience  sampling is used. Another source of lack of independence is if all percentages add up to 100%.

As for the methods used to estimate the confidence intervals, Wald intervals are still most used, and are indeed probably adequate in many situations, providing sample sizes are moderate to large and proportions are not too extreme. Certainly conditional exact intervals to the odds ratio are too conservative to use for initial screening of factors prior to a multivariate analysis, as was done in one study on Salmonella infection in chicken houses. But we give some examples where Wald intervals were certainly not adequate, as in a study on brucellosis prevalence in bison. Score method confidence intervals were appropriate for the risk ratio used in the trial to assess screening for ovarian cancer. We have also included one example of use of the rather more reliable likelihood profile intervals for the odds ratio.


What the statisticians say

Lui (2004) gives a comprehensive account of the statistical estimation of epidemiological risk including the confidence intervals for risk, odds and rate ratios. Agresti (2002) gives extensive coverage of exact and large sample approximation confidence intervals for risk, odds and rate ratios. Rothman & Greenland (1998) provide in depth treatment of interval estimation. Collett (1991) covers many aspects of modelling binary data including attaching confidence intervals to effect measures. Kahn & Sempos (1989) deals mainly with large sample methods. Thrusfield (2005) gives a basic account of confidence intervals for risk and odds ratios for veterinarians.

Reiczigel et al. (2008) propose new exact unconditional confidence intervals for the difference and ratio of proportions. Lloyd and Moldovan (2007) propose exact one-sided confidence limits for the difference between two correlated proportions, whilst Agresti & Min (2005) provide simple improved confidence intervals for comparing matched proportions. King & Zeng (2002) provide an excellent description of the two main case control designs with estimation of parameters. Agresti & Min (2001) , (2002) look at small-sample confidence intervals for the difference between proportions, and for the relative risk and odds ratio. Agresti & Caffo (2000) propose a simple interval for the difference between proportions. Troendle & Frank (2001) and Agresti (1999) compare the different methods available to obtain unbiased confidence intervals for the odds ratio. Martin & Austin (1996) propose an exact interval for the rate ratio. Morris & Gardner (1988) describe how to calculate intervals for risk and odds ratios, and for standardized ratios. Gart & Nam (1988) review approximate interval estimation of the risk ratio. Miettinen & Nurminen (1985) propose improved approximate interval estimation of ratios. Koopman (1984) describes a likelihood-based approximation confidence interval for the risk ratio. Katz (1978) describes traditional confidence intervals for the risk ratio using a logarithmic transformation.

Wikipedia give the asymptotic large sample approximations of the confidence interval for the risk ratio, and the odds ratio. Tomas Aragon describes 'epitools', the R package for epidemiologic computing and graphics.