Biology, images, analysis, design...
|"It has long been an axiom of mine that the little things are infinitely the most important" |
Confidence limits when p or f are zero
Pity the poor researcher who, after his painstakingly-obtained large sample, does not observe any flies are infected - or that any individuals had leukaemia AND lived near an incinerator. It is no joke, and surprisingly common when dealing with rare phenomena. Having found yourself in that situation your immediate impulse may be to discard your data and forget the entire thing, or go back and gather a lot more observations - if such be possible.
Nevertheless, and extraordinary as it may seem, you CAN attach confidence limits to a zero count or zero proportion - formulae do exist. But rather than merely providing these, let us invest a few minutes considering why it is possible, what is being assumed, and how such intervals can mislead us.
For sake of argument let us assume we have a sample of (n=) 10000, of which (f=p=) 0 are positive - and try to obtain binomial and Poisson confidence limits. All the results below were obtained from standard formulae - except for the mid-P-exact, which used interpolated P-values from 100000 2-tailed tests, whose null parameters (P or F) were uniformly distributed.
Given these results you might reasonably wonder which of them is 'best' - especially if you regard exact tests as the 'gold standard'. In answering that we would be wise to consider what properties we would expect a 95% confidence interval to have. A logical place to begin this is the usual situation, where we assume our statistic has correctly estimated its parameter. After all, this is what we are assuming when we estimate the standard error of p from its observed value when attaching normal approximation limits, or estimating simple percentile bootstrap limits.
When p = f = P = F = 0
To avoid confusion, let us use P to indicate the true probability of obtaining a 1 (or the true proportion of infected insects) and use F to indicate (λ=) Pn - where n is the number of insects in our sample (even if n is extremely large and undefined).
One immediate conclusion is, if none are infected (P = F = 0), it would be a waste of time gathering more data. Even so, if a confidence interval could be calculated, what properties would it have? To answer this we have to assume the ONLY source of variation is due to random sampling (so there is, for instance, no assignment error). In that situation if none are infected, every sample will yield the same result, and applying the same formula will yield the same confidence limits.
The inescapable result of this is, depending upon which formula you chose, the resulting interval would ALWAYS enclose the parameter - or would NEVER enclose it.
Therefore, in the absence of any additional variation (such as 'jittering') we cannot validly calculate a 95% interval when p = f = P = F = 0 - irrespective of what method we use (approximate, score, exact, conventional, or mid-P). So in order to attach confidence limits to p = f = 0, we must assume
When p = f = 0 but P > 0 and F > 0
Now consider the situation where our estimate is NOT the same as the parameter being estimated.
To understand why this is so, you have to understand that exact intervals are equivalent to test inversion intervals. Consider for example the P-value plot below.
As you might expect, the mid-P upper confidence limit falls inside the conventional upper confidence limit. But we were unable to estimate a lower confidence limit for either conventional exact or mid-P exact. The reason for this is that no value of the test parameter (P) can be less than our observed value of p=0. Therefore the observed result could only be tested against the LOWER TAIL of the binomial null distribution - to test p against the upper tail, p must be greater than P (or F < f). As a result, both P-value plots are L-shaped rather than
This problem does not arise with 'normal' confidence limits because you are very unlikely to observe a result equal to minus infinity (the probability is 1/infinity) so you can always test your result against a parameter which is less than it. You could of course argue that if the true proportion of infected flies (P) is greater than zero, you could still have a lower confidence limit somewhere between 0 and P.
Upper one-sided confidence limits are very important theoretically, and are perfectly valid - provided the parameter lies between the lower bound (in this case P = F = 0) and a single limit, when calculated for 95% of samples.
Notice the 1-sided Clopper Pearson upper 95% limit is the same as Hanley's, and (because n was large) the conventional exact Poisson is approximately n times that value (F = Pn ≅ 3). When n is greater than 30 Hanley's 1-sided upper limit (
To simplify matters for ourselves, let us start by considering the situation where your sample comprises just (n=) 1 randomly selected binary value - in other words it represents a Bernoulli distribution, which is a special case of the binomial distribution where n=1.
Now recall that we are using P as shorthand for the probability that a randomly-selected observation equals 1, or
For binary values there is a very straightforward relationship between those two probabilities since the probability of observing f = 0,
In the right graph our confidence limit is the probability (P) of observing f=1 that results a P-value which equals α = 0.05 - let us call that probability Plim. Notice that, because n = 1,
Now consider what happens when, instead of just n = 1 binary value, your sample comprises n independently-selected values. Obviously, if n is large, you are correspondingly less likely to observe p = f = 0. Provided your selection was random and independent, given P is the probability of observing f = 1 after n = 1 event, the probability of observing f = n events is
Applying the same reasoning to mid-P-values, given the conventional P-value for n=1 observation is
Following our earlier reasoning, for a sample of n binary values,