Validating nominal data Sensitivity, specificity and related measures

Worked example

The data given here show the ELISA optical density readings for a total of 82 individuals. Of these the gold standard test indicated that 45 were infected whilst 37 were uninfected.

Optical density True status    Infected    Uninfected 06-10 0 3 11-15 1 9 16-20 4 15 21-25 8 7 26-30 14 2 31-35 9 1 36-40 7 0 41-45 2 0

The manufacturer of the ELISA test recommended a cut-off value of 20 optical density units - in other words any individual with a reading below or equal to 20 should be taken as negative; any individual with a reading above 20 should be taken as positive.

We will first follow the recommendations of the manufacturer and use a cut-off value of 20. This would give the following values for specificity, sensitivity and predictive values:

Worked example

Using the same data as above, we now calculate the overall accuracy for this diagnostic test:

Overall accuracy  =  ^{(40 + 27)} / ₈₂  =  0.89

This gives a biased estimate of accuracy because we have not corrected for chance agreement.

In order to correct the estimate, we first work out expected values for this table assuming there is no association between the test result and the true status. This is done by multiplying the proportions obtained from the column totals by the row totals as below:

The corrected measure of agreement (kappa) is then given by

κ = [0.89 − 0.51]/[1 − 0.51] = 0.78

Hence in this case, with a kappa value of 0.78, we still have a very good level of agreement even after correcting for chance agreement.

Another useful measure of overall accuracy is the likelihood ratio. We can calculate the positive and negative likelihood ratios from the values for sensitivity and specificity:

Sensitivity  =  0.89 Specificity  =  0.73

The positive likelihood ratio is well above 1, and the negative likelihood ratio is close to 0, indicating that the test has reasonably good discriminatory power.

Worked example

In the example above we used the recommended cut-off value of 20 optical units.

There are a number of different ways to select the cut-off value for a diagnostic test. We first briefly consider two of the older, more arbitrary methods of doing this. They are based only on the distribution of test values in healthy uninfected individuals.

The distribution of values in healthy individuals is assessed, for example with a histogram.

{Fig. 2}

Any value greater than the 95th percentile of uninfected individuals is considered abnormal; that value is therefore taken as the cut-off value. If however the distribution of known negatives was taken to be 'normal', and the mean and standard deviation were computed, the 95th percentile is equivalent to the mean plus twice the standard deviation. In this example this criterion, quite arbitrarily, sets the cut-off at 33.5.

Whilst these methods are simple, there is no biological basis for defining cut-offs on this basis. We would instead recommend use of ROC plots as detailed below.

Worked example

{Fig. 3}

A much better way to select the cut-off value is to use the ROC curve. It also provides an excellent measure of overall accuracy - the area under the curve (AUC). The ROC curve for our example is given here. If false-negatives and false-positives are equally undesirable, the optimal cut-off is that point closest to the upper left-hand corner of graph. In this graph that point lies at a cut-off of 20 to 25 units.

If you wish to maximise sensitivity at the expense of specificity, a cut-off further to the right on the ROC curve should be selected. Moving the cut-off further to the left would maximise specificity at the expense of sensitivity.

Estimating the area under the curve (AUC) can be done manually, but is very time consuming. Fortunately there are a number of software packages available that can do this for you - we give details of two such packages below. In this case the AUC is estimated at 0.89 indicating a good level of diagnostic accuracy.

In recent years a different form of the ROC curve has become popular, especially in veterinary research. This is the two-graph receiver operator characteristic curve (or two-graph ROC curve). Here both sensitivity and specificity are plotted against the cut-off value. The cut-off (d₀) at which the lines cross (and hence sensitivity equals specificity) optimises the accuracy of the diagnostic test. If distributions are symmetrical, this point also maximises the mean value of sensitivity and specificity.

{Fig. 4}

In this plot, the optimal cut-off is identified as 22 optical units where both sensitivity and specificity approach 0.8. Here the line plots of estimated sensitivity and specificity values are fairly smooth. However, this is often not the case, and there are various methods used to provide smoothed curves. We consider these briefly below.

1. Win-Episcope

   In the 'Cut-off value' module, the number of animals that are really positive or negative (true status) are entered for each value of antibody titre. The package displays the distributions of titres for infected and uninfected animals as seen in the first figure below. The cut-off value can then be scrolled to see the effect on sensitivity, specificity and predictive values. Confidence intervals are also given. The module then provides the ROC curve and calculates the area under curve (AUC). This is shown in the second figure below.

{Fig. 5}

This package has its strengths and weaknesses, namely

• it is very easy to use, and data can be entered directly to the package but:
• it will only accept a rather small number of values (20) for evaluating the cut-off value;
• it does not show the cut-off values on the ROC curve (for rational selection of the cut-off);
• it does not do two-graph ROC curves

2. CMDT

   CMDT was developed to aid selection of cut-off values and evaluation of quantitative diagnostic tests. Data are most readily entered to the program as Microsoft Excel spreadsheets. Both standard and two-graph-ROC analyses can be done. The figure below shows the output for the two graph ROC using the same data as for Win-Episcope.

{Fig. 6}

The cut-off value is estimated using two methods:

• In the non-parametric method, the range of observations is divided into intervals by 250 equidistant points. For each of these points the corresponding values of sensitivity and specificity are calculated. The cut-off point, d₀, is determined from these values.
• In the parametric method, it is assumed that relationships between sensitivity or specificity and optical density can be fitted by cumulative normal distribution functions - rather than using percentiles. The cut-off point lies at the intersection of these two curves.
Since the distributions here do approximate to normal, we have given the d₀ for the parametric method.

The vertical lines represent the intermediate range for parametric and non-parametric estimates of the cut-off point. Any readings within the intermediate range marked on the graph can be regarded as borderline for clinical interpretation of the test result. Their position is set by the user, who selects desired levels of sensitivity and specificity.

As with Win-Episcope this package has its strengths and weaknesses, namely:

• it will accept large data sets, and provides a comprehensive set of analyses but:
• it will only accept data in raw form, unlike the frequency table data accepted by Win-Episcope;
• it is not as easy to use (not least because the version we downloaded did not include help files!)