Transcript Inter-rater Reliability Diagnostic

Advanced Statistical Analysis in
Epidemiology:
Inter-rater Reliability
Diagnostic Cutpoints
Test Comparison
Discrepant Analysis
Polychotomous Logistic Regression
and
Generalized Estimating Equations
Jeffrey J. Kopicko, MSPH
Tulane University School of Public Health and Tropical Medicine
Diagnostic Statistics Typically
Assess a 2 x 2 contingency
table taking the form:
True + True - Total
Test + a
b
a+b
Test -
c
d
c+d
Total
a+c
b+d
a+b+c+d
Inter-rater Reliability
Suppose that two different tests
exist for the diagnosis of a
specific disease. We are
interested in determining if the
new test is as reliable in
diagnosing the disease as the old
test (“gold standard”).
Inter-rater Reliability continued
In 1960, Cohen proposed a
statistic that would provide a
measure of reliability between the
ratings of two different
radiologists in the interpretation
of x-rays. He called it the Kappa
coefficient.
Inter-rater Reliability continued
Cohen’s Kappa can be used to
assess the reliability between two
raters or diagnostic tests. Based
on the previous contingency
table, it has the following form
and interpretation:
Inter-rater Reliability continued
Cohen’s Kappa:
2(ad  bc)

(( a  c)(c  d )  (b  d )( a  b))
where
K > 0.75
excellent reproducibility
0.4<=K<=0.75 good reproducibility
0<=K<0.4
marginal reproducibility
*Rosner, 1986
Inter-rater Reliability continued
Cohen’s Kappa is appropriately
used when the prevalence of the
disease is low and the marginal
totals of the contingency table
are distributed evenly. When
these are not the case, Cohen’s
Kappa will be erroneously low.
Inter-rater Reliability continued
Byrt, et al. proposed a solution to
these possible biases in 1994.
They called their solution the
“Prevalence-Adjusted BiasAdjusted Kappa” or PABAK. It
has the same interpretation as
Cohen’s Kappa and the following
form:
Inter-rater Reliability continued
1. Take the mean of b and c.
(b  c)
m
2
2. Take the mean of a and d.
(a  d )
n
2
3. Compute PABAK using these
means and the original Cohen’s
Kappa formula and this table.
Yes
No
Yes
n
m
No
m
n
Inter-rater Reliability continued
• PABAK is preferable in all instances,
regardless of the prevalence or the
potential bias between raters.
• More meaningful statistics regarding the
diagnostic value of a test can be
computed, however.
Diagnostic Measures
•
•
•
•
•
Prevalence
Sensitivity
Specificity
Predictive Value Positive
Predictive Value Negative
Diagnostic Measures continued
Prevalence
Definition: Prevalence quantifies the proportion of
individuals in a population who have the disease
at a specific instant and provides and estimate of
the probability (risk) that an individual will be ill at
a point in time.
Formula:
(a  c)
prevalence 
(a  b  c  d )
Diagnostic Measures continued
Sensitivity
Definition: Sensitivity is defined as the probability
of testing positive if the disease is truly present.
Formula:
a
sensitivity 
( a  c)
Diagnostic Measures continued
Specificity
Definition: Specificity is defined as the probability
of testing negative if the disease is truly absent.
Formula:
d
specificit y 
(b  d )
Diagnostic Measures continued
Predictive Value Positive
Definition: Predictive Value Positive (PV+ ) is
defined as the probability that a person actually
has the disease given that he or she tests
positive.
Formula:
a
PV 
( a  b)

Diagnostic Measures continued
Predictive Value Negative
Definition: Predictive Value Negative (PV- ) is
defined as the probability that a person actually
disease-free given that he or she tests negative.
Formula:
d
PV 
(c  d )

Example: Cervical Cancer
Screening
The standard of care for
cervical cancer/dysplasia
detection is the Pap smear.
We want to assess a new
serum DNA detection test for
the Humanpapilloma Virus.
Pap +
DNA +
DNA Total
Pap 50
5
55
Total
35
410
445
Prevalence = 55/500 = 0.110
Sensitivity = 50/55 = 0.909
Specificity = 410/445 = 0.921
PV+ = 50/85 = 0.588
PV- = 410/415 = 0.988
85
415
500
Reciever Operating Characteristic
(ROC) Curves
Sensitivities and Specificities are used to:
1. Determine the diagnostic value of a test.
2. Determine the appropriate cutpoint for
continuous data.
3. Compare the diagnostic values of two or
more tests.
ROC Curves continued
1. For every gap in continuous data, the
mean value is taken as the cutoff. This is
where there is a change in the contingency
table distribution.
2. At each new cutpoint, the sensitivity and
specificity is calculated.
3. The sensitivity is graphed versus 1specificity.
ROC Curves continued
4. Since the sensitivity and specificity are
proportions, the total area of the graph is 1.0
units.
5. The area under the curve is the statistic of
interest.
6. The area under a curve produced by
chance alone is 0.50 units.
ROC Curves continued
7. If the area under the diagnostic test curve
is significantly above 0.50, then the test is a
good predictor of disease.
8. If the area under the diagnostic test curve
is significantly below 0.50, then the test is an
inverse predictor of disease.
9. If the area under the diagnostic test curve
is not significantly different from 0.50, then
the test is a poor predictor of disease.
ROC Curves continued
10. An individual curve can be compared to
0.50 using the N(0, 1) distribution.
11. Two or more diagnostic tests can be
compared also using the N(0, 1) distribution.
12. A diagnostic cutpoint can be determined
for tests with continuous outcomes in order to
maximize the sensitivity and specificity of the
test.
Nonparametric Reciever Operating Curve (ROC) Plot of Serum RCP Level
1
0.9
1.0
True Positive Fraction (Sensitivity)
0.8
0.9
1.1
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
True Negative Fraction (1-Specificity)
0.8
1
ROC Curves continued
Determining Diagnostic Cutpoints
optimum cutpoint 
SUP sensitivit y * specificit y
ROC Curves continued
Determining Diagnostic Cutpoints
Cut Point
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
Sensitivity Specificity Sens*Spec
1
0
0
1
0.04
0.04
1
0.2
0.2
1
0.4
0.4
0.980769
0.68 0.666923
0.923077
0.82 0.756923
0.923077
0.88 0.812308
0.865385
0.96 0.83077
0.846154
0.96 0.812308
0.807692
0.98 0.791538
0.788462
1 0.788462
0.788462
1 0.788462
0.730769
1 0.730769
0.730769
1 0.730769
0.730769
1 0.730769
Nonparametric Reciever Operating Curve (ROC) Plot of Serum RCP Level
1
0.9
1.0
True Positive Fraction (Sensitivity)
0.8
0.9
1.1
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
True Negative Fraction (1-Specificity)
0.8
1
ROC Curves continued
Diagnostic Value of a Test
a1  ao
z
~ N (0,1)
sea1  a0
where a1 = area under the diagnostic test curve,
ao = 0.50, se a1 is the standard error of the area,
and se ao = 0.00.
ROC Curves continued
Diagnostic Value of a Test
For the RCP example, the area
under the curve is 0.987, with a pvalue of <0.001. The optimal
cutpoint for this test is 1.1 ng/ml.
ROC Curves continued
Comparing the areas under 2 or more curves
In order to compare the areas under two
or more ROC curves, use the same
formula, substituting the values for the
second curve for those previously defined
for chance alone.
Nonparametric Receiver Operating Characteristic Curves for Different Tests of CMV Retinitis
1
0.9
0.8
0.7
Sensitivity
0.6
0.5
0.4
0.3
Antigenemia
0.2
Digene
Amplicor
0.1
Chance
0
0
0.1
0.2
0.3
0.4
0.5
1-Specificity
0.6
0.7
0.8
0.9
1
ROC Curves continued
Comparing the areas under 2 or more curves
For the CMV retinitis example, the Digene
test had the largest area (although not
significantly greater than antigenemia).
The cutpoint was determined to be 1,400
cells/cc. The sensitivity was 0.85 and the
specificity was 0.84. Bonferronni
adjustments must be made for >2
comparisons.
ROC Curves continued
Another Application?
Remember when Cohen’s Kappa was
unstable at extreme prevalence and/or
when there was bias among the raters?
What about using ROC curves to assess
inter-rater reliability?
ROC Curves continued
Another limitation to K is that it provides
only a measure of agreement, regardless
of whether the raters correctly classify the
state of the items. K can be high,
indicating excellent reliability, even though
both raters incorrectly assess the items.
ROC Curves continued
The two areas under the curves may be
compared as a measure of overall interrater reliability. This comparison is made
by applying the following formula:
droc = (1- |Area1- Area2|)
By subtracting the difference in areas by
one, droc is on a similar scale as K,
ranging from 0 to 1.
ROC Curves continued
If both raters correctly classify the objects
at the same rate, their sensitivities and
specificities will be equal, resulting in a
droc of 1.
If one rater correctly classifies all the
objects, and the second rater misclassifies
all the objects, droc will equal 0.
Statistics for Figure 1(N=20):
Rater One:
Rater Two:
% Correct = 80 %
% Correct = 55 %
sensitivity = 0.80
sensitivity = 0.60
specificity = 0.80
specificity = 0.533
Area under ROC = 0.80
Area under ROC = 0.567
droc = 0.7667
Monte Carlo Simulation
Several different levels of disease prevalence, sample
size and rater error rates were assessed using Monte
Carlo methods.
Total sample sizes of 20, 50 and 100 were generated
each for disease prevalence of 5, 15, 25, 50, 75, and 90
percent. Two raters were used in this study. Rater One
was fixed at a 5 percent probability of misclassifying the
true state of the disease, while Rater Two was allowed
varying levels of percent misclassification.
For each condition of disease prevalence, rater error,
and sample size, 1000 valid samples were generated
and analyzed using SAS proc IML.
Figure Two: Actual Percent Agreement (N=50)
1 - ROC Curve Difference
ROC Curves continued
1
0.95
0.9
0.85 Another limitation is that K provides only
0.8
measure
of
agreement,
regardless
of
0.75
0.7 whether the raters correctly classify the
0.65 state of the items. K can be high,
a
0.6 indicating excellent reliability, even though
0.55
both raters incorrectly assess the items.
0.5
0.05
0.15
0.25
0.5
0.75
Rater Two Error Probability
Prevalence:
0.05
0.15
0.25
0.5
0.75
0.9
Figure Four: Cohen's Kappa Coefficient (N=50)
ROC Curves continued
1
1 - ROC Curve Difference
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
If both raters correctly classify the objects
at the same rate, their sensitivities and
specificities will be equal, resulting in a
droc of 0.
If one rater correctly classifies all the
objects, and the second rater misclassifies
0.05 objects,
0.15 droc will
0.25 equal 0.5
0.75
all the
1.
Rater Two Error Probability
Prevalence:
0.05
0.15
0.25
0.5
0.75
0.9
Figure Five: PABAK Coefficient (N=50)
ROC Curves continued
1
1 - ROC Curve Difference
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
If both raters correctly classify the objects
at the same rate, their sensitivities and
specificities will be equal, resulting in a
droc of 0.
If one rater correctly classifies all the
objects, and the second rater misclassifies
0.05 objects,
0.15 droc will
0.25 equal 0.5
0.75
all the
1.
Rater Two Error Probability
Prevalence:
0.05
0.15
0.25
0.5
0.75
0.9
Figure Three: Difference in ROC Curves (N=50)
ROC Curves continued
1
1 - ROC Curve Difference
0.95
0.9
0.85 If
both raters correctly classify the objects
0.8
at the same rate, their sensitivities and
0.75
specificities will be equal, resulting in a
0.7
0.65 droc of 0.
0.6
0.55 If
one rater correctly classifies all the
0.5 objects, and the second rater misclassifies
0.05 objects,
0.15 droc will
0.25 equal 0.5
0.75
all the
1.
Rater Two Error Probability
Prevalence:
0.05
0.15
0.25
0.5
0.75
0.9
Based on the above results, it appears that the
difference in two ROC curves may be a more
stable estimate of inter-rater agreement than K.
Based on the metric used to assess K, a similar
metric can be formed for the difference in two
ROC curves. We propose the following:
1.0 > droc > 0.95
excellent reliability
0.8 < droc < 0.95
good reliability
0 < droc < 0.8
marginal reliability
ROC Curves continued
From the example data provided with Figure 1, it
can be seen that droc behaves similarly to K.
The droc from these data is 0.7667, while K is
0.30. Both result in a decision of marginal interrater reliability.
However, from the ROC plot and the percent
correct for each rater, it is seen that Rater One
is much more correct in his observations than
Rater Two, with percent agreements of 80 %
and 55 %, respectively.
ROC Curves continued
Without the individual calculation of the
sensitivities and specificities, information about
the correctness of the raters would have
remained obscure. Additionally, with the large
differential rater error, K may have been
underestimated.
The difference in ROC curves allows many
advantages over K, but only when the true state
of the objects being rated is known. Finally, with
very little adaptation, these methods may be
extended to more than two raters and to
continuous outcome data.
So, we now know how to assess whether a test
is a good predictor of disease, how to compare
two or more tests, and how to determine
cutpoints. But,
What if there is no established “gold-standard?”
Discrepant Analysis
Discrepant Analysis (DA) is a commonly used
(and commonly misused) technique of
estimating the sensitivity and specificity of
diagnostic tests that are imperfect “goldstandards.” This technique often results in
“upwardly biased” estimates of the diagnostic
statistics.
Discrepant Analysis continued
Example:
Chlamydia trachomatis is a common STI that
has been diagnosed using cervical swab
culture for years. Often, though, patients only
present for screening when they are
symptomatic. Symptomatic screening may be
closely associated with organism load.
Therefore, culture diagnosis may miss carriers
and patients with low organism loads.
Discrepant Analysis continued
Example continued:
GenProbe testing has also been used to
capture some cases that are not captured by
culture. New polymerase chain reaction
(PCR) and ligase chain reaction (LCR) DNA
assays may be better diagnostic tests. But,
there is obviously no good “gold-standard.”
Discrepant Analysis continued
Example continued:
1. Culture vs. PCR
2. Culture + GenProbe vs. PCR
3. Culture vs. LCR
4. Culture + GenProbe vs. LCR
…and many other combinations.
Discrepant Analysis continued
Example continued:
Goal is to maximize the sensitivity and
specificity of the new tests, since we think that
the new tests are probably more accurate.
Major limitation is that this is often seen as a
“fishing expedition” with the great possibility of
Type I error, and inflation of diagnostic
statistics.
Polychotomous Logistic Regression
Simple logistic regression is useful when the
outcome of interest is binomial (ie: yes/no,
male/female, etc.)
Linear regression is useful when the outcome
of interest is continuous (ie: age, blood
pressure, etc.)
But what if the outcome is categorical with
more than one level?
Generalized Estimating Equations
GEE is used when there are repeated
measures on continuous, ordinal, or
categorical outcomes, and there are different
numbers of measurements on each subject.
It is useful in that it accounts for missing data
at different time points.
The interpretation of the GEE model is the
same as all other regressions.