Transcript Slide 1

Validation of Qualitative
Microbiological Test Methods
Pieta IJzerman-Boon (MSD)
Edwin van den Heuvel (TUe, UMCG/RUG)
NCS Conference
Brugge, October 2014
1
Contents
• Introduction
• Statistical Detection Mechanisms
• Validation Issues
• Likelihood-Based Inference
• Conclusions
2
Introduction
• Guidelines on validation do not agree
Validation parameters
Qualitative tests
Microbiological
guidelines
Accuracy and precision
EP
Repeatability
3
Analytical
guideline
USP
Specificity
EP/USP
ICH
Detection Limit
EP/USP
ICH
Ruggedness
USP
Robustness
EP/USP
Introduction
• In this presentation we will show an optimal
validation strategy:
–Compare methods
–Two dilutions
–Optimal densities for the two dilutions
–Required number of samples
• Optimal validation strategy differs substantially
from the guidelines
4
Statistical Detection Mechanisms
• Suppose a test sample is tested with a
qualitative test
• The sample contains X organisms
– X=0: sample is sterile
– X>0: sample is contaminated
• The outcome of the test is Z
– Z=1: positive test result
– Z=0: negative test result
5
Statistical Detection Mechanisms
• Classification of test result
Z=0
Okay
False
Negative
Z=1
Test Result
Number of Organisms
X=0
X>0
False
Positive
Okay
• So we need to look at the conditional probabilities
 x   PZ  1| X  x 
6
• The function describing this detection probability
is referred to as the detection mechanism
Statistical Detection Mechanisms
• Zero-deflated binomial mechanism:
h
 x   
x
1  1 h 1  p 
if
if
x0
x0
– h is the false positive rate:  (0)=h
– p is the detection proportion: if h=0 then it is the
probability to detect just one organism:  (1)=p
– If h=0 and p=1 the test method is perfect
– h and p are related to specificity and accuracy
– The binomial mechanism (h=0) was introduced in
Van den Heuvel and IJzerman-Boon (2013)
7
Statistical Detection Mechanisms
h =0, p=0.85
h =0.10, p=0.70
8
Validation Issues
• Estimate detection mechanism via experiments
– Exact low spikes of X cannot be generated
– Hence the detection probability  (x) cannot be
estimated, only the average proportion over samples
• Expected proportion of positive test results:
– Assume that the number of organisms X ~ Poi(l
E  X   1  1 h e pl
9
Validation Issues
• The detection proportion p cannot be estimated
– Without knowledge on the average number of
organisms l in the test samples
– With serial dilution experiments
• The false positive rate h can always be estimated
using samples from a blank dilution (l=0)
Compare alternate with compendial method
–Using the same l for both methods
–Likelihood ratio test (LRT)
10
Likelihood-Based Inference
Experimental Design
• Suppose we test samples from the same dilution
with two methods
– Alternate method: i=1
– Compendial method: i=2
– Dilution has on average l organisms per sample
– Number of samples tested per method: n
• Expected proportion of positive results now
depends on method i (i=1,2):
i  1  1 hi e p l
i
11
Likelihood-Based Inference
Experimental Design
• Asymptotic distribution of LRT for comparing these
2
proportions converges to 1 ( nc )-distribution with
nc  n
2( 1  2 )2
( 1  2 )( 2  1  2 )
• Hence, power can be optimized by maximizing nc
– Bacterial density l can be optimized independently
from sample size n
– There is a single optimal density
12
Likelihood-Based Inference
Experimental Design
Compendial:
h2=0.01
p2=0.95
13
Likelihood-Based Inference
Simulations
Simulation Results: Single dilution
• Average density l
• Detection proportions pAL=0.7 and pCM=1
• Power (%) of likelihood ratio test LRT for differences in
detection probabilities for various false positive rates
l
1.0
1.5
2.0
2.5
3.0
14
hAL=hCM=0
hAL=0.05, hCM=0
hAL=hCM=0.05
n=150 n=200 n=250 n=150 n=200 n=250 n=150 n=200 n=250
65.3
74.6
85.8
49.1
56.1
68.5
61.0
71.7
83.2
69.2
80.4
88.5
57.3
68.5
79.6
67.2
77.3
86.1
70.1
81.8
89.6
60.4
72.3
82.7
67.3
79.0
87.8
67.5
80.4
89.2
60.2
71.9
82.8
63.9
77.3
85.9
64.8
76.8
84.4
57.8
71.9
79.8
61.3
75.0
82.9
Conclusions
Optimal strategy when parameters are unknown
– Compare alternate with compendial method
– Two dilutions are needed
1. Blank dilution
2. Dilution with on average l~2 organisms
– Sample size should be at least n=200
15
• False positive rates can be tested with LRT
• Accuracy pAL/pCM can be tested with appropriate CIs
as an alternative for the LRT for the ratio pAL/pCM
(IJzerman-Boon and Van den Heuvel, 2014)
Conclusions
• Differences with guidelines
– Only specificity and accuracy need to be considered
– Two dilutions are needed, using five 10-fold dilutions
is a loss of power
– The optimal density is ~2 CFU/unit, ~5 CFU/unit is
much too high
– Use 200 instead of 5 samples per method and dilution
to detect a 30% drop in accuracy with 80% power
16
References
• IJzerman-Boon PC, Van den Heuvel ER, Validation of
Qualitative Microbiological Test Methods, Submitted, 2014.
• Van den Heuvel ER, IJzerman-Boon PC, A Comparison of
Test Statistics for the Recovery of Rapid Growth-Based
Enumeration Tests, Pharmaceutical Statistics, 2013; 12(5):
291-299.
• EP 5.1.6 Alternative Methods for Control of Microbiological
Quality
• USP <1223> Validation of Alternative Microbiological
Methods
• ICH Q2 (R1) Validation of Analytical Procedures
17