Transcript Presentation - Pakistan Society Of Chemical Pathology

PAKISTAN SOCIETY OF CHEMICAL PATHOLOGISTS
DISTANCE LEARNING PROGRAMME IN CHEMICAL PATHOLOGY
LESSON NO 14
Method Validation and Quality
Management
(Short Name: MVQM)
BY
SURG COMMODORE AAMIR IJAZ
MCPS, FCPS, FRCP (EDIN)
PROFESSOR OF PATHOLOGY /
CONSULTANT CHEMICAL PATHOLOGIST
BAHRIA UNIVERSITY MEDICAL &DENTAL COLLEGE /
PNS SHIFA KARACHI
DR LENA JAFFERY, FCPS (CHEM PATH)
INSTRUCTOR AKU KARACHI
Extended Matching Questions
(EMQs)
a. Analytical Errors
m. Method Evaluation
b.
Analytical Measurement Range
n. Method Implementation
c.
Comparison of Method Experiment
o. Method Validation
d. Constant Systemic Error
p. Post-analytical Errors
e. Correlation Coefficient
q. Pre-analytical Errors
f. Delta Check
r.
g. Differential Plot
s. Random Error
h. F Test
t.
i. Instrument Calibration
u. Six Sigma
j. Internal Quality Control
v. T Test
k. Lower limit of Detection
w. Trouble Shooting
l. Lower limit of
Quantification
x.
Verification of Instrument
Calibration
y.
Z-Value
Proportional Systemic Error
SDI
Q:1. An important parameter given in National
External Quality Assurance Programme showing
performance of a participating lab and calculated by
following formula:
Lab Result- Method Mean
SD
Best match:
t. SDI
Standard Deviation Index (SDI)
• This term is commonly used in External Quality
Assurance (EQA) programmes.
• It is an indicator of accuracy of EQA samples as it
provides a comparison with target value.
• It is quite close to the statistical term Z-value or Zscore but this term is more generalized and may be
used in any situation.
Q:2. A Researcher has developed a new
method for his research project. He wants to
know whether his method is appropriate to
carry out research or not.
Best match:
m. Method Evaluation
Method Evaluation vs Method Validation
Method Evaluation
: Highly complex methods
should be studied more thoroughly. Any methods that are
modified or developed by the laboratory itself must be
evaluated extensively.
Method Validation:
The term used for most
moderately complex methods that have been well studied
by manufacturers as part of their own development
process. The laboratory, therefore ,can perform less
extensive studies to validate method performance.
Regulatory Laws and Bodies
CLIA: Clinical Laboratory Improvement Act (USA)
ISO: International Standard Organization
ISO 15189: Standard dealing with Clinical Laboratories
CAP: College of American Pathologists
PNAC: Pakistan national Accreditation Council
ISO Terminology
Measurand refers to the particular analyte or test.
Trueness: is used to describe the “closeness of agreement
between the mean obtained from a large serious of
measurements and a true value.” This is equivalent to the
terms bias and systematic error
Uncertainty describes a range of values that correspond to a
given test result, e.g., a test result of 200 may have a
“standard uncertainty” (SD, CV) of 4 units or 2%,.This concept
sounds and looks similar to precision, but the estimate of
uncertainty also incorporates any bias or trueness, thus it is
actually closer to the idea of total error.
Q:3. Chemical Pathologist Resident has
been assigned a task by her supervisor to
evaluate the effects of lipaemia on the
estimation of ALT.
Best match:
d. Constant Systemic Error
Interference Study
Constant Systematic Error
Factors contributing to constant error are independent of
analyte concentration, caused by;
 An interfering substance in all samples or in reagents that give
rise to a false signal.
 Non chemical source e.g. improper blanking of samples or the
reagents.
Interfering Substance
• Reaction between interfering substance and the reagent;
Lack of Specificity.
• Substance interfering in the reaction between reagent &
analyte (in coupled enzymatic methods using oxidase –
per oxidase reaction, hydrogen per oxide intermediate is
destroyed by endogenous reducing agents e.g. ascarbic
acid.
• Interfering substance may also inhibit or destroy the
reagent, so it remains in suboptimal amount for the
reaction with analyte.
Interfering Substance (cont)
•
Hemolysis
•
Lipemia
•
Icterus
•
Related compounds
•
Drugs
•
Dietary substances
•
Sample additives etc.
Evaluation Experiments for Estimating
Specific Types of Analytical Errors
Type of Analytical
Error
Constant
Systematic Error
Evaluation Experiments
Preliminary
Final
Interference
Comparison
with
Comparative
Method
Q:4. Chemical Pathologist Resident has
plotted the readings of Quality Control
material on a Levey-Jennings (LJ) plot
and applied Westguard rules.
Best match:
j. Internal Quality Control
Q:5. Consultant Chemical Pathologist
wants to determine the lowest level of
TSH which can be reported with accuracy
and precision..
Best match:
l. Lower Limit of Quantification
Verification of Limit of Detection (LOD) and
Limit of Quantitation (LOQ)
•
LOD is the smallest amount that the method can detect to
determine presence or absence of analyte.
•
LOQ is the smallest amount the method can measure
quantitatively.
•
Conventionally LOD is defined as the lowest value that
significantly exceeds the measurements of a blank sample.
Q:6. Defining the standard operating
procedures and documenting the
procedure, selecting an appropriate
Quality Control material for monitoring
routine performance, and training
personnel to operate the new method.
Best match:
n. Method Implementation
Q:7. Head of the Department of Chemical Pathology
has instructed all his residents to check whether the
adjustable pipettes of the Department are working
properly or not. They test the pipettes using
gravimetric measurement of various volumes of
distal water measured by these pipettes.
Best match:
x. Verification of Instrument Calibration
QC of Instruments
Instrument Calibration : Carried by professional
organization either at the time of manufacturing or
later as part of maintenance.
Verification of Instrument Calibration :
This is a procedure adopted by the end-user for routine
maintenance. It is also part of the QC of intsruments
which all lab staff should know and master.
Q:8. In a new Laboratory Reagent kit of Beta HCG
the Chemical Pathologist wants to determine the
highest level of the Beta HCG which can be
reported without dilution.
Best match:
b. Analytical Measurement Range
Analytical Measurement Range (AMR)
• Range of analyte values that a method can directly
measure on the specimen, without any dilution, or
other pretreatment, not part of the usual assay
process.
• AMR must be verified before a method is introduced,
and checked at least every 6 months (and after
recalibration or major maintenance) while in use.
AMR (cont)
• AMR verification must include three levels—low, midpoint, high.
• One can use commercial linearity materials, proficiency testing
(PT) samples or patient samples with known results, standards or
calibrators.
• It can also be done by calibration verification, if three samples that
span the measurement range are used.
• In the absence of commercial materials, one will need to create
one’s own materials. High and low samples can be mixed to create
a mid-point sample.
The linear-data plotter
•
It is used with the data collected in the linearity experiment,
where the purpose is to assess the analytical range over which
patient results may be reported.
• The response of the method is plotted on the y-axis versus the
relative concentration or assigned values of the samples or
specimens on the x-axis.
• The “reportable range” is generally estimated as the linear
working range of the analytical method.
Clinically Reportable Range (CRR)
• It is the range of analyte values that are reported
as a quantitative result, allowing for specimen
dilution or other pretreatment used to extend the
actual AMR.
• CRR is a clinical decision by the laboratory
director, and does not require experiments or revalidation; however, dilution or concentration
protocols must be specified in methods.
solutions and utilizing a curve-fitting routine to establish the calibration function.
nswered by determining the reportable range.
nderstand the terms and meanings in the CLIA regulations.
values on the x-axis, as shown in the figure below.
Q:9. In a tertiary care hospital a Consultant Chemical
Pathologist has joined the department after getting training
from Japan. Although the Quality Control (QC) and the
External Quality Assurance (EQA) results are acceptable,
he is not happy with the performance of the lab. He
launches a new programme for a marked reduction of the
lab errors and sets a target of < 3.4 errors per million.
Best match:
u. Six Sigma
What’s in a name?
• Sigma is the Greek letter representing the standard deviation
of a population of data
• Sigma is a measure
of variation
(the data spread)
σ
μ
What does variation mean?
• Variation means that a
process does not produce
the same result every time
20
15
10
5
0
• Some variation will exist in
all processes
-5
-10
• Variation directly affects customer experiences
The pizza delivery example
..
• Customers want their pizza delivered fast!
• Guarantee = “30 minutes or less”
• We measured performance and found an average
delivery time of 23.5 minutes?
• On-time performance is great, right?
• Our customers must be happy with us, right?
How Often are we Delivering on Time?
Answer: Look At The Variation!
30 min. or
less
s
0
10
20
x
30
40
50
 Managing by the average doesn’t tell the whole story. The average
and the variation together show what’s happening.
Introduction to six sigma
• Statistical measure of quality
• Based on rigorous process based performance
• Process for continuous improvement
• To improve process in any business
• Changes ways of thinking
• Creates a special infrastructure of people within the
organization
• Defects per million
• 1 million =1000,000
Relating sigma to defect levels
DPMO
(Defects per million
opportunities)
Error free rate
Six sigma
Five sigma
3.4
233
99.9997%
99.977%
Four sigma
Three sigma
Two sigma
6210
66810
308500
99.4%
93%
69%
One sigma
691500
31%
In short
• Processes that operate with "six sigma quality" over
the short term are assumed to produce long-term
defect levels below 3.4 defects per million
opportunities
Q:10. Lab Technologist has spiked a sample of
serum with 10% glucose and estimated the
difference in glucose concentration in this
sample before and after the spiking.
Best match:
r. Proportional Systemic Error
The Recovery Study
Proportional Systematic Error
• An error that is always in one direction, and the
magnitude of which is percentage of concentration of
analyte being measured.
• It is most often caused by incorrect assignment of the
amount of substance in the calibrator.
• If the calibrator has more analyte than is labeled (120
mg/dl glucose instead of 100 mg/dl which is labeled on
calibrator), all unknown determinations would be low, and
vise versa.
Proportional Systematic Error (Cont)
• Proportional error may also be caused by side reaction of
the analyte e.g. metabolite of the analyte also contributing
in the reaction
• Higher analyte concentration with higher metabolite giving
higher percentage of error.
Evaluation Experiments for Estimating
Specific Types of Analytical Errors
Type of Analytical
Error
Proportional
Systematic Error
Evaluation Experiments
Preliminary
Final
Recovery
Comparison
with
Comparative
Method
Q:11. On arrival of new stock of reagent kits of
Cholesterol, a Resident Chemical Pathologist
carries out an experiment lasting for 40 days.
After this experiment he calculates the bias
and applies ‘t Statistic’ for determination of
error.
Best match:
c. Comparison of Method Experiment
The Comparison Of Methods
Experiment
(The Mother Of Experiments)
Final Experiment for Accuracy
a. Patient samples with a wide range of values e.g.
Serum Cholesterol levels from 3.0 mmol/L to 7.0
mmol/L are used for the tests.
b. Every day 5-8 samples are estimated
c. Test lasts for 40 days
d. Then bias is calculated.
e. Statistics are applied.
Statistics used for Comparison of Method
•
Linear Regression Stat
•
t Statistics
Q:12. Pathologist has carried out Serum
Calcium level in a control material for 30
times consecutively and calculated Mean
and SD.
Best match:
s. Random Error
Random Error (RE) “Imprecision”
• An error either positive or negative, the direction & exact
magnitude of which can not be predicted.
• Factors contributing to random error are those that affect the
reproducibility of the measurement. These include
 Instability of instrument
 Variations in the temperature
 Variations in reagents & calibrators & calibration curve stability
 Variability in handling Techniques e.g. pipetting, mixing & timing
 Variability in operators
Estimation of RE from Replication Data
• It is commonly accepted that a minimum of 20 samples
should be measured in the time period of interest.
• A larger number of samples will give a better estimate of
the random error.
Statistics Calculation of RE from Replication Data
Statistics Calculation on the replication results
from the method being tested.
1. Mean
2. Standard Deviation (S)
3. Coefficient of Variation (CV)
Dispersion Stat
• The mean, or average of the group of results, describes the central location
of the measurements.
• The S describes the expected distribution of results, i.e., 66% are expected
to be within ± 1 S of the mean, 95% within ± 2 S of the mean, and 99.7%
within ± 3 S of the mean.
• The CV, or coefficient of variation, is equal to the S divided by the mean,
multiplied by 100 to express in percent.
• The histogram displays the distribution of results. Ideally, the distribution
should appear Gaussian, or “normal.”
Evaluation Experiments for Estimating Specific
Types of Analytical Errors
Type of
Analytical Error
RE
Evaluation Experiments
Preliminary
• The within-run S
• The within-day S
Final
Between-day S," or
"total" imprecision
Q:13. Plebani and Carraro have carried out series
of studies on nature and impact of laboratory errors.
Their data indicate that some types of errors
contribute greatly in inappropriate patient care.
These errors need thorough assessment and
prevention.
Best match:
a. Analytical Error
Errors in Lab Tests
• Commonest errors are Pre-analytical errors
• But Analytical errors lead to most of the inappropriate
patient management.
• The probable reason for this phenomena is that the
Analytical Errors are usually so subtle that they go
undetected and clinicians base their strategy on these
results.
• On the other hand Pre- and Post analytical errors are
mostly too obvious to be believed.
Q:14. Result of Glucose level of 23.7 mmol/L was forwarded
to a Chemical Pathology Resident. Although satisfied with
QC and EQA results, she was hesitant to authorize it. She
searched the Lab Information System to find any previous
results of the same patient but failed to find any. Then she
called the Medical Ward where the patient was admitted.
The Resident Medicine told her that the patient is a known
diabetic and is admitted with a gangrene right foot.
Best match:
f. Delta Check
Delta Check
• Delta check compare the current test result with a
previous result from the same test obtained over a short
period of time (within 96 hours) for the same patient
• A “delta check” failure or alert occurs if there is a
discrepancy in the patient results.
• When the difference between a patient’s present lab
result and their previous result exceeds a predefined limit
within a predefined length of time
Delta Check: Main Goal
Implications for the Patient
• Small delta value, or difference, in serial measurements:
Patient is stable (for that analyte)
• Large delta value (one or many) in serial measurements:
Why Bother Using Delta Checks?
• Early error identification have considerable implications for
patient care and safety
• Deadly errors due to incorrect drug dosing, anticoagulation
therapy, cardiac intervention, blood transfusion, etc. from
erroneous lab results
• Predictive value for detecting true specimen errors is between 0.4
and 6%
• Studies have found that the majority of delta check failures
(>75%) can be attributed to true changes in the patient’s
medical condition
• Providers need to be alerted to large biological variation in their
patients, may indicate need for intervention
Causes of Discrepant Results
Q:15. SD determined with 21 replications for a commercial
Triglycerides assay is 0.23 mmol/L while the data given in
the kit literature shows SD 0.18 mmol/L carried out on 42
replications. Chemical Pathologist uses a proper statistical
procedure to compare this variation and determine the
acceptability of the test kit.
Best match:
h. F-test
F-test
Used for comparison of Standard Deviations (SD)
• Calculate the F-value, larger SD squared divided by smaller
SD squared, e.g. (4)2/(3)2 = 16/9 = 1.78.
• Look up the critical F-value for 20 degrees of freedom (df=N-1)
in the numerator and 30 df in the denominator in the F-table,
where the value found should be 1.93.
• In this case, the calculated-F is less than the critical-F.
Q:16. The Internal QC data of Creatinine has shown
frequent violation of Wesgard`s Rules. The Resident
Chemical Pathologist asks the concerned Technologist to
constitute a new vial of QC material and repeat the test.
But there is no improvement. Then test was repeated after
sequential change of Standard, Reagent vial and the
measuring instrument until the desirable results were
achieved.
Best match:
w. Trouble Shooting
Q:17. To find out the outliers in a 40-day
procedure mentioned in Q.11, a graphical
method is used and these outliers are
discarded on daily basis.
Best match:
g. Differential Plot
What this Plot Indicates?
1. There is one suspicious
point in the difference
plot.
2. Note also that the points
tend to scatter above the
line at low concentrations
and below the line at high
concentrations,
suggesting there may be
some constant and/or
proportional systematic
errors present.
• For methods that are not
expected to show one-toone agreement, for
example enzyme analyses
having different reaction
conditions, the graph
should be a “comparison
plot”
• Displaying the test result
on the y-axis versus the
comparison result on the xaxis.
Short Answer Questions
(SAQs)
Q:18 What are the Westgard Rules
used for evaluation of an LJ Plot?
(plz see next slide)
Westgard Rules
12s : One control observation exceeds control limit set at 2SD is a
warning sign
13s : One control observation exceeds 3SD is a random error subject to
rejection rule
22s : Two consecutive control observations exceed 2SD is a systematic
error subject to rejection rule
R4s : One control observation exceeds the +2SD and the second control
observation exceeds –2SD is a random error subject to rejection rule
41s : Four consecutive readings crossing 1SD on one side is a
systematic error subject to rejection rule
10x : Ten consecutive control readings on one side of the mean is a
systematic error subject to rejection rule.
Q:19 You are required to carry out Method
Validation procedure of a Bilirubin kit:
a. Which types of Characteristics you would
like to perform.
b. Please write down the Performance
Characteristics you would like to examine for
this kit.
Suggested Answer to Q.19a
Which types of Characteristics you would
like to perform.
(plz see next slide)
.
Types of Characteristics of Method Validation
• Application characteristics :
• Methodology characteristics
• Performance characteristics
Suggested Answer to Q.19b
Please write down the Performance
Characteristics you would like to examine for
this kit.
(plz see next slide)
.
Performance characteristics
•
The reportable range
•
imprecision
•
bias vs the reference method
•
interference
•
Recovery
•
total errors less than 10%
•
Robustness
•
Limit of detection (LOD)
•
Limit of quantitation (LOQ)
•
Ruggedness
•
Selectivity
•
System Suitability.
Thank You and Best Of Luck