Transcript Proficiency Tests

Proficiency Tests
John H McB Miller
Laboratory Department (DLab)
European Department for the Quality of Medicines
Council of Europe
Strasbourg, France
1|
QCL Training Seminar, Tanzania | 5-7 Dec 07
Interlaboratory Studies
Interlaboratory Testing
Description
• Proficiency testing
• Continuing assessment of
technical competence
• Collaborative study
• Validation of a specific method
• Certification study
• Establishing the best estimate
of the time value of an analyte
in a reference material
• Co-operative study
• Laboratory assessment of
samples and methods (eg
educational studies)
2|
QCL Training Seminar, Tanzania | 5-7 Dec 07
Proficiency testing by interlaboratory comparisons
 To determine the competence of individual laboratories to
perform specific tests or measurements
 To monitor the performance of laboratories overtime
3|
QCL Training Seminar, Tanzania | 5-7 Dec 07
ISO Guide 43-1
Interlaboratory comparisons may be used to:
a) determine the performance of individual labs for specific
tests or measurements and to monitor labs’ continuing
performance;
b) identify problems in labs & initiate remedial actions which
may be related to, for example, individual staff performance
or calibration of instrumentation;
4|
QCL Training Seminar, Tanzania | 5-7 Dec 07
ISO Guide 43-1
c) establish the effectiveness & compatibility of new test or
measurement methods & similarly to monitor established
methods
d) provide additional confidence to lab clients;
e) identify interlaboratory differences;
5|
QCL Training Seminar, Tanzania | 5-7 Dec 07
ISO Guide 43-1
Interlaboratory comparisons are conducted for a number of
purposes and may be used by participating laboratories and
other parties eg establishment of a reference material
6|
QCL Training Seminar, Tanzania | 5-7 Dec 07
ISO Guide 43-2
6.
Use of results by laboratory accreditation bodies
6.1 The results from proficiency testing schemes are useful for
both participating laboratories & accreditation bodies.
There are, however, limitations ............. that proficiency
testing alone should not be used by laboratory
accreditation bodies in their accreditation processes.
7|
QCL Training Seminar, Tanzania | 5-7 Dec 07
ISO Guide 43-2
8|
6.2
If a laboratory submits a result(s) which fall outside
acceptance criteria for a specific scheme, a laboratory
accreditation body should have procedures for acting on such
results
6.3
Such procedures should include early reporting to the
laboratory of its results with an invitation for the laboratory to
investigate and comment on its performance.
QCL Training Seminar, Tanzania | 5-7 Dec 07
Quality Manual
Laboratories should have a section in their Quality Manuals:
- covering participation in proficiency testing
- how the results are used to demonstrate the competence of the
laboratory
- procedures to be followed when unsatisfactory performance is
reported
- records of participation in PT scheme
- corrective action reports
9|
QCL Training Seminar, Tanzania | 5-7 Dec 07
PT Scoring System
Z-score = ct
SD
c = calculated reported value
t = true value
SD = target
RSZ = Z
m
m = number of tests
RSSZ = Z2 m
10 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Assigned Value,

x


The assigned value ( x ) may be the “true” or the
consensus value.

True value may be a theoretical value or known value from
“spiking’ of known quantity of a known quantity of analyte
to a sample.
Consensus value based on the results of the participants.
The consensus value is determined by the application of
robust statistics (eg median value, mean interquartile
range, Huber’s robust mean), to avoid the influence of
“outliers” in the overall mean.
11 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Target standard deviation (TSD)
 This is set based on experience, reported or expected
precision of the techniques used and according to fitnessfor-purpose.
 The TSD must be realistic.
 The TSD should be consistently applied from round to round
for the same technique/procedure in a PT scheme so that
performance can be assessed over time.
12 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Ranking
Z-score ≼ 2 satisfactory
≽ 2≼ 3
>3
13 |
questionable
unsatisfactory
QCL Training Seminar, Tanzania | 5-7 Dec 07
Outliers are indicated using three test statistics:
Cochran’s test for outlying variances, Grubbs’ single test for
outlying means and Grubbs’ paired test for outlying means, to
be applied in this order. If a laboratory is excluded, the cycle is
repeated from Cochran’s test until no outliers remain.
14 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
CochranÕs test for out lying variances is applied as follows: Calculate for each
maxö2
laboratory the within laboratory variance ö2 and then
. Th is value is compared
ö2
with the tabled critical va lues for the number of individual determinations r. If it
exceeds the cr itical va lue, the la boratory with the maximum variance is removed and
the cyc le is continued with CochranÕstest.
15 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Grubbs’ single test for outlying means is applied as follows:
Calculate the between laboratory standard deviation ˆ of the
mean values. Calculate the standard deviations ˆ L and ˆ H
excluding the lowest and the highest mean value respectively
*
and let ˆ * be the smallest of these two values. Calculate 1  ˆˆ

and compare it with the tabled critical values. If it exceeds the
critical value, the corresponding laboratory is excluded and the
cycle continued with Cochran’s test
16 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
GrubbsÕ paired test for outlying means is applied as follows: Calculate the between
laboratory standard deviation ö of the mean values. Calculate the standard
deviations ö2 L , ö2 H and öLH excluding the two lowest, the two highest, and the
highest and the lowest mean value respectively and let ö* be the smallest of these
ö*
three va lues. Calculate 1 
and co mpare it with the tabled critical values. If it
ö
exceeds the cr itical va lue, the corresponding laboratories are excluded and the cycle
continued with CochranÕstest.
17 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Robust Statistics
Huber’s mean for calculation of the
« consensus » value is preferred to using
elimination of results by tests for outliers
18 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Scoring over time
RSD -
Used for detecting consistent bias
RSSZ -
Magnitude of deviations.
Cancellation of significant Z-scores if
opposite sign is limited
19 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
z
The rescaled sum of squared z-scores RSSZ is calculated as RSSZ = 
m
2
where m
is the number of squared z-scores added (one for each sample). Critical values of
RSSZ depend o n m according to the table:
20 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
m
2
3
4
5
6
21 |
‘Doubtful’
> 3.00
> 2.60
> 2.37
> 2.21
> 2.10
‘Unacceptable’
> 4.61
> 3.78
> 3.32
> 3.02
> 2.80
QCL Training Seminar, Tanzania | 5-7 Dec 07
Proficiency Testing
Corrective actions:
- PT schemes can be either mandatory or voluntary
- Voluntary schemes: PT records examined by external
auditors during ISO 17025 assessment
- Mandatory schemes: Corrective action reports must be set
within a defined time limit to the organisers for assessment
and approval (or not). Failure to do so will result in a sanction
22 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
OMCL Network
Proficiency Testing Scheme
 Initially open to OMCls of the European Union & other
OMCLs associated with the European Pharmacopoeia
(member & observer states)
 Now open to any lab. on a fee paying basis
 minimum of 4 tests/year
 voluntary scheme
23 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
WHO External Quality Assurance Assessment Scheme
(EQAAS)
Started in 2001 for selected regional medicines control
laboratories
Phase 4 began in 2007
24 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
PTS 20
Atomic Absorption Spectrophotometry
Incorrect programming of instrument resulting in
insufficient significant figures
Calibration
1 significant figure 1200 ppm K
3 significant figure 1290 ppm K
11/28 questionnaire/unsatisfactory results
25 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
PTS 025
Titration of the conjugate acid of organic bases
* Corrected for water content (not required)
A lab. reported a difference of 2% in results between
potentiometric & visual end-point. However, incorrect
standardisation procedure was employed
* Deterioration in response of the electrode
23/46 questionnaire/unsatisfactory results
26 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Liquid Chromatography Assay of Indapamide
98.33}
98.64}
98.11}
Mean = 98.7
RSD = 0.40
True value:- 99.75
z-score: 2.13
Repeatability of ref. sol. (n=6) :- 0.57
27 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
Liquid Chromatography Assay of Indapamide
Mg
20.28
28 |
Corrected
Area
Mean
²%
Content
1
2
-
-
-
-
1698
1691
1675
1668
1672
20.32
1669
1668
1643
1642
1643
20.79
1696
1701
1632
1636
1634
20.97
1716
1721
1637
1641
1639
< 1.0 98.65 99.46
20.56
1690
1696
1644
1649
1647
99.11 99.94
Ref.
Sol.
Test
Sol.
Area
1.8
98.33 99.1
QCL Training Seminar, Tanzania | 5-7 Dec 07
Phase 4 : Procedure 1 - Table 1
Raw data & scoring of participating laboratories
Semi-micro determination of water
29 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
30 |
QCL Training Seminar, Tanzania | 5-7 Dec 07
EQAS on water content by Karl Fischer
Number & percentage of participating laboratories having shown satisfactory
performance (z-score < 2)
31 |
Water content
Phase 1
(2001)
Phase 3
(2005)
Phase 4
(2007)
No of Participants
7
33
35
No of labs with
satisfactory results
4
19
23
% of labs with
satisfactory results
57%
58%
66%
QCL Training Seminar, Tanzania | 5-7 Dec 07
EQAAS
Comments:
32 |
-
From the 3 participants who failed on the first exercise, 2 of them did not
participate in the subsequent studies. Concerning the 3rd one their
results slightly improve Phase 3 and were satisfactory in Phase 4.
-
From the 14 participants who failed in the second exercise, 8 of them
reported satisfactory results on the 3rd one (3 did not participate and 3
didn’t show any improvement)
-
There doesn’t seem to be an improvement in the general trend. The
overall performance of laboratories using this technique is not very
satisfactory and could be improved. However, it has to be pointed out
that the determination of water by Karl-Fischer is problematic even for
experienced laboratories as we can see from the results reported by the
laboratories (including OMCLs) participating in our regular PTS
programme.
QCL Training Seminar, Tanzania | 5-7 Dec 07
References
 International Organization for S tandardization, ISO/IEC
Guide 43-1: 1997, Proficiency testing by interlaboratory
comparisons Ń Part 1: Development and operation of
proficiency testing schemes.
 International Organization for S tandardization, ISO/IEC
Guide 43-2: 1997, Proficiency testing by interlaboratory
comparisons Ń Part 2: Selection and us e of proficiency
testing schemes by laboratory accreditation bodies.
 International Organization for Standardizatio n, ISO 57252-1994: Statistical methods.
 Association of O fficial Analytical Chemis ts, Guidelines fo r
Collaborative Study Procedure to Validate Ch aracteristics
of a Method of Analysis, AOAC Ar lington, (1990).
 The Royal Society of Chemistry, Proficiency Testing in
Analytical Chemistry, Teddingto n, (1997).
 International Organisation of Standardisation ISO/IEC
Standard ISO13528 : 2005, Statistical Methods for use i n
Proficiency Testing by Interlaboratory Comparis ons.
33 |
QCL Training Seminar, Tanzania | 5-7 Dec 07