Transcript Searching for the Holy Grail of a PSD Profile Comparator

Searching for the Holy Grail
of a Single PSD Profile Comparator
Douglas S. Lee, Ph.D
Nonclinical Statistics & Biostatistical Applications
Pfizer Global Research & Development
1
Introduction
•
The FDA is interested in the problem of determining the equivalence of
aerodynamic particle size distributions (PSDs) between a generic and
innovator product as well as part of a suite of tests to demonstrate
chemical and manufacturing control.
FDA (1999) Draft Guidance: Bioavailability and
Bioequivalence Studies for Nasal Aerosols and Nasal Sprays
for Local Action .
Proposed statistic removed
while the PQRI process
identifies the best science
FDA (2003) Current Draft Guidance: Bioavailability and
Bioequivalence Studies for Nasal Aerosols and Nasal Sprays
for Local Action.
ACPS
Product Quality Research
Institute
PSD Profile Comparison
Working Group
Future FDA guidance/recommendations for statistical analysis
DSLee 860-441-0745
2
Introduction
Aerodynamic Particle Size Distribution – though involving
measurement by impaction or impingement - is defined by
the agency on a categorical basis and includes sites of
deposition where the particles are not physically sized
(e.g. valve/actuator, throat, pre-separator, and case).
50
Mean (%)
40
30
20
10
0
0
5
10
15
as
e
C
r
Fi
lte
7
St
g.
6
St
g.
5
St
g.
4
St
g.
3
St
g.
2
St
g.
1
St
g.
0
St
g.
ro
at
20
Th
Std. Dev. (%)
•
DSLee 860-441-0745
3
Introduction
•
To infer that the mean test profile is suitably equivalent to the mean
reference profile, the null hypothesis that the mean profiles are different
has to be tested and rejected . . .
55
50
45
40
35
30
25
20
15
10
5
0
Reference Mean Profile
Test Mean Profile
at
o
r
Th
g.
t
S
0
g.
t
S
1
g.
t
S
2
g.
t
S
3
g.
t
S
4
g.
t
S
5
g.
t
S
6
g.
t
S
7
r
li te
F
C
e
as
DSLee 860-441-0745
4
Introduction
•
. . . taking stage to stage variation – as well as correlation in variances
between stages - into account.
55
50
45
40
35
30
25
20
15
10
5
0
“Spaghetti” Plot of Individual PSDs
at
o
r
Th
g.
t
S
0
g.
t
S
1
g.
t
S
2
g.
t
S
3
g.
t
S
4
g.
t
S
5
g.
t
S
6
g.
t
S
7
r
li te
F
C
e
as
DSLee 860-441-0745
5
Introduction
•
There is a strong desire for a “one-size-fits-all” test that
results in a single test value that can be compared against a
single critical value.
The “one-size-fits-all” aspect of the desired test
is the Holy Grail . . .
DSLee 860-441-0745
6
Outline
•
This presentation . . .

Outlines the algorithm proposed in the 1999 draft guidance.

Describes the results of an investigation on the performance of
the proposed procedure in the equal formulation or equal
aerosol performance case (e.g. where the sets of reference and
test PSD profiles are equal).

Highlights technical issues with the algorithm as originally
proposed.
DSLee 860-441-0745
7
Disclaimer
•
The observations include work executed in support of the efforts of
the PQRI working group but may not necessarily reflect the final
consensus opinion of the group. The PQRI process is currently ongoing.
!!!?
@%!!
&%!!@
?*#
##@!!
DSLee 860-441-0745
8
FDA Algorithm
•
•
•
Obtain three lots each of
a candidate (test) and
comparator (reference)
product.
Ref. Lot 1
Ref. Lot 2
Ref. Lot 3
Test Lot 1
Test Lot 2
Test Lot 3
Select 10 inhalers from
each lot at random and
collect a PSD profile
from each.
Pool the 30 test product
profiles into one group
and pool the 30 reference
product profiles into
another group.
Reference
Test
DSLee 860-441-0745
9
FDA Algorithm
•
•
Test
Select one profile at
random from the test
group and two profiles
from the reference
group (such that R R’)
to create a TRR’ triplet.
Reference

X Ri  X Ri 

 X Ti 

S
2




X Ri  X Ri 

i
0.5 *  X Ti 

2


2
Calculate the difference
between the test profile
and the average of two
reference profiles as a
c2 statistic:
dTRR
DSLee 860-441-0745
10
FDA Algorithm
•
Reference
Also, calculate the difference
between the two reference
profiles as a c2 statistic:

X Ri  X Ri 
d RR  
i 0.5 * X Ri  X Ri 
S
•
Calculate the ratio (rd) of
the two c2 statistics as:
2
dTRR
rd 
d RR
DSLee 860-441-0745
11
FDA Algorithm
Repeat the process of selecting TRR’ triplets from the Test and
Reference pools, 500 times with replacement and then calculate the
mean ratio (Rdavg.) from the 500 individual (rd) ratios.
T
R R'
rd1
rd2
rd3
rd4
rd5
rd6
rd7
rd8
frequency
•
Average of 500 rd's = Rdavg.
0
2
4
6
8
10
rd
rd500
DSLee 860-441-0745
12
FDA Algorithm
•
•
Repeat the process of selecting 500 triplets and calculating Rdavg.
300 times (again with replacement) to obtain the 95th percentile of
the distribution of Rdavg..
Compare this last test statistic against a critical value to test the null
hypothesis of non-equivalence.
Test statistic:
Critical Value
95th percentile of
the distribution
of 300 Rdavg.'s
DSLee 860-441-0745
13
FDA Algorithm
•
•
The intent of the algorithm is to obtain an estimate of the variation in
the profile difference of the test and reference groups scaled by the
variation in the reference group.
Large values of the ratio imply that

the distance between the mean profiles of the test and reference
groups is large
and/or

the profile to profile variation of the test group is greater than that
of the reference group.
DSLee 860-441-0745
14
PQRI Activities
•
•
Initially, the agency was interested in validating a critical value to which
the test statistic could be compared.
After discussion, the working group decided to start at the beginning
and examine the performance of the algorithm in the equal
formulations case - where there is no difference in the mean profiles of
the test and reference groups - with respect to:



•
general profile shape,
pattern of stage to stage variation,
and number of stages
The objective was to begin characterization of the statistical properties
of the algorithm in order to determine the feasibility of defining critical
values based on test parameters – rather than upon aspects of the
samples themselves.
DSLee 860-441-0745
15
Experimental Design approach to examining the statistical
performance of the FDA algorithm for the equal formulation case
•
We defined three key parameters for defining sets of simulated
rank-ordered PSD profiles in terms of shape and pattern of
variation:

Expected (target) percent on rank-ordered stage defined by beta distribution
parameters.

First rank-ordered stage standard deviation.

CV slope for rank-ordered profiles.
•
In addition, we have a fourth variable – number of stages.
•
How did we come up with these metrics?
DSLee 860-441-0745
16
Rank-ordered PSD profiles actually represent families of PSD profiles . . .
The 95th percentile of Rdavg. by the FDA algorithm for the set of
reference and test profiles with the mean PSD profile below is 3.93.
37.69
37.69
13.43
40
13.43
50
Reference
Test
21.66
21.66
4.26
4.26
Mean
30
3.18
3.18
3.11
3.11
3.95
3.95
0.92
0.86
0.86
0.39
0.39
0.92
1.51
1.51
0.68
0.68
3.58
3.58
1.24
1.24
10.49
10.49
2.39
2.39
8.61
8.61
1.83
1.83
6.08
6.08
1.46
2.38
2.38
0.49
0.49
10
1.46
20
0
0
Std. Dev.
•
5
10
15
20
Throat
Pre_sep
P(-1)
P0
P1
P2
P3
P4
P5
Filter
Case
DSLee 860-441-0745
17
Rank-ordered PSD profiles actually represent families of PSD profiles . . .
This set of mean reference and test profiles yields a value of 3.93
for the 95th percentile of Rdavg. (using same randomization seed as
the last example).
50
37.69
37.69
13.43
Test
40
13.43
Reference
21.66
21.66
4.26
4.26
Mean
30
10.49
3.95
0.92
10.49
3.95
0.92
2.39
3.58
1.24
2.39
3.58
1.24
2.38
0.86
0.39
2.38
0.86
0.39
0.49
1.51
0.49
1.51
0.68
3.18
0.68
3.18
3.11
6.08
3.11
6.08
1.46
8.61
1.46
8.61
1.83
10
1.83
20
0
0
Std. Dev.
•
5
10
15
20
Throat
Pre_sep
P(-1)
P0
P1
P2
P3
P4
P5
Filter
Case
DSLee 860-441-0745
18
Rank-ordered PSD profiles actually represent families of PSD profiles . . .
This set of reference and test mean profiles also yields a value of 3.93
for the 95th percentile of Rdavg. (using same randomization seed as the
last two examples).
40
Reference
37.69
37.69
50
Test
21.66
21.66
4.26
4.26
Mean
30
20
Throat
Pre_sep
P(-1)
P0
P1
2.38
2.38
0.49
3.18
0.49
3.18
3.11
3.95
3.11
3.95
0.92
8.61
0.92
8.61
13.43
13.43
15
1.83
10.49
2.39
10
1.83
10.49
2.39
6.08
1.46
3.58
6.08
3.58
1.24
1.51
1.24
1.51
0.68
0.86
0.68
0.86
5
0.39
0
0
0.39
10
1.46
20
Std. Dev.
•
P2
P3
P4
P5
Filter
Case
DSLee 860-441-0745
19
Std. Dev.
10
5
15
20
Ordered
Rank 1
Ordered
Rank 2
Ordered
Rank 3
Ordered
Rank 4
Ordered
Rank 5
Ordered
Rank 6
Ordered
Rank 7
0
0
Ordered
Rank 8
Ordered
Rank 9
Ordered
Rank 10
0.86
P3
0.39
0.86
P2
0.39
1.51
P1
0.68
1.51
P0
0.68
2.38
P(-1)
P4
30
20
P5
15
20
10
Filter
Case
Std. Dev.
6.08
6.08
Throat
Pre_sep
P(-1)
P0
10
15
20
P1
P2
P3
P4
P5
DSLee 860-441-0745
3.95
Filter
2.38
2.38
0.49
0.49
3.18
3.11
3.95
3.18
0.92
3.11
0.92
8.61
21.66
8.61
1.83
4.26
1.83
21.66
4.26
37.69
20
37.69
10.49
30
13.43
Mean
37.69
37.69
40
13.43
10.49
6.08
2.39
6.08
2.39
1.46
3.58
10
1.46
3.58
1.51
1.24
1.51
1.24
0.68
0.86
5
0.39
0
0
0.68
20
0.86
10.49
10.49
50
0.39
3.95
3.95
3.58
3.58
2.38
2.38
0.86
0.86
1.51
1.51
3.18
3.18
21.66
21.66
B
13.43
13.43
2.39
2.39
0.92
0.92
1.24
1.24
0.49
0.49
0.39
0.39
0.68
0.68
3.11
3.11
8.61
8.61
Mean
30
1.46
1.46
1.83
1.83
4.26
21.66
21.66
40
0.49
Pre_sep
2.38
Throat
0.49
Case
5
4.26
3.18
3.18
3.95
3.95
0.86
10
3.18
0
3.18
10
Std. Dev.
3.11
3.11
0.92
0.86
1.51
20
3.11
0
0
0.39
0.92
3.58
3.58
1.51
10.49
10.49
8.61
8.61
6.08
50
3.11
3.58
3.58
3.95
3.95
0
0.39
0.68
0.68
1.24
1.24
2.39
2.39
4.26
4.26
1.83
6.08
37.69
37.69
A
1.24
1.24
0.92
0.92
Filter
6.08
P5
1.46
P4
6.08
8.61
1.46
1.83
2.38
30
1.46
10
1.83
P3
1.83
P2
8.61
P1
10.49
P0
2.39
10.49
P(-1)
2.39
Pre_sep
1.46
0.49
2.38
10
21.66
21.66
Throat
37.69
0.49
Mean
•
4.26
4.26
40
13.43
5
13.43
20
13.43
15
37.69
Std. Dev.
40
13.43
Mean
Rank-ordered PSD profiles actually represent families of PSD profiles . . .
The same 95th percentile of Rdavg.is obtained from the three
example sets because they share the exact same rank ordered
profile (as well as rank order specific stage variances).
50
C
Case
50
The 95th percentile of the c2 ratio
statistic equals 3.93 for each example
profile as well as the rank ordered
profile.
(same simulation seed for each determination)
Ordered
Rank 11
20
Rank-ordered PSD profiles actually represent families of PSD profiles . . .
•
This simplifies the
definition of profile “shape”
when simulating data sets
of PSD profiles.
We can cover the “PSD
profile “water front” using
easily defined distribution
parameters.
(a, b) = (1, 4)
Percent on Stage
•
(a, b) = (1, 2)
(a, b) = (1, 1)
Rank-Ordered Stage
DSLee 860-441-0745
21
Underlying patterns of profile variation . . .
We now want to also systematically define and deal with stage
to stage differences in standard deviations for rank-ordered
profiles.
37.69
37.69
13.43
40
13.43
50
21.66
21.66
4.26
4.26
Mean
30
10
15
20
Ordered
Rank 1
Ordered
Rank 2
Ordered
Rank 3
Ordered
Rank 4
Ordered
Rank 5
Ordered
Rank 6
Ordered
Rank 7
Ordered
Rank 8
0.86
0.86
0.39
1.51
0.68
Ordered
Rank 10
0.39
1.51
2.38
Ordered
Rank 9
0.68
2.38
5
0.49
3.18
3.11
0
0
0.49
3.18
3.58
3.11
3.58
1.24
3.95
0.92
1.24
3.95
6.08
0.92
6.08
1.46
8.61
1.83
1.46
8.61
10.49
2.39
10
1.83
10.49
2.39
20
Std. Dev.
•
Ordered
Rank 11
DSLee 860-441-0745
22
Underlying patterns of profile variation . . .
•
•
Examination of the IPAC-RS PSD database revealed that generally, the
coefficient of variation (e.g. relative standard deviation) is either relatively
constant across the stages or increases more or less linearly with decreasing
rank-order of the stages (e.g. declining mass recovered on stage).
As a consequence, the stage specific standard deviation is generally
proportional to the percent retained on stage and decreases with declining
percent on stage at a rate equal to or less than the rate at which the mean
percent on stage decreases.
DSLee 860-441-0745
23
Underlying patterns of profile variation . . .
Corresponding ranges for first rank ordered stage standard
deviation and CV slope used to calibrate simulation data sets:
Standard Deviation of First Ranked Stage vs. CV slope Conditioned by
Levels of the Standard Deviation of the Last Ranked Stage.
0
0.3
0.6
0.9
1.2
1.5
0
2
4
6
8
10
SD last rank
ordered stage
15
CV slope
•
0
SD first rank ordered stage
DSLee 860-441-0745
24
Experimental Design approach to examining the statistical
performance of the FDA algorithm for the equal formulation case
•
Response surface
design – 500 sets of
Reference and Test
profiles per design
point.
15
10.0
7.5
Standard Deviation of
First Rank-Ordered Stage
5.5
Stage to Stage
increase in RSD (%)
(Percent Increase In RSD from one
Rank-Ordered Stage to the Next)
0.0
1.0
Uniform
Linear
"Skewed"
•
Design repeated across
four levels of stage: 4, 7,
10, 13
Rank-Ordered Profile "Shape"
DSLee 860-441-0745
25
Experimental Design approach to examining the statistical
performance of the FDA algorithm for the equal formulation case
•
Patterns of rank-ordered profiles/variation at extremes of the
design space.
DSLee 860-441-0745
26
Experimental Design approach to examining the statistical
performance of the FDA algorithm for the equal formulation case
•
•
•
•
For each 500 pairs of test and reference profile sets, the distribution of
all possible (13,050) values of rd were calculated and the 95th
percentiles rd’s were calculated.
The median 95th percentile rd was then tabulated for each design point.
The point estimate of the 95th percentile of the distribution of rd was
contrasted with the critical value from an F-distribution at a = 0.05 with
the appropriate dfs.
The ratio of two independent central chi-squares will be distributed as
a central F with (S-1, S-1) df’s.
DSLee 860-441-0745
27
Experimental Design approach to examining the statistical
performance of the FDA algorithm for the equal formulation case
•
If the value of the difference (median rd95 – Fcrit) is less than zero, then
the distribution of rd from which Rdavg is estimated in the FDA
algorithm has a shorter tail than expected if rd is distributed as an F. If
the difference is greater than zero, then the distribution of rd has a
longer tail than otherwise expected.
Expected F(S-1, S-1) distribution
"Long-tailed" rd distribution
"Short-tailed" rd distribution
1.0
density
0.8
"short-tailed" rd95
Expected F(S-1, S-1)95
0.6
"long-tailed" rd95
0.4
0.2
0.0
0
1
2
3
4
5
2
3
4
DSLee 860-441-0745
5
28
Experimental Design approach to examining the statistical
performance of the FDA algorithm for the equal formulation case
•
•
The contrast value was modeled over the design space.
To demonstrate stability of rd in the equal formulation case, the
contrast value should not be sensitive to location in the design space.
DSLee 860-441-0745
29
Results
DSLee 860-441-0745
30
Results
DSLee 860-441-0745
31
Results
DSLee 860-441-0745
32
Results
DSLee 860-441-0745
33
Results
2
1
0
-1
-2
6
Deviation from 4
2
F(0.05, s-1, s-1)
0
Rank-Ordered Profile "Shape"
-2
"Skewed"
10
5.5
Initial SD
>0
1
10
5.5
<0
15
0
1
7.5
0
15
2
1
0
1
-2
6
-2
Linear
10
5.5
1
15
0
Uniform
10
5.5
<0
1
7.5
0
1
7.5
0
RSD Ramp
4
15
>0
<0
0
7.5
15
<0
1
7.5
0
RSD Ramp
7
15
10
5.5
1
>0
<0
0
7.5
15
2
1
0
1
-2
<0
1
0
7.5
15
2
1
0
-1
-2
10
5.5
>0
1
15
2
1
0
-1
-2
-2
10
5.5
10
5.5
<0
7.5
6
2
1
0
-1
-2
2
1
0
-1
-2
10
5.5
<0
Deviation from 4
2
F(0.05, s-1, s-1)
0
Initial SD
<0
>0
7.5
Deviation from 4
2
F(0.05, s-1, s-1)
0
Initial SD
2
1
0
-1
-2
10
5.5
<0
1
7.5
0
15
2
1
0
-1
-2
10
5.5
<0
1
0
7.5
RSD Ramp
10
15
10
5.5
<0
1
7.5
0
15
RSD Ramp
13
Number of Stages in Profile
DSLee 860-441-0745
34
Discussion
•
These are not particularly surprising results . . .

The assumption of independence for dTRR’ and dRR’ is violated because
the same reference profiles appear in both the numerator and
denominator of each individual rd.
Consequently, the distribution of rd can only be only F-like.

Setting the question of lack of independence aside, dTRR’ and dRR’ are
only asymptotically distributed as c2’s.

The distribution of rd will only approximate an F-like distribution when the
ratio for the smallest observed average percent on stage, divided by the
number of stages in the profile, exceeds a value of one and/or the
number of stages increases.
Consequently, the distribution of rd is particularly sensitive to rankordered profile shape and number of stages.
DSLee 860-441-0745
35
Discussion
•
These simulation results point out several difficulties with
respect to the desire for a “one-size-fits-all” test . . .



It suggests that equivalence test values will need to be defined on a
product by product basis.
The “recipe” for defining equivalence test values without resorting to
inclusion of properties of the samples themselves, may not be tractable.
Depending upon the rank-ordered profile shape and number of stages,
pooling of stages might be required.
DSLee 860-441-0745
36
Discussion
•
Other technical considerations . . .

The mean Rd is not always defined . . . and is not necessarily the 95th percent
upper confidence bound for Rdavg.

X Ri  X Ri 
d RR  
i 0.5 * X Ri  X Ri 
S
2
DSLee 860-441-0745
37
Discussion
•
Other technical
considerations . . .


The FDA algorithm doesn’t treat
the creation of triplets in a
balanced fashion – it admits
triplets for which both reference
profiles may come from the
same lot.
This is easily remedied by the
sampling approach of Cheng
and Shao where lot-triplets are
first defined. However, this does
not address the fundamental
distributional issues.
T
R R'
rd1
rd2
rd3
rd4
rd5
rd6
rd7
rd8
rd500
DSLee 860-441-0745
38
Discussion
To get over the hump, it will be critical to clearly define
what is meant by equivalence of profiles.
as
e
C
Fi
lte
r
7
St
g.
6
St
g.
5
St
g.
4
St
g.
3
St
g.
2
St
g.
1
St
g.
0
55
50
45
40
35
30
25
20
15
10
5
0
St
g.
•
The working group is continuing work on the problem.
Th
ro
at
•
DSLee 860-441-0745
39
Acknowledgements
Pfizer:
Mark Berry
Jeff Blumenstein
Yukun Ren
Greg Steeno
PQRI PSD Profile Working Group:
David Christopher (chair)
Svetlana Lyapustina
Wallace Adams, Craig Bertha,
Peter Byron, Bill Doub, Craig Dunbar,
Walter Hauck, Jolyon Mitchell,
Beth Morgan, Steve Nichols, Yinou Pang,
Guirag Poochikian, Gur Singh,
Terry Tougas, Yi Tsong, Ron Wolff, Bruce
Wyka,
DPTC Liaisons: Jeff Blumenstein
Michael Golden
DSLee 860-441-0745
40
Back up slides . . .
DSLee 860-441-0745
41
Underlying patterns of profile variation . . .
•
Blinded Profiles 3 – 14 (from top left to bottom right by line)
DSLee 860-441-0745
42
Underlying patterns of profile variation . . .
•
Blinded Profiles 15 – 26 (from top left to bottom right by line). Profile pattern 16 CV
for last stage is off-scale (~480) and the overall pattern of CV change with
descending rank-order of stages is better characterized as exponential.
DSLee 860-441-0745
43
Spaghetti plots of IPAC-RS profiles
Blinded Profiles 1 – 4 (from top left to bottom right by line)
DSLee 860-441-0745
44
Spaghetti plots of IPAC-RS profiles
Blinded Profiles 5 – 8 (from top left to bottom right by line)
DSLee 860-441-0745
45
Spaghetti plots of IPAC-RS profiles
Blinded Profiles 9 – 12 (from top left to bottom right by line)
DSLee 860-441-0745
46
Spaghetti plots of IPAC-RS profiles
Blinded Profiles 13 – 16 (from top left to bottom right by line)
DSLee 860-441-0745
47
Spaghetti plots of IPAC-RS profiles
Blinded Profiles 17 – 20 (from top left to bottom right by line)
DSLee 860-441-0745
48
Spaghetti plots of IPAC-RS profiles
Blinded Profiles 21 – 24 (from top left to bottom right by line)
DSLee 860-441-0745
49
Spaghetti plots of IPAC-RS profiles
Blinded Profiles 25 – 27 (from top left to bottom right by line)
DSLee 860-441-0745
50