Transcript A Knowledge-based Theory of Governance Choice: A Problem

FDA and Pharmaceutical Manufacturing
Research Projects
Jeffrey T. Macher
Jackson A. Nickerson
Co-Principal Investigators
Presentation Overview
 Executive summary
 Project goals
 Data collection and synthesis
 Analysis methodology
 Findings
 Development opportunities and constraints
Executive Summary
 We develop statistical models that predict the:
 Probability of a facility being chosen for inspection.
 Effect of investigator training, experience, and individual
effects on the probability of investigational outcomes.
 Characteristics and identities of facilities that correlate with
the probability of non-compliance.
 We present initial results for each of these analyses.
 We identify additional opportunities and next steps to
create value along with some constraints.
FDA Research Project Goals
 Risk-based assessment of FDA cGMP outcomes.
 Identify underlying ability of investigators and their training.
 Identify underlying compliance of each facility.
 Identify attributes (currently recorded by the FDA)
that impact inspection outcomes.
 Transfer “learning” to FDA.
Progress to Date
 Just as new drugs go through
 Discovery
 Development and
 Commercialization….
 Our model and this presentation concludes the
discovery phase of our project.
 Please think of our model as a “platform” that can be
developed to assess a variety of compliance issues.
FDA Project Approach
 Compile and link FDA databases.
 Estimate the likelihood of various outcomes:
 NAI, VAI, OAI; Warning Letters; Field Alerts; Product
Recalls.
 based on…
 compound/product, facility, firm, FDA district, investigator
and training derived factors.
 in order to …
 evaluate the allocation of investigational resources.
 inform effectiveness of investigator training and
management.
FDA Databases
 DQRS (Field alerts)
 EES
 FACTS (Inspections) – CDER only
 Product Listing
 Product Recalls
 Product Shortages
 Facility Registration (DRLS)
 ORA Training database
 Warning letter database
Data Preparation
 Started with FACTS (1990-2003).
 Manufacturing facilities only.
 Assembled investigator training database:
 Identified corporate ownership by plant by year and
firms operating at a specific facility each year.
 Constructed facility-year data
 Added observations for years NOT inspected.
 Corrected FEI/CFN mismatches.
 Constructed numerous other variables.
Some basic “facts” about the FDA data
 Years covered:
 Total number of facilities inspected:
 Total number of “Pac codes”:
 Total number of “Inspections”:
 Total number of investigators:
FY 1990-2003
3753
38,341
14,162
783
Empirical Methodology
 Inspection
 Probability of choosing a facility to inspect.
 Detection
 Probability of a non-compliance inspection outcome.
 Noncompliance
 Probability of noncompliance, inspection, and detection.
 Detection control estimation.
Inspection
 Groups of variables:
 Technology variables
•
•
•
•
•
Rx
Gel Cap
Liquid
Parenteral
Bulk
 Industry variables
Prompt Release
Soft Gel Cap
Powder
Lg. Vol. Parent.
Sterile
• Vitamins (IC 54)
• Antibiotics (IC 56)
Ext or Delayed Rel
Ointment
Gas
Aerosol
Suppositories
Necessities (IC 55)
Biologics (IC 57)
 Inspection decision variables
• Ln(Days between inspections)
• Surveillance = reason for inspection (0 = Compliance)
• Last inspection outcome (1 = OAI, 0 = NAI, VAI)
 Years 1992-2003 (binary variables for each year)
Inspection: Explained Variance
Probit analysis of decision to inspect.
Technology variables
Industry variables
Inspection Decision variables
Year dummy variables
D R2
12%
9
20
~0
Cumulative R2
12%
21
51
51
Omitted categories: Human Drugs (IC 60-66), select technologies, Year dummies 1990-91.
Foreign inspection included in analysis but uniquely identifies many inspections and is dropped from the analysis.
Technology Variables:
Change in Probability of Inspection
Rx
0.13 **
Gas
-0.68 **
Promp Rel.
-0.19 **
Parenteral
-0.32 **
Ext/del Rel.
-0.19 **
Lg Vol Parent.
-0.08 +
Gel Cap
-0.25 **
Aerosol
-0.26 **
Soft Gel Cap
-0.36 **
Bulk
-0.37 **
Ointment
-0.32 **
Sterile
-0.07 **
Liquid
-0.30 **
Suppositories
-0.23 **
Powder
-0.37 **
**
*
+
99% confidence interval
95% confidence interval
90% confidence interval
Omitted categories: Not Classified, Bacterial antigens, Bacterial
vaccines, Modified bacterial vaccines, Blood serum, Immune serum.
Industry and Inspection Variables:
Change in Probability of Inspection
Industry Variables
Inspection Variables
Antibiotics (IC 56)
0.19 **
Ln(Days btwn Insp)
-0.28 **
Vitamins (IC 54)
0.11 **
Surveillance
-0.84 **
Necessities (IC 55)
-0.06 **
Last outcome
0.13 **
Biologics (IC 57)
-0.07 **
Omitted category: Human drugs
**
*
+
99% confidence interval
95% confidence interval
90% confidence interval
Days Between Inspections
1
0.9
0.8
Probability
of
Inspection
0.7
0.6
Probability of
0.5
Inspection
0.4
0.3
0.2
0.1
0
0
1
2
3
4
Years
SinceLast
Last Inspection
Years
Since
Inspection
5
6
Detection
 Groups of variables
 Technology
 Industry
 Training
• Total training days prior to inspection (other than 5 main drug courses)
• Drug course 1: Basic drug school
• Drug course 2: Advanced drug school
• Drug course 3: Pre-approval inspections
• Drug course 4: Active Pharmaceutical Ingrediant Mfg.
• Drug course 5: Industrial sterilization
 Investigator Experience
• Number of inspections in the prior 12 months
• Number of inspections in the prior 12-24 months
 ORA District Office
 Investigator Classification
• A consolidation of position classifications
Detection: Explained Variance
Probit analysis of decision to inspect.
Technology variables
Industry variables
Training and Experience vars.
Office and Position variables
Investigator effect
DR2
0.9 %
0.3
0.3
1.4
4.2
Cumulative R2
0.9 %
1.2
1.5
2.9
7.1
Training and Experience Variables:
Change in Probability of Detection1
Total training days prior to inspection (less 1-5)
Drug course 1: Basic drug school
-2.2E-03
0.07 *
Drug course 2: Advanced drug school
-0.05
Drug course 3: Pre-approval inspections
-0.23 **
Drug course 4: Activ. Ingred. Mfg.
-0.15 *
Drug course 5: Industrial sterilization
No. of inspections in the prior 12 months
No. of inspections in the prior 12-24 months
1Without
investigator fixed effects.
0.08 *
4.8E-03 +
-1.4E-03
ORA Office and Classification Variables:
Change in Probability of Detection2
ORA Office Variables
Position Variables
ORA LOS
0.07 +
Compliance
ORA KAN
-0.06 +
ORA NYK
-0.07 *
ORA SJN
-0.09 **
ORA SRL
0.04
Microbiologist
-0.02
Investigator
-0.04
-0.10 *
Chemist
-0.05
ORA ATL
-0.10 **
Eng/Sci
-0.07
ORA DAL
-0.10 **
ORA SAN
-0.11 **
Dist/Reg. Admin.
-0.10 +
ORA DET
-0.13 **
FDA Bureau
-0.15 *
ORA NWE
-0.15 **
Technician
-0.18
All other ORA off. insignificant.
2With
investigator fixed effects.
2
3
Distribution of Investigator Abilities
0
1
425 Investigators
.6
.4
.2
0
-.2
Probability of Plant Noncompliance by Investigator (with >50 inspections)
Non-compliance
 Detection Control Estimation
 Relatively new procedure used in academic literature.
 Used for assessing tax evasion, EPA compliance, and other
applications.
 FDA application more complicated than other applications.
 Assume three actors:
 Facility decides level of compliance.
 Inspection decision-maker chooses when to inspect.
 Investigator chooses detection or not.
 Estimate all three processes simultaneously.
Non-compliance model
 Assume inspection decisions are non-random.
 Assumption is different from other applications.
 Construct a likelihood function that models the
probabilities of:
 a plant being selected for inspection and
 the outcome of the inspection.
Constructing a Likelihood Function
The likelihood that
facility i is not
inspected
The likelihood that
facility i is noncompliant
L1i = 1
L2i = 0
The likelihood that
facility i is compliant
L1i = 0
The likelihood that
facility i is inspected
L3i = 1
L2i = 1
L3i = 0
The likelihood that
facility i is found
non-compliant
The likelihood that
facility i is found
compliant
Likelihood Function
 Three probabilities are combined to form the function:
 Probability that a non-compliant facility is inspected and
detected:
L1i=1, L2i=1, L3i=1
 Probability of inspecting and not detecting noncompliance:
• probability that the facility is compliant:
L1i=0, L2i=1
• probability that noncompliance goes undetected:
L1i=1, L2i=1, L3i=0
 Probability that a facility is not inspected in a given year:
L2i=0
Estimating the Likelihood Function
 Select covariates associated with non-compliance,
selection, and detection.
 Non-compliance: facility-related characteristics.
 Selection: factors currently used in selecting facilities.
 Detection: investigator-related factors.
 Use a maximum likelihood estimation to find
coefficient estimates that maximize the function.
 Initialize parameter estimates with results from inspection
and detection analyses.
Change in Probability of Non-compliance
Rx
-0.10
*
-0.09
-0.05
-0.04
0.08
-0.13
-0.13
Prompt rel.
0.07
Ext/Del rel.
0.17
+
0.21
+
0.13
0.14
Gel cap
0.20
+
0.19
+
0.05
0.06
Soft gel cap
-7.E-05
0.02
-0.04
-0.04
Ointment
0.11
0.08
-0.18
-0.15
Liquid
0.21
-0.04
-0.03
Powder
Gas
Parenteral
Lg. vol Parent.
Aerosol
Bulk
*
0.22
+
4.E-03
-0.01
-0.26
-0.22
-0.24
0.15
0.41
0.36
0.14
0.14
-0.04
-0.01
-0.25
-0.26
-0.27
0.08
0.11
-0.07
-0.24
+
0.08
-0.18
**
-0.15
+
-0.24
+
-0.27
Sterile
0.09
0.09
0.03
0.01
Suppositories
0.12
0.12
-0.26
-0.27
81570
55371
22456
17499
Number of obs.
+
Vitamins
0.07
0.17
Necessary
0.13
0.12
Antibiotics
0.23
Biologics
-0.05
0.06
2.E-03
-3.E-03
No. Products/Plant
-2.E-03
-1.E-03
No. Dose forms/Plant
-4.E-03
-0.01
No. D.F. Routes/Plant
-3.E-04
0.00
No. Thera. Classes/Plant
No. Sponsor Appl./Plant
0.02
Detection
*
0.22
0.02
Ownership change (t=0)
0.16
Ownership change (t=1)
-0.13
Ownership change (t=2)
-0.09
Ownership change (t=3)
0.34
Firms per plant
Inspection
**
-0.07
Technology
Yes
Yes
Yes
Yes
Plant Select
No
Yes
Yes
No
Training
Yes
Yes
Yes
Yes
No. of obs
81570
55371
22456
17499
*
**
+
Predicted Level of Facility Non-compliance
For 50 Most Inspected Facilities
28
41
45
25 35
4 33 38
13 42
14 43 30 37
10 39 50
49 46
22
17 31 12 40
1 34 26 47 8 36 21 32 18 23 2 44 3
5 19 16 9 29 20 15 7 27
28 41 45 25 35 4 33 38 13 42 14 43 30 37 10 39 50 49 46 22 17 31 12 40
1 34 26
Statistically more noncompliant than the mean facility.
Statistically not different from the mean facility.
Statistically more compliant than the mean facility.
47 8 36 21
32 18 23
2 44 3
5
19
16
9 29
20 15
7
27
Immediate Implications
 Inspection and Non-compliance
 New suggestions for inspection choices.
• Use non-compliance analysis to assess risk of any given
facility, firm, or technology.
– Increase focus on particular facilities and attributes.
– Ownership changes.
 Mixed strategy inspection plan.
 Detection
 Use detection analysis to assess quality of investigators and
their training.
 Focus investigator activities to build and maintain short-run
experience.
Broader Implications
 Our statistical methods provide a test-bed for asking and
answering management and oversight questions.
 Further development is needed.
 DCE has potentially broad applicability to CDER and other
centers at the FDA including CBER, food, etc..
 What facilities are most at risk of non-compliance?
• Base-line non-compliance
• Technology
• Ownership changes, etc.
 What manufacturers are more/less prone to non-compliance.
 DCE has implications for the type, format, and processing of
data to be collected and analyzed.
Development Opportunities
 Additional variables can and are being constructed to
examine additional issues.
 Recall, shortages, supplement filings.
 More fine-grain information on technology, manufacturing
knowledge, organizational capabilities.
 Evaluate manufacturer data collected in our study.
 More heavily weight more recent investigations.
 Expand to full set of investigators and facilities
(requires additional computational resources).
 Evaluate endogeneity concerns.
Development Constraints
 Software/computer limitation.
 Data preparation/man-power.
 Funding resources are nearly exhausted.
 Teaching.
Current Plan
 Document current progress in a white paper.
 Further develop data in hand (EES, Shortages, etc.).
 We received cooperation from the gold sheets.
 Work with you to develop plan for transferring results
to FDA.
 Look for additional funding sources.