Transcript presentation_Haijun5-15-2009-19-4-50

Bayesian Hierarchical Models
for Detecting Safety Signals
in Clinical Trials
H. Amy Xia and Haijun Ma
Amgen, Inc.
MBSW 2009, Muncie, IN
March 20, 2009
Disclaimer: The views expressed in this presentation
represent personal views and do not necessarily represent
the views or practices of Amgen.
Outline
• Introduction
• A motivating example
• Bayesian Hierarchical Models
– Meta analysis of Adverse Events data from multiple
studies incorporating MedDRA structure
– Incorporate patient level data
– Effective graphics
• Closing Remarks
Three-Tier System for Analyzing
Adverse Events in Clinical Trials
• Tier 1: Pre-specified Detailed Analysis and
Hypothesis Testing
– Tier 1 AEs are events for which a hypothesis has
been defined
• Tier 2: Signal Detection among Common Events
– Tier 2 AEs are those that are not pre-specified and
“common”
• Tier 3: Descriptive Analysis of Infrequent AEs
– Tier 3 AEs are those that are not pre-specified and
infrequent
Gould 2002 & Mehrotra 2004
SPERT White Paper 2008
Multiplicity Issue in Detecting
Signals Is Challenging
• Detection of safety signals from routinely
collected, not pre-specified AE data in clinical
trials is a critical task in drug development
• Multiplicity issue in such a setting is a
challenging statistical problem
– Without multiplicity considerations, there is a potential
for an excess of false positive signals
– Traditional ways of adjusting for multiplicity such as
Bonferroni may lead to an excessive rate of false
negatives
– The challenge is to develop a procedure for flagging
safety signals which provides a proper balance
between ‘no adjustment’ versus ‘too much
adjustment’
Considerations Regarding Whether
Flagging an Event
•
•
•
•
Actual significance levels
Total number of types of AEs
Rates for those AEs not considered for flagging
Biologic relationships among various AEs
1st two are standard considerations in the
frequentist approach. The 2nd two are not,
but relevant in the Bayesian approach
-- Berry and Berry, 2004
Bayesian Work in Signal Detection
• Spontaneous adverse drug reaction
reports
– Gamma Poisson Shrinker (GPS) on FDA
AERS database (DuMouchel,1999)
– Bayesian Confidence Propagation Neural
Network (BCPNN) on WHO database (Bate,
et al. 1998)
• Clinical trial safety (AE) data
– Bayesian hierarchical mixture modeling (Berry
and Berry, 2004)
Meta Analysis
• Glass (1976)
Meta-analysis refers to a statistical analysis that
combines the results of some collection of
related studies to arrive a single conclusion to
the question at hand
• Meta-analysis based on
– aggregate patient data (APD meta-analysis)
– Individual patient data (IPD) meta-analysis
• Bayesian modeling is a natural choice to
incorporate the complex hierarchical structure of
the data
George Chi, H.M. James Hung,
Robert O’Neill (FDA CDER)
“Safety assessment is one area where
frequentist strategies have been less
applicable. Perhaps Bayesian approaches
in this area have more promise.”
(Pharmaceutical Report, 2002)
An Example
• Data from four double-blind placebo-controlled studies
on drug X. Study populations are similar.
• Sample sizes:
Study
Drug X
N
Drug X
Subj-yr
Placebo
N
Placebo
Subj-yr
Study A
57
28.25
55
19.02
Study B
486
104.75
166
34.93
Study C
390
85.44
193
40.97
Study D
312
68.78
306
65.91
• After converting all AEs into same MedDRA version,
reported AEs are coded to 464 PTs under 23 SOCs
and 233 HLTs
SYSTEM ORGAN CLASS
INFECTIONS AND INFESTATIONS
HIGH LEVEL TERM
HERPES VIRAL
INFECTIONS
INFECTIONS AND INFESTATIONS
PREFERRED TERM STUDY n_0 n_1 N_0
N_1
rt_0
rt_1
HERPES SIMPLEX
1
2
3
4
0
0
1
2
3
5
8
2
55
166
193
306
57
486
390
312
0.00
0.00
0.52
0.65
5.26
1.03
2.05
0.64
UPPER RESPIRATORY
TRACT INFECTIONS
SINUSITIS
1
2
3
4
2
6
0
4
8
19
8
11
55
166
193
306
57
486
390
312
3.64
3.61
0.00
1.31
14.04
3.91
2.05
3.53
INFECTIONS AND INFESTATIONS
UPPER RESPIRATORY
TRACT INFECTIONS
RHINITIS
1
2
3
4
0
0
1
0
0
0
4
0
55
166
193
306
57
486
390
312
0.00
0.00
0.52
0.00
0.00
0.00
1.03
0.00
SKIN AND SUBCUTANEOUS TISSUE
DISORDERS
URTICARIAS
URTICARIA
1
2
3
4
0
1
0
1
1
1
4
4
55
166
193
306
57
486
390
312
0.00
0.60
0.00
0.33
1.75
0.21
1.03
1.28
SKIN AND SUBCUTANEOUS TISSUE
DISORDERS
PURPURA AND RELATED
CONDITIONS
ECCHYMOSIS
1
2
3
4
0
0
0
0
0
4
5
3
55
166
193
306
57
486
390
312
0.00
0.00
0.00
0.00
0.00
0.82
1.28
0.96
INJURY, POISONING AND
PROCEDURAL COMPLICATIONS
NON-SITE SPECIFIC
INJURIES NEC
EXCORIATION
1
2
3
4
0
0
0
0
0
7
0
1
55
166
193
306
57
486
390
312
0.00
0.00
0.00
0.00
0.00
1.44
0.00
0.32
INJURY, POISONING AND
PROCEDURAL COMPLICATIONS
NON-SITE SPECIFIC
PROCEDURAL
COMPLICATIONS
PROCEDURAL PAIN
1
2
3
4
0
0
0
2
0
0
1
1
55
166
193
306
57
486
390
312
0.00
0.00
0.00
0.65
0.00
0.00
0.26
0.32
N_0: sample size in placebo arm; N_1: sample size in treatment arm
n_0: # subject with AE in placebo arm; n_1: # subject with AE in treatment arm
rt_0: subject incidence in placebo arm; rt_1: subject incidence in treatment arm
Proposed Bayesian Approach
• Hierarchical mixture models for aggregated
binary responses was constructed based on the
work by Berry & Berry(2004)
– Explore impact of using different MedDRA hierarchy
– Inclusion of study effects
– Further extended to a hierarchical Poisson mixture
model, to account for different exposure/follow-up
times between patients
• Individual patient level models are discussed
• Implemented the above models with available
software
– WinBUGS for model implementation
– S-Plus graphics for inference
MedDRA
• MedDRA (the Medical Dictionary for Regulatory
Activities Terminology) is a controlled vocabulary
widely used as a medical coding scheme.
• MedDRA Definition (MSSO):
– MedDRA is a clinically-validated international medical
terminology used by regulatory authorities and the
regulated biopharmaceutical industry. The
terminology is used through the entire regulatory
process, from pre-marketing to post-marketing, and
for data entry, retrieval, evaluation, and presentation.
https://eudract.emea.europa.eu/docs/medDRA/Introduction_To_MedDRA.ppt
MSSO: Introduction to MedDRA
http://www.fda.gov/cder/present/dia-699/dia628/index.htm
MedDRA and Pharmacovigilance - The Way Forward, 7/8/99
http://www.fda.gov/cder/present/dia-699/dia628/index.htm
MedDRA and Pharmacovigilance - The Way Forward, 7/8/99
Example of MedDRA Hierarchy
SOC = Respiratory, thoracic and
mediastinal disorders
HLGT = Respiratory tract
infections
SOC = Infections and
infestations
HLGT = Viral infectious
disorders
HLT =Viral upper respiratory
tract infections
HLT = Influenza viral
infections
PT = Influenza
https://eudract.emea.europa.eu/docs/medDRA/Introduction_To_MedDRA.ppt
MSSO: Introduction to MedDRA
Hierarchical Structure of MedDRA
• Bayesian hierarchical models allow for explicitly
modeling AEs with the existing coding structure
– AEs in the same SOC more likely to be similar within than across
SOCs.
– Allow for this possibility, but does not impose it, depending on
the actual data
– SOC tends to be too broad. HLT is more closely related to
medical concepts.
• In fact, clinical and safety people would (informally)
consider the similarity of the AEs, say, within SOCs when
they review AE tables
– For example, if differences in several CV events were observed,
then each would be more likely to be causal than if differences
came from medically unrelated areas (eg, skin, neurological,
thrombosis, cancer)
• Bayesian hierarchical modeling allows a scientific,
explicit, and more formal way to take it into consideration
Notations
• Study i=1,…I, SOC b=1,…B and PT
j=1,…kb
• Data: For AEibj
– Treatment group: Yibj incident events observed
in Nit patients with Tibj subjects’ exposure
– Control group: Xibj incident events observed in
Nic patients with Cibj subjects’ exposure
Bayesian Meta Analysis
Hierarchical Logistic Regression
Yibj ~ Binomial(Nit , tibj ), Xibj ~ Binomial(Nic , cibj ),
where tibj and cibj are eventratesfor AEibj in thetreatmentand
controlgroups, respectively.
•Common treatment effect for same PT across
studies logit (c )   , logit (t )    
ibj
ibj
ibj
ibj
bj
ibj ~ N( bj ,  2,bj ) ,  bj ~ N( b ,  2b )
MixtureP riorfor t reatm
ent effectlog - OR :
Stage 1 prior: bj~  b (0)  (1   b ) N (  b ,  2b );
Stage 2 prior:  b ~ Beta( p ,  p );
 p ~ exp( ) I ( p  1);
Stage 3 prior:
 p ~ exp(  ) I (  p  1)
Bayesian Meta Analysis
Hierarchical Logistic Regression (Cont.)
• Treatment effect with additive study effects:
logit(cibj )  ibj , logit(tibj )  ibj   bj   i Studyi
ibj ~ N( bj ,  2,bj ) ,  bj ~ N( b ,  2b )
 bj~  b (0)  (1   b ) N (  b ,  2b )
 i ~ N (0,0.01)
• A random treatment effect/multiplicative model:
logit(cibj )  ibj , logit(tibj )  ibj   ibj
ibj ~ N( bj ,  2,bj ) ,  bj ~ N( b ,  2b )
 ibj~ N(bj ,  2bj ) , bj~  b (0)  (1   b ) N (  b ,  2b )
Bayesian Meta Analysis
Hierarchical Logistic Regression (Cont.)
• Other priors
– Stage 1 λbj ~N(μλb , σ2λb );
– Stage 2 μλb~N(μλ0, σ2λ0 ); σ2λb ~IG(αλ, βλ);
μθb ~N(μθ0, σ2θ0 ); σ2θb ~IG(αθ, βθ)
- Stage 3 μλ0~N(μλ00 , σ2λ00 ); σ2λ0 ~IG(αλ00, βλ00)
μθ0~N(μθ00 , σ2θ00 ); σ2θ0 ~IG(αθ00, βθ00)
Hyperparameters μλ00, σ2λ00, αλ00, βλ00, μθ00,
σ2θ00, αθ00, βθ00, αλ, βλ , αθ, βθ are fixed constants
Inference
• AEbj is flagged if
– Pr( θbj > d*| Data) > p, where θbj is log-OR in
Binomial models and log-RR in Poisson
models.
– d* and p are all prespecified constants.
• Graphs are useful tools in deciphering
data and presenting results
Model Selection
• Deviance Information Criteria (DIC) was used to
compare models with same data
• Limited sensitivity analyses were done to check
the robustness of the models
• Different levels of MedDRA structures were used
– SOC/PT, HLT/PT and SOC/HLT/PT
• Treatment effect with additive study effects
model using SOC/PT structure was chosen
Bayesian Meta Analysis
Hierarchical Log-linear Regression
• Poisson models
– Adjust for different exposures in treatment and control
– Assume constant hazard over time
– Unless AEs are fairly common or follow up of studies are
quite unbalanced between treatment arms, usually are not
very different from Binomial models
Yibj~Pois(tibj Tibj ); Xbj~Pois(cibj Cibj)
where tibj and cibj are event rates, and Tibj and Cibj for AEibj are
total exposure times in the treatment and control groups,
respectively
log(cibj )=λibj; log(tibj )=λibj + θbj ,
Note that θbj =log(RRbj)
2.5
Fisher's Exact Test 2-sided P-values
2.0
SIN USITIS
1.5
FATIGUE
EXCORIATION
0.5
1.0
HERPES SIMPLEX
0.0
- log10 (Fisher exact p-value)
ECCHY MOSIS
GASTROINTESTINAL DISORDER...
IN FECTIONS AND INFESTATIO...
SKIN AND SUBCUTANEOUS TIS...
NEOPLASMS BENIGN, MALIGNA...
GENERAL DISORDER S AND ADM...
PSY CHIATRIC DISOR DERS
IN JU RY , POISONING AND PRO...
RESPIR ATOR Y , TH ORACIC AND...
CARDIAC DISORDERS
MUSCULOSKELETAL AND CONNE...
REPRODU CTIVE SY STEM AND B...
RENAL AN D URINARY DISORD E...
EY E DISORDER S
IN VESTIGATIONS
NERVOUS SY STEM DISORDERS
METABOLISM AND N UTRITION ...
IMMUNE SY STEM DISORDERS
EAR AN D LABY RINTH DISORDE...
SURGICAL AND MEDICAL PROC...
VASCULAR D ISORDER S
ENDOCR INE DISORD ERS
HEPATOBILIARY DISORDERS
BLOOD AND LY MPHATIC SY STE...
-0.01
0.0
0.01
Raw Risk Difference(%), Treatment - Placebo
0.02
Peto's Method
M-H TA CC Method
ECCHYMOSIS
FATIGUE
SINUSITIS
SINUSITIS
FATIGUE
HERPES SIMPLEX
HERPES SIMPLEX
ECCHYMOSIS
DYSPEPSIA
HEADACHE
INJECTION SITE BRUISING
INJECTION SITE BRUISING
HEADACHE
DYSPEPSIA
EXCORIATION
INFLUENZA
URTICARIA
NASOPHARYNGITIS
BASAL CELL CARCINOMA
DIZZINESS
TENSION HEADACHE
BRONCHITIS
DIZZINESS
URTICARIA
BRONCHITIS
PHARYNGITIS
INFLUENZA
UPPER RESPIRATORY TRACT INFECTION
GASTROENTERITIS VIRAL
TENSION HEADACHE
0
1
2
log Odds Ratio
3
0
1
2
log Odds Ratio
3
ECCHY MOSIS
0.4
0.6
BLOOD AND LY MPHATIC SY STE...
CARDIAC DISORDERS
EAR AN D LABY RINTH DISORDE...
ENDOCR INE DISORD ERS
EY E DISORDER S
GASTROINTESTINAL DISORDER...
GENERAL DISORDER S AND ADM...
HEPATOBILIARY DISORDERS
IMMUNE SY STEM DISORDERS
IN FECTIONS AND INFESTATIO...
IN JU RY , POISONING AND PRO...
IN VESTIGATIONS
METABOLISM AND N UTRITION ...
MUSCULOSKELETAL AND CONNE...
NEOPLASMS BENIGN, MALIGNA...
NERVOUS SY STEM DISORDERS
PSY CHIATRIC DISOR DERS
RENAL AN D URINARY DISORD E...
REPRODU CTIVE SY STEM AND B...
RESPIR ATOR Y , TH ORACIC AND...
SKIN AND SUBCUTANEOUS TIS...
SURGICAL AND MEDICAL PROC...
VASCULAR D ISORDER S
URTIC ARIA
0.2
SIN USITIS
HERPES SIMPLEX
BASAL CELL CARC INOMA
FATIGUEBLOOD PRESSURE INCREASED
TENSION HEADACHE
DIZZINESS
FOLLICULITIS
DY
SPEPSIA
ANXIETY
PHARY
NGITIS
MIGRAINE
HY
PERSENSITIVITY
DERMAL
CY
NERVE
COMPRESSION
CAROTID
ARTERY
STEN
OSIS
BLOOD
CHOLESTER
OL
IN CREASED
FLUSHIN
G ST
LIVER
DISOR
DER
MEMORY
IMPAIRMENT
GASTROENTERITIS
VIRAL
HY
DIZZINESS
PERTONIA
POSTUR
AL
INSU
DISTUR
BAN
CE
INSY
ATTENTION
HEADACHE
SENSORY
DISTURBANC
E FFICIEN CY
NEURALGIA
SY
NCOPE
CEREBROVASCULAR
FACIAL
FURUNC
PALSY
LE
ACCIDENT
SOMNOLENCE
HY
PERSOMN
IAEL
CY
STITIS
VAGIN
AL
INFECTION
DY
SGEUSIA
CARPAL
TUNN
NDROME
BRONCH
ITIS
RESTLESS
LEGS
SY
DROME
ENDODONTIC
PRNOCEDURE
ACNE
THROAT
IRRITATION
SKIN
IN
FEC
TION
PARAESTHESIA
VIR
AL
UPPER
RESPIRATORY
TR AC T INFEC TION
LETHARGY
SCIATICA
SEBORRHOEIC
KER
ATOSIS
ECZEMA
BURNING
SEN
SATION
EAR
W
EIGHT
INFECTION
DEC
REASED
IN
FEC
TED
IN
SEC
T
BITE
BRONCH
VERTIGO
BLOOD
ITIS
GLU
COSE
CHRONIC
INCREASED
IN
NIGHT
FLUEN
SW
ZA
EATS
SY
LABY
NCOPE
RIN
THITIS
VASOVAGAL
ERECTILE
DY
SFUNCTION
ONY
W
EIGHT
CH
OMY
INCREASED
COSIS
BODY
TEMPERATU
RE
INCREASED
DRY
EY
EINFECTION
HERPES
RHINORRHOEA
VIRUS
INFECTION
MALAISE
CELLULITIS
CONJUNCTIVITIS
STAPH
Y
LOCOCCAL
HY
FUNGAL
EXCORIATION
POAESTHESIA
INFECTION
VIR
NASAL
AL
INFECTION
CON
GESTION
LARY
NGITIS
VERTIGO
POSITIONAL
ONY
CH
OMADESIS
TOOTH
SKIN
SKIN
LESION
PAPILLOMA
ABSCESS
ABDOMINAL
INFECTION
GINGIVAL
DIVER
TINEA
TICULITIS
VERSICOLOU
R
STRESS
INCONTIN
ENCE
SIN
HAEMATURIA
US
HEADACHE
GUTTATE
SIALOADENITIS
SKIN
HAEMORRH
PSORIASIS
AGE
BIPOLAR
TINNITUS
IHISTIOCY
DISOR
DER
ALCOHOLISM
IN
URINARY
PREMATURE
FEC
TED
CRURIS
INCONTINENCE
SEBACEOUS
EJACU
LATION
CY
ST OF ARTERY
RHINITIS
MASTITIS
PULPITIS
IR
RITABILITY
DENTAL
CAT
ACUTE
POSTOPERATIVE
HY
STRESS
PERCHOLESTEROLAEMIA
SCR
SINUSITIS
ATCH
DISEASE
W
OUND
INFECTION
ARTERIOSC
EPISTAXIS
LEROSIS
CORONARY
VAGIN
APPENDICITIS
DY
SPHONIA
AL
TIOUS
CANDIDIASIS
MONONUCLEOSIS
PUNCTURE
AGITATION
ATTEN
TION
SITE
DEFICIT/HY
INFECTION
PERACTIVITY
DISORDER
GENITAL
EY
ELID
INFEC
INFECTION
TION
FEMALE
TONSILLITIS
CANDIDIASIS
ERY
PNEUMONIA
ARRHY
NERVOUSNESS
TENDONITIS
SIPELAS
THMIA
TINEA
ACTINIC
INFECTION
KER
ATOSIS
RASH
PUSTU
LAR
IN
JECTION
SITE
BRUISING
IN
GROW
ING
NAIL
DY
BUNDLE
SPHORIA
BR
ANCH
BLOCK
LEFT
SUBCUTANEOUS
ABSCESS
ORAL
TRIGONITIS
FALL
PU
STULE
CERVICITIS
CONDY
OTITIS
FIBROUS
EXTERNA
LOMA
ACUMINATUM
TOMA
URINARY
CERVICAL
R
DY
ETENTION
SPLASIA
LIPOMA
CALCULUS
DERMATITIS
BLADDER
CON
TAC
T
EAR
SQUAMOUS
SLEEP
D
ISCOMFOR
APNOEA
CELL
SY
T
CARCINOMA
NDROME
SKIN
LY
SIN
MPHOEDEMA
US
CONGESTION
PROSTATE
CANCER
LOW
ER
RESPIRATOR
Y TRACT
INFECTION
POLLAKIURIA
ALLER
JECTION
GIC
COUGH
SITE
HAEMATOMA
TERTRIGO
LIBIDO
D
EC
REASED
IN
SNEEZIN
DY
TERVERTEBRAL
SPNOEA
G
DISC
DEGENERATION
SNORING
NASOPHARY
N
GITIS
TOOTH
DY
CONJUNCTIVAL
SPNOEA
INFECTION
EXER
HAEMORR
TIONAL
HAGE
ABDOMINAL
FOOT
FR
AC
TU
PAIN
RE
UPPER
GASTROINTESTINAL
INFLAMMATION
SHOULDER
PAIN
NEPHROLITH
HY
PERVENTILATION
IASIS
VAGIN
RHINITIS
AL
LACERATION
ALLERGIC
VULVOVAGIN
RHONC
HI
AL
DRY
NESS
HAND
FRACTURE
OTITIS
MEDIA
JOINT
DRY
EY
POST-TRAUMATIC
ELID
MOUTH
EFFUSION
OEDEMA
PR
URITUS
SEASONAL
FLATU
LENC
ALLERGY
E
BENIGN
ABDOMINAL
PROSTATIC
DISTEN
SION
HY
PERPLASIA
EY
E
INFLAMMATION
OVARIAN
CATARACT
HOT
FOOD
FLUSH
POISON
CY
ST
ING
RUPTURED
DIPLOPIA
GASTROOESOPHAGEAL
REFLU
X DISEASE
STOMACH
DISCOMFORT
HY
GASTROINTESTINAL
PERTENSION
PERMOTILITY
PAIN
GINGIVAL
THERMAL
BLEEDING
BUR
N
DEPRESSION
IN
SOMNIA
D
ISORDER
ABDOMINAL
TENDERNESS
ROAD
SCAPULA
TRAFFIC
FRACTUR
ACCIDEN
E
T
TOOTH
FRAC
TUR
EPAIN
HEAT
STROKE
MY
TOOTHACHE
ALGIA
INJURY
ANIMAL
BITE
W
OUND
BACK
PAIN
TENDON
INJURY
CONTUSION
DIARRHOEA
NAUSEA
0.0
P(OR > 1|Data)
0.8
1.0
Posterior Summary of Bayesian Hierarchical Meta-Analysis Model
-3
-2
-1
0
log-OR posterior means
1
2
Inferences of Binomial Hierarchical
Model with Mixture Prior
SOC
GENERAL
DISORDERS AND
ADMINISTRATION
SITE
INFECTIONS AND
INFESTATIONS
PT
FATIGUE
HERPES
SIMPLEX
INFECTIONS AND
INFESTATIONS
SINUSITIS
SKIN AND
SUBCUTANEOUS
TISSUE
DISORDERS
ECCHYMOSIS
Two
sided
Post Post Post
Post
Post
Fisher's Prob Prob Prob
Prob
Prob
Exact
OR=1 OR>1 OR>1.1 OR>1.2 OR>2
0.019
0.82
0.18
0.18
0.18
0.10
0.039
0.75
0.25
0.25
0.25
0.21
0.012
0.72
0.27
0.27
0.27
0.19
0.005
0.11
0.89
0.89
0.89
0.89
Bayesian Patient Level Models
• IPD models to include within patient correlation and
patient level factors while incorporating MedDRA coding
hierarchy
• Data from one study is used:618 subject, 207 unique
AEs, N = 127926
Ykbj ~ Bernoulli(tkbj ), X kbj ~ Bernoulli(ckbj ),


logit(ckbj )  bj   Zk   k , logit(tkbj )  bj   bj   Zk   k

where Zk is theset of covariatesfor subject k

 2
 ~ MVN(0,  b I),  k ~ Norm al(0,  2 )
Simulation Study
• Simulation scheme:
– Randomly assign subjects to treatment or placebo to create a
“null” scenario
– Adverse events within subject remain unchanged to maintain the
SOC/PT hierarchy
– 1000 simulated datasets
• Family-wise error rates (also FDR in this case) of
Fisher’s exact text unadjusted for multiplicity and
Poisson regression with mixture prior are compared
• Percentage of simulated datasets yielding Y=0, 1, 2 or
≥3 incorrectly flagged adverse events out of 464 PTs are
also compared
Distribution of Y (%)
FWER/
FDR
1
2
>=3
Non-adjusted 2sided Fisher’s
exact test
2-sided test, p-value <=0.05
99.1
2.7
5.4
91
Non-adjusted 1sided Fisher’s
exact test
p-value<=0.05
90.1
22.7
26.9
40.5
Bayes
Hierarchical
Poisson Model *
c=1, p=0.7
10.2
7.5
2.0
0.7
c=1, p=0.8
6.1
4.6
1.3
0.2
c=1, p=0.9
2.8
2.7
0.1
0.0
c=1.2, p=0.7
10.2
7.6
1.9
0.7
c=1.2, p=0.8
6.0
4.6
1.2
0.2
c=1.2, p=0.9
2.7
2.6
0.1
0.0
c=2, p=0.7
7.0
5.3
1.5
0.2
c=2, p=0.8
3.7
3.3
0.4
0.0
c=2, p=0.9
1.3
1.2
0.1
0.0
RR≠1, p=0.7
13.8
10.3
2.5
1.0
RR≠1, p=0.8
7.9
6.3
1.4
0.2
RR≠1, p=0.9
3.6
3.5
0.1
0.0
*
Pr(RRbj  c | Data)  p
Simulation Study
• 464 independent tests with alpha=0.05 would
yield in average about 23 signals and have
FDR=1 if no multiplicity is adjusted for.
• Correlation of the AE data reduced the error rate
in our simulation study
• But the FDR is still as high as 99.1%. For 91%
cases there are at least 3 falsely identified
signals.
• The FWERs/FDRs for all Bayes model results
are much lower and acceptable.
Closing Remarks
• Current traditional approach of flagging
routinely collected AEs based on
unadjusted p-values or CIs can result in
excessive false positive signals
– As a result, it can cause undue concern for
approval/labeling/post marketing commitment
• Commonly used meta-analysis methods
for aggregated binary outcome (OR)
– Peto’s method is not recommended for
severely unbalanced studies or common
events unless treatment effects are small
– MH method: needs continuity correction
Closing Remarks (Cont.)
• Bayesian meta-analysis hierarchical mixture modeling
provides a useful tool to analyze data from multiple
studies and address multiplicity
– Allows for explicitly modeling AEs with the existing MedDRA coding
structure
– Use a mixture prior by allowing a point mass on equality of the
treatment and control rates
– Study differences can be accounted for
– No need to add continuity correction. Double zero studies are included.
– For less common AEs and studies without a great amount of follow-up
variation between treatment groups, inferences from Poisson regression
and logistic regression models are very similar
• Computation for signal detection using IPD is
challenging
• Graphics are effective in displaying flagged signals
Future Work
• More sensitivity analysis on the performance of the
models
• Further simulation study on type II error and operating
characteristics of Bayesian models
• Zero-inflated Poisson model might be a good approach
for relatively healthy population
• Incorporating severity information of AEs
• Multi-axial structure of MedDRA coding system
• The field of clinical trial signal detection is still in its
infancy
– More research and practice are needed
– Statisticians need to work with clinicians/safety scientists closely
to further advance this field
References
•
•
•
•
•
•
•
•
Bate A, Lindquist M, Edwards, IR, Olsson S, Orre R, Lansner A, and De
Freitas RM (1998). A Bayesian neural network method for adverse drug
reaction signal detection. Eur J Clin Pharmacol 54:315-321
Berry S and Berry D (2004) Accounting for multiplicities in assessing drug
safety: a three-level hierarchical mixture model. Biometrics, 60: 418-426
Chi G, Hung HMJ, and O’Neill R (2002). Some comments on “Adaptive
Trials and Bayesian Statistics in Drug Development” by Don Berry. In
Pharmaceutical Report, Vol 9, 1-11
Crowe B, Xia A, Watson D, Shi H, Lin S, Kuebler J, Berlin J, et al. (2008).
Recommendations for Safety Planning, Data Collection, Evaluation and
Reporting During Drug, Biologic and Vaccine Development: A Report of the
PhRMA Safety Planning, Evaluation and Reporting Team (SPERT).
Manuscript in preparation.
DuMouchel W (1999). Bayesian data mining in large frequency tables, with
an application to the FDA Spontaneous Reporting System (with discussion).
The American Statistician 53:177-202
Gould AL. Drug safety evaluation in and after clinical trials. Deming
Conference, Atlantic City, 3 December 2002
Mehrotra, DV and Heyse, JF (2004). Multiplicity considerations in clinical
safety analysis. Statistical Methods in Medical Research 13, 227-238
Spiegelhalter DJ, Best NG, Carlin BP and van der Linde A (2002) Bayesian
measures of model complexity and fit (with discussion). J. Roy. Statist. Soc.
B. 64, 583-640.
Thank You!
Back-up Slides
50
0
10
20
30
40
Min. 0
1st Qu. 0
Median 0
Mean 0.003
3rd Qu. 0.003
Max. 0.333
0.00
0.10
0.20
0.30
80
Incidence Rate of Treatment Arm
0
20
40
60
Min. 0
1st Qu. 0
Median 0
Mean 0.003
3rd Qu. 0
Max. 0.164
0.00
0.05
0.10
0.15
Incidence Rate of Placebo Arm
Bayesian Hierarchical Model for AE
(Berry & Berry 2004)
• PT level assumptions:
 ij ~ N (  i ,  2 )
 ij ~  i I [0]  (1   i ) N (  i ,  2 )
• SOC level
assumptions:
i
 i
i
~
N (   0 , T2 )
~
~
N (   0 , T2 )
Beta(  ,   )
• Global
assumptions:
 0
 0
~
N (  00 , T200 )
~
N (  00 , T200 )
T2
~
IG( T ,  T )
T2
~
IG( T ,  T )
 2
 2


~
IG(  ,   )
~
IG(  ,   )
~
Exp( ) I (   1)
~
Exp( ) I (    1)
SOC: Injury, Poisoning and
Procedural Complications
Preferred Term
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Alcohol Poisoning
Animal Bite
Arthropod Bite
Back Injury
Cartilage Injury
Chemical Eye Injury
Closed Head Injury
Concussion
Contrast Media Reaction
Contusion
Epicondylitis
Excoriation
Eye Injury
Fall
Foot Fracture
Foreign Body in Eye
Hand Fracture
Heat Exhaustion
Heat Stroke
Hip Fracture
Injury Corneal
Jaw Fracture
Joint Injury
Joint Sprain
Ligament Injury
Ctrl
r(%)
0.00
0.14
0.56
0.69
0.14
0.14
0.00
0.28
0.14
0.83
0.14
0.00
0.14
0.00
0.00
0.14
0.00
0.14
0.00
0.00
0.14
0.14
0.00
0.28
0.14
Trt
r(%)
0.08
0.24
0.24
0.24
0.00
0.00
0.08
0.00
0.00
0.80
0.08
0.64
0.08
0.32
0.32
0.00
0.24
0.00
0.08
0.08
0.00
0.08
0.08
0.16
0.08
diff
(Trt-Ctrl )
0.080
0.102
-0.315
-0.453
-0.139
-0.139
0.080
-0.278
-0.139
-0.030
-0.059
0.643
-0.059
0.321
0.321
-0.139
0.241
-0.139
0.080
0.080
-0.139
-0.059
0.080
-0.117
-0.059
Preferred Term
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
Limb Injury
Mouth Injury
Muscle Injury
Muscle Strain
Neck Injury
Pain Trauma Activated
Pneumothorax Traumatic
Post Procedural Complication
Post Procedural Pain
Procedural Pain
Rib Fracture
Road Traffic Accident
Scapula Fracture
Skeletal Injury
Skin Laceration
Splinter
Sunburn
Tendon Injury
Thermal Burn
Tooth Fracture
Tooth Injury
Traumatic Haematoma
Wound
Wrist Fracture
Ctrl
r(%)
0.14
0.00
0.00
0.69
0.14
0.14
0.00
0.14
0.00
0.28
0.14
0.00
0.00
0.42
1.11
0.14
0.28
0.00
0.00
0.00
0.00
0.28
0.14
0.14
Trt
r(%)
0.16
0.08
0.08
0.72
0.08
0.40
0.08
0.00
0.08
0.08
0.08
0.16
0.08
0.00
0.72
0.08
0.24
0.08
0.16
0.08
0.16
0.08
0.24
0.00
diff
(Trt-Ctrl )
0.022
0.080
0.080
0.028
-0.059
0.263
0.080
-0.139
0.080
-0.197
-0.059
0.161
0.080
-0.417
-0.388
-0.059
-0.037
0.080
0.161
0.080
0.161
-0.197
0.102
-0.139