Transcript Is Your Design Statistically Sound?

Is Your Study Statistically Sound?
Michelle Secic, M.S.
President
Secic Statistical Consulting, Inc.
www.secicstats.com
Is Your Study Statistically Sound?
•
•
•
•
•
•
Primary goals
CRF’s
Sample size
Randomization
Ethics
Reporting Guidelines
Primary goals
- Can your goals be met with the design
you picked?
- Primary hypothesis
- Primary endpoints
Primary goals
Examples
• Can goals be met with the design you picked?
Paired design - pre and post intervention
• All subjects get intervention and study design will allow
testing the effect of the intervention.
• BUT, without a control group…
– can only say that the intervention had a significant effect
– no statement can be made on how it compares to current
standard
Primary goals
Examples
• Primary Hypothesis
“This study is designed to prove that
device Z controls knee pain.”
“This study is designed to assess the
effects of device Z on local knee pain 24
hours post anterior cruciate ligament knee
surgery compared to the standard device.”
Primary goals
Examples
• Primary endpoint
How is the primary endpoint measured?
Primary goals
Examples
• Clear(easy)
death = alive/dead
gender = male/female
• Convoluted (more decisions involved)
height = feet or centimeters?
pain = visual analog scale or morphine usage via
self pump or via number of pills?
Primary goals
Examples
• Surrogate (more complex issues)
current lesion size = stage of cancer
– state the appropriateness of using
surrogate - references, etc.
– if study is for assessing the
appropriateness of surrogate, need to have
longevity - to capture actual endpoint for
assessment of agreement
CRF’s
– Typically the last thing developed prior to
submission - rushing can lead to problems
of omission
– With a draft of the CRF’s statisticians are
able to determine if the appropriate data
will be collected for answering the specific
questions that the study is designed to
answer
CRF’s
Examples
- Researchers claim their study will address
prevention of repeated heart attacks in a
population that has already suffered from a
heart attack.
- Stack of CRF’s arrive and you realize
there was not a CRF on family history.
UGH!
- Involve a statistician!
Sample size
– Big differences need fewer subjects
– Small differences need more subjects
difference 
sample size
– Little variability needs fewer subjects
– Lots of variability needs more subjects
variability 
sample size
Sample size
– Example: Percent of pressure ulcers
study mattress vs. control mattress
– Want to show study mattress is as good as (or
substantially equivalent to) control mattress
– Assume:
• Control group has a rate of 20%
• Allow 10% difference while still calling groups
equivalent
• Need approximately 200 patients in each group
Sample size
– Other factors that affect sample size:
•
•
•
•
•
Power
Significance level
Attrition rate
Number of objectives
Number of interim analyses
Sample size
Power
Probability of correctly concluding that the data
support the desired hypothesis:
Basic Examples:
- probability of correctly concluding
equivalence in an equivalence trial
- probability of correctly concluding
superiority in a comparative trial
Also called “1- beta” - should be at least 80%
Sample size
Significance level
Probability of incorrectly concluding that the
data do not support the desired hypothesis
(i.e., probability of missing the desired finding):
Basic Examples:
- probability of incorrectly concluding
non-equivalence in an equivalence trial
- probability of incorrectly concluding
non-superiority in a comparative trial
Also called “alpha” - should 5% or less
Sample size
Attrition Rate
% of subjects on which you expect incomplete
data due to:
-
withdrawal
death
long study follow-up
complications that prevent them from
completing the study
Sample size
Number of Objectives
An increase in the number of objectives causes
an increase the required sample size
objectives 
sample size
If there are 2 primary objectives (safety & efficacy,
for example), then the power/alpha have to be
adjusted to incorporate both objectives into the
calculations
Sample size
Number of Interim Analyses
An increase in the number of interim analyses
causes an increase the required sample size
interims 
sample size
The more time you plan to ‘peek’ at the data, the
more subjects you will need for your study
Sample size
Number of Interim Analyses
- May want to take early peeks at the data
• Long studies
• Expensive studies
• Better efficacy than expected
Randomization
What is Randomization?
- A way of dividing subjects into groups in
such a way that the characteristics of
the subjects in the groups are balanced
(i.e., similar proportions of males and
females, similar ages, etc.)
- To achieve this, we allow chance to decide
which group each subject is allocated to,
so each subject is equally likely to be
allocated to any of the available groups
Randomization
Why Randomize?
- Want to be able to conclude that any
differences found between the groups is
due to the intervention not because the
subjects were inherently different from
the start
Randomization
How To Randomize?
- Toss coin
- Problems with this method include:
- No audit trail
- Researcher can toss again, if they do not like
the result
- No way to prevent ‘runs’
- Computer generation
- Preferred method
- Excel has random number generator
- Any statistician can generate the randomization
scheme
Randomization
Decisions in Randomization
- Blocking
- Purpose of blocking is to prevent ‘runs’
- Example
- Block comparison groups by a size of 6
- This guarantees that each block of 6
assignments will have an equal number of
treatment assignments (i.e., ABABAB, ABBAAB,
BBBAAA, etc.)
Randomization
Decisions in Randomization
- Stratification
- Purpose of stratification is to remove
inherent imbalances
- Example
- Know there are many more women than men
who develop breast cancer
- Have a new preventative drug that you want to
ensure works on both genders
- Then you stratify by gender to ensure there are
equal numbers of males and females, since you
know, inherently, that randomization alone will
not work.
Randomization
Decisions in Randomization
- Minimization
- Further ensure balance in subject characteristics
between groups when randomization and
stratification are still not enough
- Example
- The first patient is randomized to either A or B.
- Researcher picks all factors that need to be reviewed
by the computer (i.e., age, history of risk, weight,
etc.).
- When subsequent subjects are recruited and their
prognostic characteristics noted, their allocation is
decided such that the overall imbalance in the
groups at that point is minimized.
Randomization
Decisions in Randomization
- Blinding/Masking
- Reduces bias by preventing patients, caregivers,
and even statisticians from knowing who is in the
experimental group and who is in the control group.
- In a single-masked study, only the patients are
masked.
- In a double-masked study, the patients and data
collectors (the caregivers, investigators, researchers,
coordinators, etc.) are masked.
- Although rare, in a triple-masked study, the patients,
data collectors, and data evaluators are masked.
Randomization
Decisions in Randomization
- Procedures
- Lists
- Sealed envelopes
- IVRS
Randomization
Decisions in Randomization
- Must be prepared by someone who will not be
involved in the recruitment
- Open lists are not preferred, since bias
(intentional or unintentional) can easily enter
into the process
- Sealed, labeled, numbered, non-transparent
envelopes are adequate for most studies
- IVRS is preferred for large, long-term, multicenter studies
Ethics
• Ethics (3 basic principles)
(Hulley & Cummings, 1988)
– inform subjects - consent
– ensure benefits of research are
proportionate to the risks assumed by
individual subjects (not just to those who
may benefit later)
– no single group (disadvantaged or
vulnerable) should bear a disproportionate
share of the risk
Ethics
• Ethics/Issues in:
randomization - prevention of selection bias due
to determining who is assigned to each group
(Meinert, 1986)
– Best for studying efficacy, but may lead to some ethical
concerns
– Researcher believes one device/drug is really superior,
or is best for particular subgroups
– Not comfortable randomizing to a non-intervention
group (placebo)
– Cannot have placebo arm when standard of care has
already been established (i.e., heparin in cardiology
trial is always considered ‘placebo’ arm)
Ethics
• Ethics/Issues in:
blinding - preventing intervention
related bias due to measurement or
ascertainment errors (Meinert, 1986)
– Sometimes blinding is not possible - need to
discuss how potential bias will be addressed
– Especially crucial with subjective data
Reporting Guidelines
• Reporting Guidelines
– Thinking about how you will report your findings
or present them to the public aids in designing a
“good” study
– Ask researcher, “What do you want your main
conclusion statement to be?”
• helps researcher narrow in on broad ideas such as,
“I want to test this new device.”
Book with comprehensive guidelines:
How To Report Statistics in Medicine, (Lang &
Secic, 1997)
Is Your Study Statistically Sound?
• References
– Hulley SB, Cummings SR. Designing Clinical
Research. Williams & Wilkins, Baltimore,
MD, 1988.
– Lang T, Secic M. How To Report Statistics in
Medicine: Annotated Guidelines for Authors,
Editors, and Reviewers. American College of
Physicians, Philadelphia, PA, 1997.
– Meinert CL. Clinical Trials: Design, Conduct,
and Analysis. Oxford University Press, Inc.,
New York, NY, 1986.