Transcript Sample Size and Power

Sample Size and Power
Laura Lee Johnson
[email protected]
Tuesday, November 2, 2004
Objectives
• Intuition behind power and sample
size calculations
• Common sample size formulas for
the tests
• Tying the first three lectures
together
Take Away Message
• Get some input from a statistician
– This part of the design is vital and
mistakes can be costly!
• Take all calculations with a few grains
of salt
– “Fudge factor” is important!
• Analysis Follows Design
Outline
•
•
•
•
•
•
•
Power
Basic Sample Size Information
Examples (see text for more)
Changes to the basic formula
Multiple comparisons
Poor proposal sample size statements
Conclusion and Resources
Power Depends on Sample Size
• Power = 1-β = P( reject H0 | H1 true )
– “Probability of rejecting the null
hypothesis if the alternative
hypothesis is true.”
• More subjects  higher power
Power is Effected by…..
• Variation in the outcome (σ2 )
– ↓ σ2
→ power ↑
• Significance level (α)
–↑α
→ power ↑
• Difference (effect) to be detected (δ)
–↑δ
→ power ↑
• One-tailed vs. two-tailed tests
– Power is greater in one-tailed tests than in
comparable two-tailed tests
Power Changes
• 2n = 32, 2 sample test, 81% power,
δ=2, σ = 2, α = 0.05, 2-sided test
• Variance/Standard deviation
– σ: 2 → 1 Power: 81% → 99.99%
– σ: 2 → 3 Power: 81% → 47%
• Significance level (α)
– α : 0.05 → 0.01 Power: 81% → 69%
– α : 0.05 → 0.10 Power: 81% → 94%
Power Changes
• 2n = 32, 2 sample test, 81% power, δ=2,
σ = 2, α = 0.05, 2-sided test
• Difference to be detected (δ)
– δ : 2 → 1 Power: 81% → 29%
– δ : 2 → 3 Power: 81% → 99%
• Sample size (n)
– n: 32 → 64 Power: 81% → 98%
– n: 32 → 28 Power: 81% → 75%
• One-tailed vs. two-tailed tests
– Power: 81% → 88%
Power should be….?
• Phase III: industry minimum = 80%
• Some say Type I error = Type II
error
• Many large “definitive” studies
have power around 99.9%
• Proteomics/genomics studies: aim
for high power because Type II
error a bear!
Power Formula
• Depends on study design
• Not hard, but can be VERY algebra
intensive
• May want to use a computer
program or statistician
Outline
•
•
•
•
•
•
•
Power
Basic Sample Size Information
Examples (see text for more)
Changes to the basic formula
Multiple comparisons
Rejected sample size statements
Conclusion and Resources
Sample Size Formula Information
• Variables of interest
– type of data e.g. continuous, categorical
•
•
•
•
Desired power
Desired significance level
Effect/difference of clinical importance
Standard deviations of continuous
outcome variables
• One or two-sided tests
Sample Size and Study
Design
•
•
•
•
•
•
•
Randomized controlled trial (RCT)
Block/stratified-block randomized trial
Equivalence trial
Non-randomized intervention study
Observational study
Prevalence study
Measuring sensitivity and specificity
Sample Size and Data
Structure
•
•
•
•
Paired data
Repeated measures
Groups of equal sizes
Hierarchical data
Notes
• Non-randomized studies looking
for differences or associations
– require larger sample to allow
adjustment for confounding factors
• Absolute sample size is of interest
– surveys sometimes take % of
population approach
More Notes
• Study’s primary outcome is the variable
you do the sample size calculation for
– If secondary outcome variables
considered important make sure sample
size is sufficient
• Increase the ‘real’ sample size to reflect
loss to follow up, expected response
rate, lack of compliance, etc.
– Make the link between the calculation and
increase
Purpose→Formula→Analysis
• Demonstrate superiority
– Sample size sufficient to detect
difference between treatments
• Demonstrate equally effective
– Equivalence trial or a 'negative' trial
– Sample size required to demonstrate
equivalence larger than required to
demonstrate a difference
Outline
•
•
•
•
•
•
•
Power
Basic sample size information
Examples (see text for more)
Changes to the basic formula
Multiple comparisons
Rejected sample size statements
Conclusion and Resources
Sample Size in Clinical Trials
•
•
•
•
Two groups
Continuous outcome
Mean difference
Similar ideas hold for other
outcomes
Phase I: Dose Escalation
• Dose limiting toxicity (DLT) must
be defined
• Decide a few dose levels (e.g. 4)
• At least three patients will be
treated on each dose level (cohort)
• Not a power or sample size
calculation issue
Phase I (cont.)
• Enroll 3 patients
• If 0/3 patients develop DLT
– Escalate to new dose
• If DLT is observed in 1 of 3 patients
– Expand cohort to 6
– Escalate if 3/3 new patients do not
develop DLT (i.e. 1/6 develop DLT)
Phase I (cont.)
• Maximum Tolerated Dose (MTD)
– Dose level immediately below the
level at which ≥2 patients in a cohort
of 3 to 6 patients experienced a DLT
• Usually go for “safe dose”
– MTD or a maximum dosage that is
pre-specified in the protocol
Phase I Note
• Entry of patients to a new dose
level does not occur until all
patients in the previous level are
beyond a certain time frame where
you look for toxicity
• Not a power or sample size
calculation issue
Phase II Designs
• Screening of new therapies
• Not to prove efficacy
• Sufficient activity to be tested in a
randomized study
• Issues of safety paramount
• Small number of patients
Phase II Design Problems
• Placebo effect
• Investigator bias
• Might be unblinded or single
blinded treatment
• Regression to the mean
Phase II Example : Two-Stage
Optimal Design
• Single arm, two stage, using an
optimal design & predefined
response
• Rule out response probability of
20% (H0: p=0.20)
• Level that demonstrates useful
activity is 40% (H1:p=0.40)
• α = 0.10, β = 0.10
Phase II:
Two-Stage Optimal Design
• Seek to rule out undesirably low
response probability
– E.g. only 20% respond (p0=0.20)
• Seek to rule out p0 in favor of p1;
shows “useful” activity
– E.g. 40% are stable (p1=0.40)
Two-Stage Optimal Design
• Let α = 0.1 (10% probability of
accepting a poor agent)
• Let β = 0.1 (10% probability of
rejecting a good agent)
• Charts in Simon (1989) paper with
different p1 – p0 amounts and
varying α and β values
Table from Simon (1989)
Blow up: Simon (1989) Table
Phase II Example
• Initially enroll 17 patients.
– 0-3 of the 17 have a clinical response
then stop accrual and assume not an
active agent
• If ≥ 4/17 respond, then accrual will
continue to 37 patients.
Phase II Example
• If 4-10 of the 37 respond this is
insufficient activity to continue
• If ≥ 11/37 respond then the agent will be
considered active.
• Under this design if the null hypothesis
were true (20% response probability)
there is a 55% probability of early
termination.
Sample Size Differences
• If the null hypothesis (H0) is true
• Using two-stage optimal design
– On average 26 subjects enrolled
• Using a 1-sample test of proportions
– 34 patients
– If feasible
• Using a 2-sample randomized test of
proportions
– 86 patients per group
Phase II: Historical Controls
• Want to double disease X survival
from 15.7 months to 31 months.
• α = 0.05, one tailed, β = 0.20
• Need 60 patients, about 30 in each
of 2 arms; can accrue 1/month
• Need 36 months of follow-up
• Use historical controls
Phase II: Historical Controls
• Old data set from 35 patients treated at
NCI with disease X, initially treated
from 1980 to 1999
• Currently 3 of 35 patients alive
• Median survival time for historical
patients is 15.7 months
• Almost like an observational study
• Use Dixon and Simon (1988) method for
analysis
Phase III Survival Example
• Primary objective: determine if
patients with metastatic melanoma
who undergo Procedure A have a
different overall survival compared
with patients receiving standard of
care (SOC)
• Trial is a two arm randomized
phase III single institution trial
Number of Patients to Enroll?
• 1:1 ratio between the two arms
• 80% power to detect a difference
between 8 month median survival and
16 month median survival
• Two-tailed α = 0.05
• 24 months of follow-up after the last
patient has been enrolled.
• 36 months of accrual
Phase III Survival
• Look at nomograms (Schoenfeld
and Richter). Can use formulas
• Need 38/arm, so let’s try to recruit
42/arm – total of 84 patients
• Anticipate approximately 30
patients/year entering the trial
Sample Size Example
• Study effect of new sleep aid
• 1 sample test
• Baseline to sleep time after taking the
medication for one week
• Two-sided test, α = 0.05, power = 90%
• Difference = 1 (4 hours of sleep to 5)
• Standard deviation = 2 hr
Sleep Aid Example
•
•
•
•
n
1 sample test
2-sided test, α = 0.05, 1-β = 90%
σ = 2hr (standard deviation)
δ = 1 hr (difference of interest)
( Z1 / 2  Z1  ) 2  2
2
(1.960  1.282) 2 22

 42.04  43
2
1
Sample Size:
Change Effect or Difference
• Change difference of interest from 1hr
to 2 hr
• n goes from 43 to 11
(1.960  1.282) 2
n
 10.51  11
2
2
2
2
Sample Size: Change Power
• Change power from 90% to 80%
• n goes from 11 to 8
• (Small sample: start thinking about
using the t distribution)
(1.960  0.841) 2
n
 7.85  8
2
2
2
2
Sample Size:
Change Standard Deviation
• Change the standard deviation from 2
to 3
• n goes from 8 to 18
(1.960  0.841) 3
n
 17.65  18
2
2
2
2
Sleep Aid Example: 2 Sample
• Original design (2-sided test, α = 0.05, 1-β =
90%, σ = 2hr, δ = 1 hr)
• Two sample randomized parallel design
• Needed 43 in the one-sample design
• In 2-sample need twice that, in each group!
• 4 times as many people are needed in this
design
n
2( Z1 / 2  Z1  ) 2  2
2
2(1.960  1.282) 2 22

 84.1  85  170 total!
2
1
Sample Size:
Change Effect or Difference
• Change difference of interest from 1hr
to 2 hr
• n goes from 72 to 44
2(1.960  1.282) 2
n
 21.02  22  44 total
2
2
2
2
Sample Size: Change Power
• Change power from 90% to 80%
• n goes from 44 to 32
2(1.960  0.841)2 22
n
 15.69  16  32 total
2
2
Sample Size:
Change Standard Deviation
• Change the standard deviation from 2
to 3
• n goes from 32 to 72
2(1.960  0.841) 2 32
n
 35.31  36  72 total
2
2
Conclusion
• Changes in the detectable difference have
HUGE impacts on sample size
– 20 point difference → 25 patients/group
– 10 point difference → 100 patients/group
– 5 point difference → 400 patients/group
• Changes in α, β, σ, number of samples, if it is
a 1- or 2-sided test can all have a large
impact on your sample size calculation
2N 
4( Z1 / 2  Z1  ) 2  2
2
Sample Size:
Matched Pair Designs
• Similar to 1-sample formula
• Means (paired t-test)
– Mean difference from paired data
– Variance of differences
• Proportions
– Based on discordant pairs
Examples in the Text
•
•
•
•
Several with paired designs
Two and one sample means
Proportions
How to take pilot data and design
the next study
Outline
•
•
•
•
Power
Basic Sample Size Information
Examples (see text for more)
Changes to the basic formula/
Observational studies
• Multiple comparisons
• Rejected sample size statements
• Conclusion and Resources
Unequal #s in Each Group
• Ratio of cases to controls
• Use if want λ patients randomized to the
treatment arm for every patient randomized
to the placebo arm
• Take no more than 4-5 controls/case
n2   n1   controls for every case
n1 
( Z1 / 2  Z1  ) 2 ( 12   22 /  )
2
K:1 Sample Size Shortcut
• Use equal variance sample size
formula: TOTAL sample size
increases by a factor of
(k+1)2/4k
• Total sample size for two equal
groups = 26; want 2:1 ratio
• 26*(2+1)2/(4*2) = 26*9/8 = 29.25 ≈ 30
• 20 in one group and 10 in the other
Unequal #s in Each Group:
Fixed # of Cases
• Case-Control Study
• Only so many new devices
• Sample size calculation says n=13
cases and controls are needed
• Only have 11 cases!
• Want the same precision
• n0 = 11 cases
• kn0 = # of controls
How many controls?
n
k
2n0  n
• k = 13 / (2*11 – 13) = 13 / 9 = 1.44
• kn0 = 1.44*11 ≈ 16 controls (and 11 cases)
– Same precision as 13 controls and 13
cases
# of Events is Important
• Cohort of exposed and unexposed
people
• Relative Risk = R
• Prevalence in the unexposed
population = π1
Formulas and Example
Risk of event in exposed group
R
Risk of event in unxposed group
n1 
( Z1 / 2  Z1  )
2
 #of events in unexposed group
2( R  1)
n2  Rn1  #events in exposed group
2
n1 and n2 are the number of events in the two groups
required to detect a relative risk of R with power 1-
N  n1 /  1  # subjects per group
# of Covariates and # of Subjects
• At least 10 subjects for every variable
investigated
– In logistic regression
– No general justification
– This is stability, not power
– Peduzzi et al., (1985) biased regression
coefficients and variance estimates
• Principle component analysis (PCA)
(Thorndike 1978 p 184): N≥10m+50 or
even N ≥ m2 + 50
Balanced designs are easier to
analyze
• Equal numbers in two groups is
the easiest to handle
• If you have more than two groups,
still, equal sample sizes easiest
• Complicated design = simulations
– Done by the statistician
Outline
•
•
•
•
•
•
•
Power
Basic Sample Size Information
Examples (see text for more)
Changes to the basic formula
Multiple comparisons
Rejected sample size statements
Conclusion and Resources
Multiple Comparisons
• If you have 4 groups
– All 2 way comparisons of means
– 6 different tests
• Bonferroni: divide α by # of tests
– 0.025/6 ≈ 0.0042
• High-throughput laboratory tests
DNA Microarrays/Proteomics
• Same formula (Simon et al. 2003)
– α = 0.001 and β = 0.05
– Possibly stricter
• Simulations (Pepe 2003)
– based on pilot data
– k0 = # genes going on for further study
– k1 = rank of genes want to ensure you get
P[ Rank (g) ≤ k0 | True Rank (g) ≤ k1 ]
Outline
•
•
•
•
•
•
•
Power
Basic Sample Size Information
Examples (see text for more)
Changes to the basic formula
Multiple comparisons
Rejected sample size statements
Conclusion and Resources
Rejected Sample Size
Statements
• "A previous study in this area recruited
150 subjects and found highly
significant results (p=0.014), and
therefore a similar sample size should
be sufficient here."
– Previous studies may have been
'lucky' to find significant results, due
to random sampling variation.
No Prior Information
• "Sample sizes are not provided
because there is no prior information
on which to base them."
• Find previously published information
• Conduct small pre-study
• If a very preliminary pilot study, sample
size calculations not usually necessary
Variance?
• No prior information on standard
deviations
– Give the size of difference that may
be detected in terms of number of
standard deviations
Number of Available Patients
• "The clinic sees around 50 patients a year, of
whom 10% may refuse to take part in the
study. Therefore over the 2 years of the
study, the sample size will be 90 patients. "
• Although most studies need to balance
feasibility with study power, the sample size
should not be decided on the number of
available patients alone.
• If you know # of patients is an issue, can
phrase in terms of power
Outline
•
•
•
•
•
•
•
Power
Basic Sample Size Information
Examples (see text for more)
Changes to the basic formula
Multiple comparisons
Rejected sample size statements
Conclusion and Resources
Conclusions
• Changes in the detectable difference have
HUGE impacts on sample size
– 20 point difference → 25 patients/group
– 10 point difference → 100 patients/group
– 5 point difference → 400 patients/group
• Changes in α, β, σ, number of samples, if it is
a 1- or 2-sided test can all have a large
impact on your sample size calculation
4( Z1 / 2  Z1  ) 
2
2N 

2
2
No Estimate of the Variance?
• Make a sample size or power table
• Use a wide variety of possible
standard deviations
• Protect with high sample size if
possible
Analysis Follows Design
• Questions → Hypotheses →
• Experimental Design → Samples →
• Data → Analyses →Conclusions
• Take all of your design information to a
statistician early and often
– Guidance
– Assumptions
Resources: General Books
• Altman (1991) Practical Statistics for Medical
Research. Chapman and Hall
• Bland (2000) An Introduction to Medical
Statistics, 3rd. ed. Oxford University Press
• Armitage, Berry and Matthews (2002)
Statistical Methods in Medical Research, 4th
ed. Blackwell, Oxford
• Fisher and Van Belle (1996, 2004) Wiley
• Simon et al. (2003) Design and Analysis of
DNA Microarray Investigations. Springer
Verlag
Sample Size Specific Tables
• Continuous data: Machin et al. (1998)
Statistical Tables for the Design of Clinical
Studies, Second Edition Blackwell, Oxford
• Categorical data: Lemeshow et al. (1996)
Adequacy of sample size in health studies.
Wiley
• Sequential trials: Whitehead, J. (1997) The
Design and Analysis of Sequential Clinical
Trials, revised 2nd. ed. Wiley
• Equivalence trials: Pocock SJ. (1983) Clinical
Trials: A Practical Approach. Wiley
Resources: Articles
• Simon R. Optimal two-stage
designs for phase II clinical trials.
Controlled Clinical Trials. 10:1-10,
1989.
• Thall, Simon, Ellenberg. A twostage design for choosing among
several experimental treatments
and a control in clinical trials.
Biometrics. 45(2):537-547, 1989.
Resources: Articles
• Schoenfeld, Richter. Nomograms for
calculating the number of patients needed
for a clinical trial with survival as an
endpoint. Biometrics. 38(1):163-170, 1982.
• Bland JM and Altman DG. One and two sided
tests of significance. British Medical Journal
309: 248, 1994.
• Pepe, Longton, Anderson, Schummer.
Selecting differentially expressed genes from
microarry experiments. Biometrics.
59(1):133-142, 2003.
Resources: URLs
• Sample size calculations simplified
– http://www.tufts.edu/~gdallal/SIZE.HTM
• Statistics guide for research grant
applicants, St. George’s Hospital Medical
School
– http://www.sghms.ac.uk/depts/phs/guide/
size.htm
• Software: nQuery, EpiTable, SeqTrial, PS
(http://www.mc.vanderbilt.edu/prevmed/ps/)
Questions?