Transcript B. Herman Presentation

How to Torture Your Statistician:
Ben Herman
ACRIN Biostatistics Center
Brown University
Providence, RI
ACRIN Fall Meeting – RA Session
ACRIN
Fall
Meeting – RA or
Session
Title
of Presentation
Statistical
Presentation Conference Here
29 September 2010
September
29,2010
27
March 2016
Acknowledgements:
Divider (sub-section) Slide
•
Although they undoubtedly would like to distance themselves from the title and the rest of this talk, ACRIN
receives funding from the National Cancer Institute through the grants U01 CA079778 and U01 CA080098
* Thanks to all the web sites/sources from which I liberally borrowed much of this material (not all of which is cited)
ACRIN Fall Meeting – RA Session
September 29,2010
Divider (sub-section) Slide
Disclaimer: The view expressed in the presentation are wholly my own, they do not
necessarily represent ACRIN, ACRIN’s data collection policies, Data Management, real
statistics, or the general views of the Biostatistics Center. Shhh… nobody knows I’m giving
this talk.
ACRIN Fall Meeting – RA Session
September 29,2010
Charlatans, Liars & Cheats?
And Everybody Else
“The group was alarmed to find that if you are a
labourer, cleaner or dock worker, you are twice as
likely to die than a member of the professional
classes”, from The Sunday Times, 31st August 1980.
ACRIN
TitleFall
of Presentation
Meeting – RA or
Session
Conference Here
27
September
March 2016
29,2010
Fortune Example
• As quickly as you can do the following:
• 2+2+2=?
• 7+7+7=?
• What is the first VEGETABLE that
comes to mind?
– Tomatoes are a fruit……
– Did you say Carrot?
• 98% of normal people do!
ACRIN
TitleFall
of Presentation
Meeting – RA or
Session
Conference Here
27
September
March 2016
29,2010
Know your subjects
Divider (sub-section) Slide
Why are Cooperative Group statisticians
so hard to break?
Not beholden to the hypothesis
No vested interest in the outcome
Can make objective assessments
&recommendations to the DSMC
ACRIN Fall Meeting – RA Session
September 29,2010
Know your subjects
 Normal stressors
Opportunities for Mayhem!
What does an ACRIN Statistician Do?
Divider (sub-section) Slide
Study Design and Analysis Plans
Monitor Trial Progress
Aggregate Information
Report to Monitors (DSMC, NCI, CIP, etc.)
Data Analyses and Reports/Papers
ACRIN Fall Meeting – RA Session
September 29,2010
Know your subjects
Normal stressors
Tools of the trade
The soft Underbelly of Statistics
Hit ‘em Where it Hurts!
Divider (sub-section) Slide
Information
Information
Information
ACRIN Fall Meeting – RA Session
September 29,2010
Know your subjects
Normal stressors
Tools of the trade
 Control information Example
• A $50 Million a year company has entry level
Divider (sub-section) Slide
positions open for people willing to work their way
up. The company pays $30M in salary compensation
to its 150 employees. Therefore, the Average salary at
the company is $200,000/yr should you take a job?
Max = $19M
Median= $200K
Mean = $200K
ACRIN Fall Meeting – RA Session
Median= $10K
Mean = $200K
September 29,2010
Know your subjects
Normal stressors
Tools of the trade
 Control Information Example
 Effective Information Control
Use general termsDivider
like (sub-section)
average Slide
Never disclose your assumptions and bin your data
Need Blinding or other interesting study designs
Expect P-values but ignore power(until the end)
Refuse to accurately identify the Sample/Population
Vigorously deny any potential sources of Bias
Explore every possible hypothesis conceivable
ACRIN Fall Meeting – RA Session
September 29,2010
Examples:
Interesting Designs
P-values v Power
 Sample/Population
 Assumptions/Bias
Divider (sub-section) Slide
Conclusions As with many interventions intended to
prevent ill health, the effectiveness of parachutes has not
been subjected to rigorous evaluation by using randomized
controlled trials. Advocates of evidence based medicine
have criticized the adoption of interventions evaluated by
We think that
everyone might benefit if the most
radical protagonists of evidence
based medicine organized and
participated in a double blind,
randomized, placebo controlled,
crossover trial of the parachute.
using only observational data.
ACRIN Fall Meeting – RA Session
September 29,2010
Examples:
 Interesting Designs
 P-values v Power
 Sample/Population
 Assumptions/Bias
Power v P: measures of Probability not Pain
Divider (sub-section) Slide
Null Hypothesis: The effect we are trying to DISPROVE
Alpha (a): Probability of being WRONG!  Set a priori
(Falsely rejecting the Null Hypothesis)
Power
Probability of being right given the assumptions.
(Correctly rejecting the Null Hypothesis)
P-Values
A measure of evidence
under the NULL Hypothesis
ACRIN Fall Meeting – RA Session
September 29,2010
Examples:
 Interesting Designs
 P-values v Power
 Sample/Population
 Assumptions/Bias
Examples
Divider (sub-section) Slide
Were
gonna be rich!
Hypothesis: Roulette table pays off red 20% or more than black
Null Hypothesis: Black and Red occur equally (%R-%B<20%)
Alpha: reject at the a=5% level (I.e., P-Value <0.05)
80% Power: used to calculate sample size = 100
BAD
Unplanned looks at the data /Multiple looks at the data
-- If you look at the data 20 times you expect to discover
at least 1 false significant result
Result: 61 red of 100 cases (P>0.05)
"No one can possibly win at roulette unless he steals money from
the table while the croupier isn't looking."
— Albert Einstein
ACRIN Fall Meeting – RA Session
September 29,2010
Examples:
 Interesting Designs
 P-values v Power
 Sample/Population
 Assumptions/Bias
Population: The group you want to study (generalize)
(sub-section) Slide
Sample: The subjects Divider
you actually
study
Population: Americans meeting screening guidelines for CRC
Sample: 1600 Asymptomatic Americans >50 yrs old using
Protocol specified prep, technique, parameters, etc.
To maximize pain at analysis: Refuse to identify any abnormality
observed during accrual that might allow the statistician to subset or
re-categorize the data.
ACRIN Fall Meeting – RA Session
September 29,2010
Examples:
 Interesting Designs
 P-values v Power
 Sample/Population
 Assumptions/Bias
Birth Control is 99.9 effective when used according to directions
Divider (sub-section) Slide
•1 in 1000 fail but what
is the population they are talking about
Is it a single regimen?
Is it a single dose?
Is it per person (1 out of every 1000 users)?
Who are the Failures?
Is there something special about these cases?
How do you collect this special information
ACRIN Fall Meeting – RA Session
September 29,2010
Examples:
 Interesting Designs
 P-values v Power
 Sample/Population
 Assumptions/Bias
Assumptions may introduce bias into the study
(sub-section)aSlide
-Forms will alwaysDivider
be completed
certain way
-Procedures will always go as planned
-Technical data/Lab values are not significant
-A Yes/No response will be sufficient to answer the question
-All Bias is identifiable
Bias: a systematic error that may alter the outcome
-Approach only those patient you think will complete
-Only help some people complete forms
-Unblind readers to results
Beware: statisticians have ways to correct/report on some forms of bias if
they know about it – So never let them know about it until it is too late!
ACRIN Fall Meeting – RA Session
September 29,2010
Examples:
 Interesting Designs
 P-values v Power
 Sample/Population
 Assumptions/Bias
The Monty
MitchyHall
Schnall
Problem
Problem
Divider (sub-section) Slide
1
ACRIN Fall Meeting – RA Session
2
3
September 29,2010
Examples:
 Interesting Designs
 P-values v Power
 Sample/Population
 Assumptions/Bias
The Mitchy Schnall Problem
If the Host is biased Switch
Divider (sub-section) Slide
If the Host is unbiased the
probability after switching is 50%
1/3 Chance
2/3 Chance
1
2
3
1/3
1/3
1/3
ACRIN Fall Meeting – RA Session
September 29,2010
DON’T ASK, DON’T TELL
Divider (sub-section) Slide
Why are statisticians so secretive?
How do we get those secrets out of them?
ACRIN Fall Meeting – RA Session
September 29,2010
Don’t Ask:
Random Blinding
Feedback
• Reduce Bias byDivider (sub-section) Slide
– Randomization: choose people or treatments at
random (in a reproducible manner)
– Masking (Blinding): Do not let the patient know
their treatment assignment
– Double Masking: Don’t let anyone know the
treatment assignment
– Approach everyone in a
predefined systematic manner
ACRIN Fall Meeting – RA Session
September 29,2010
Don’t Ask:
Random Blinding
 Feedback
• Feedback loops (Divider
tell me(sub-section)
how I’mSlidedoing)
– Past events changes current events
• Data Mining/ Exploration/Hypothesis generation
– Collected data is explored for correlations/associations
– A Large number of “Chance” associations must be explored
• Hypothesis testing
– A Hypothesis is developed
– Data is collected to test the Hypothesis
ACRIN Fall Meeting – RA Session
September 29,2010
Don’t Ask:
Random Blinding
 Feedback
• Feedback loops or “Divider
Tell us
how good are
(sub-section) Slide
we doing!”
– Research is not Training it is hypothesis testing
Training and testing data must be kept separate.
– Defeats the purpose of masking
– Leads to uncorrectable bias and unstable performance
– Introduces unknown confounders into the analysis
• Data Mining or “Maybe
it was this effect!”
– At the a=0.05 level, we expect 5% of comparisons to have
an effect size that exceeds the threshold.
– Associations are not causal relationships
ACRIN Fall Meeting – RA Session
September 29,2010
Don’t Ask:
 Random Blinding
 Feedback
Data Mining
• DNA testing is 99.99%
accurate
Divider (sub-section) Slide
– It’s wrong in 1 out of 10,000
• If a DNA database has 20,000 individuals
– 86% chance of matching a random donor
• If a DNA database has 40,000 individuals
– 98% chance of matching a random donor
• Should we have a national DNA database?
ACRIN Fall Meeting – RA Session
September 29,2010
KISS (Keep it Simple, Stupid!)
Divider (sub-section) Slide
The biggest threat to the Primary Aim
of a study are the Secondary Aims.
After the key elements required for
analysis and monitoring, collect
whatever is easy and meaningful.
ACRIN Fall Meeting – RA Session
September 29,2010
Divider (sub-section) Slide
WHAT HAPPENS:
AFTER YOU HIT ENTER
Humans Vs Computers
Statisticians Vs Humans
Data Vs Information
Queries
ACRIN Fall Meeting – RA Session
September 29,2010
After you hit Enter:
Information Flow
 Garbage collection
 Goals
 Misinformation
Divider (sub-section) Slide
Data Entry
(DM/HQ)
Data Collection
Patient Level
CRAs
Central
Database
DM
Biostats
Analysis
Database
Web Based DE
(CRA/Sites)
ACRIN Fall Meeting – RA Session
September 29,2010
After you hit Enter:
 Information Flow
 Garbage collection
 Goals
 Misinformation
Lady
GIGO
Divider (sub-section) Slide
Garbage in, Garbage out
Computers process numbers
Humans interpret everything
“5” + “A”
Computer sees 53+65=118 (“v”?)
“5”+ “Bob” = ???
Minimize errors:
 Double-data entry
 Real time data queries
ACRIN Fall Meeting – RA Session
 Range/Logic checks
 Cross form validation
September 29,2010
After you hit Enter:
 Information Flow
 Garbage collection
 Goals
 Misinformation
Lady GIGO
(cont)
Divider (sub-section) Slide
Cooperative Garbage
Statisticians don’t like data from
cooperative/outside groups
Data form out side:




Different Aims

Not familiar with our study 
Different priorities

Inability to query in a
timely manner
 Long delays in getting data
ACRIN Fall Meeting – RA Session
No control of data
Hard to audit source
Hard to assign
responsibility for data
September 29,2010
After you hit Enter:
 Information Flow
 Garbage collection
 Goals
 Misinformation
•Statisticians
Divider (sub-section) Slide
Interact with Computers and Data
Aggregate Data
Don’t generally care about individual cases
Want all meaningful data available
• RAs/PIs/DM
Interact with People and Data
Focus on individual cases
Deal with individual data elements
ACRIN Fall Meeting – RA Session
September 29,2010
After you hit Enter:
 Information Flow
 Garbage collection
 Goals
 Misinformation
Nobody wants to manipulate data,
Divider (sub-section) Slide
so who corrects the data?
Current PHS Policies on Research Misconduct (42 CFR Parts 50 and 93)
define falsification as, "manipulating research materials, equipment, or
processes, or changing or omitting data or results such that the research is
not accurately represented in the research record.”
• RAs: Want data to be as accurate as possible so put
additional information on forms and in comments.
• DMs: Want database to accurately reflect what is on
the forms
• Stats: Want data that is meaningful and analyzable in
aggregate
ACRIN Fall Meeting – RA Session
September 29,2010
Analyze this
Let the experts show you how it’s done
Divider (sub-section) Slide
ACRIN Fall Meeting – RA Session
September 29,2010
Analyze this
Let the experts show you how it’s done
extensive proably invades nipple
has 2-3 ductal extensions+5mm satellite
1tiny multifocal nodule inf sag loc 79.3
Divider
(sub-section)
Slide
long ax actuallyAP~60mm
nr pectoral musc
22 mm 79/232
long dia = oblique cc,meas on AP/coronal
47 ML, MIP 107 Sag
many morphologic patterns
92/224
mass enhance pattern N/A, remove gradual
A/P MIP ML MIP, S-I MIP
E53=can't assess
ActuallySeroma not cyst, measured on MIP
add area enhancement w multi lob dom mas
don't use case for vol/ser
anterolat mass likely = more ca,
MIP measurements
lge ax. nodes; el 176=1
MIP-surrounded by fluid
area of enh.in mass ext. from it is stip
ML&APax87.9ser 8/14SIsgse7/14I33/66L90.1
area of lateral enhance.,indeterminate
multicentric spiculated masses;e93=severe
assoc. field effect superimposed on mass
Multifocality & area enhan leads to ty C
broad area 6:00, narrow at 12:00
None E93=severe
can't evaluate axilla-fatsat failed
not enough room for comments
dominant mass w/assoc.uncontained enhanc
post-surq changes in axilla
e174 = unable to assess
prior sentinal node biopsy
COMMENTS
E6=10
E93,94=unknown
E93=SEVERE
E93=severe
ACRIN Fall Meeting – RA Session
Pt is post-surg biopsy & ax. LN dissect
Q17=3
rim enhancing cyst upper inner quadrant
T2ax 19/39, 95/224
September 29,2010
How to Torture Your Statistician:
simple ways to maximize pain!
ACRIN Fall Meeting – RA Session
ACRIN
Fall
Meeting – RA or
Session
Title
of Presentation
Statistical
Presentation Conference Here
29 September 2010
September
29,2010
27
March 2016