Transcript (+) studies to traditional journals

Peer review After Results are Known
Are we “PARKing” the Cart Before the Horse?
Erick Turner, M.D.
Associate Professor, OHSU Dept. of Psychiatry
Lead Investigator, Reporting Bias, Scientific Resource Center, AHRQ
Staff Psychiatrist, VA Portland Health Care System
Former FDA Medical Officer
A Day at the (Horse) Races
•
•
•
•
•
•
•
7 horses
You pick horse #1
...and they’re off!
Winner is...#2
Window teller asks: “Which horse did you bet on?”
You respond, “Er, #2, of course!”
Collect your winnings!
• What’s wrong with this picture?
The Academic Publication Rat Race
• Conceive study
• Write protocol (for your eyes only)
– Hypothesis  primary outcome
– Lots of 2° scales & analytic methods
• The more, the merrier!
• Collect data
• Analyze data as prespecified
• Oops! P> 0.05?
– Try out alternate methods p.r.n.
– Torture data until it confesses
• Then, and only then…
Write it up
or not!
• If you choose to (nonpublication)
• When you choose to (delayed publication)
• How you choose to (outcome reporting bias =
HARKing)
Hypothesizing
After the
Results are
Known
* Kerr NL. HARKing: Hypothesizing After the Results are Known.
Personality and Social Psychology Review. 1998 1998;2(3):196-217.
Our study in NEJM
5
Overview of our study
N Engl J Med. 2008 Jan 17;358(3):252-60
• 12 antidepressants starting with fluoxetine
(Prozac)
• Obtain drug approval packages
• Identify all premarketing trials (k=74)
• Track each study into published literature
• Two questions:
1- Was the study published?
2- If published, was it HARKed / spun?
6
Journal version of antidepressant trials
FDA version – same trials
What became of all those negative studies?
Admitted to a
negative finding
(sort of)
Unpublished
“Spun” from
negative to
positive
The 11 pigs with lipstick: ‘Enhanced’ P values...
Journal / Year
Journal
FDA
<.05
.316
≤ .05
0.018
≤ .05
.084
≤.01
.210
<.10
.50(-)
.045
.253
.014
.054
Human Psychopharm 1995
.015
.19
Int Clin Psychopharm 1992
J Clin Psychiat 1996
J Clin Psychiat 2002(a)
J Clin Psychiat 2002(b)
Psychopharm Bull 1990
<.05
<.01
.009
0.0004
<.05
not reported
≤.05
.224
.35
.146
.254
.25
.193
.142
Acta Psychiatr Scand 1992
Biol Psychiat 1995
Curr Ther Res 1986
Eur Psychiat 2006
Psychopharm Bull 1993
10
The experts weren’t fooled, were they?
Drug
First
Author
Year
Journal
Journal
impact
factor
Citing
articles
Citing SRs
and MAs
Nyth
1992
Acta Psychiatr Scand
4.9
197
13
Montgomery
1992
Int Clin Psychopharm
2.7
41
5
Goldstein
2002
J Clin Psychiat
5.8
243
23
Perahia
2006
Eur Psychiat
3.3
77
21
Rickels
1986
Curr Ther Res
0.4
11
8
mirtazapine
Khan MC
1995
Human Psychopharm
2.1
23
3
nefazodone
Cohn
1996
J Clin Psychiat
5.8
61
9
paroxetine CR
Golden
2002
J Clin Psychiat
5.8
68
9
paroxetine IR
Cohn
1990
Psychopharm Bull
n/a
20
7
sertraline
Fabre
1995
Biol Psychiat
9.2
136
12
Mendels
1993
Psychopharm Bull
n/a
96
11
Median
4.9
68
9
citalopram
duloxetine
fluoxetine
venlafaxine IR
HARKing / outcome reporting bias
It’s so easy
The Texas Sharpshooter Fallacy
12
13
Knowing results
biases journal reviewers, too
• Emerson et al. Arch Intern Med (2010) vol. 170 (21)
pp. 1934-9
• Phony manuscripts circulated for review
• Methodological errors sprinkled in
• Two versions of results
– Significant  errors missed  methods fine
– Nonsignif  errors caught  methods flawed
What can we do about this?
• De-emphasize results
• Emphasize the scientific process itself
– Scientific question
– Methodological rigor
• Time point of interest:
– Study completion  study inception
Methods & results
based on study inception
• Registration (esp. ClinicalTrials.gov)
– Study existence
– 1° vs. 2° vs. post-hoc outcomes / analytic methods
– Easily checked by reviewers (seldom done)
• Results
– Pharma studies
• Drugs@FDA
– How-to article: Turner EH, BMJ 2013
• ClinicalTrials.gov, results since 2008
NIH-sponsored trials
Soon subject to same rules
Instead of full manuscripts, review
protocols (sans results)
• Protocol ≠ methods section (written ARK)
• After study is completed
– Preliminary decision based on quality of science
– Why don’t more traditional journals require protocols?
• Before study inception
– Like FDA, which reviews x 2, before AND after
– Impossible to prejudge study based on results
– Being applied to academic enterprise...
Journals offering Registered Reports:
RR
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
AIMS Neuroscience
Attention, Perception, and Psychophysics
Cognition and Emotion
Comparative Political Studies*
Comprehensive Results in Social Psychology
Cortex
Drug and Alcohol Dependence
eLife -- Reproducibility Project: Cancer Biology*
Experimental Psychology
Journal of Accounting Research
Journal of Business and Psychology
Journal of Media Psychology
Nutrition and Food Science Journal
Perspectives on Psychological Science
Royal Society Open Science
Social Psychology
Working, Aging and Retirement
Frontiers in Cognition*
Journals offering similar pre-registered formats
•
•
•
Communication Methods and Measures
Journal of Experimental Psychology: General
Journal of Experimental Social Psychology
In pipeline at 10+ other journals
*special issue
Thank you
Supplemental slides
Who’s responsible?
Industry vs. academia
• Pharmaceutical company-sponsored research
– They’re corporations…Duh!
• Increase drug sales  serve shareholders
– Their academic KOLs (aka “thought leaders)
• Consulting, speaking, authorship
• $$, prestige
• Government-sponsored research (Not just industry!)
– NIH-sponsored trials
• Dickersin & Min, Online J Curr Clin Trials. 1993 Apr 28;Doc No 50
– Canadian Institutes of Health Research (CIHR)
• Protocols compared with pubs for 48 trials funded by CIHR, 1990-1998
• Chan, et. al. CMAJ. 2004 Sep 28;171(7):735-40.
22
Ef f ect Size Infl at ion in " Spun" Journal Art icles
-0 .5
-0 .2
0
0 .2
0 .5
0 .8
0 .2 9
paroxet ine CR ( 4 4 8 & 4 4 9 )
0 .3 5
0 .2 3
venlaf axine IR ( 6 0 0 A-3 1 3 )
0 .3 1
0 .2 2
duloxet ine ( HMAY-B)
11
‘enhanced’
effect sizes
0 .3 2
0 .2 7
sert raline ( 1 0 3 )
0 .3 8
0 .2 2
cit alopram ( 8 9 3 0 3 )
0 .4 0
0 .2 6
0 .4 7
paroxet ine IR ( 0 3 -0 0 2 )
0 .2 7
duloxet ine ( HMAQ-A)
0 .5 0
0 .1 7
cit alopram ( 8 6 1 4 1 )
0 .4 3
0 .2 3
mirt azapine ( 8 5 0 2 7 )
0 .6 7
0 .1 5
nef azodone ( CN1 0 4 -0 0 6 )
0 .6 2
-0 .2 1
fl uoxet ine ( 2 5 )
0 .5 2
0 .2 3
Overall
0 .4 0
Instead of full manuscripts, review
protocols (sans results)
• After study is completed
– Preliminary decision based on quality of science
– Problem: investigators know results 
• (+) studies to traditional journals
• (-) results to journals using this method
– Why don’t more traditional journals require protocols?
• Before study inception
– Like FDA, which reviews x 2, before AND after
– No results with which to prejudge study
– Can this be applied to the academic enterprise?