Transcript From biomedical research to health improvement

FROM BIOMEDICAL RESEARCH TO
HEALTH IMPROVEMENT
Grant Lewison
[email protected] and
[email protected]
evaluametrics
Evaluation of research through publication metrics
PUTTING BIOMEDICAL RESEARCH INTO PRACTICE
Citations in
Applications
Clinical
Patents
New drugs
trials
papers
for funds
Decisions
Papers
in
on funding
journals
Clinical
Better
guidelines
diagnosis
Better
clinical care
Medical
New
education
vaccines
Conduct of
Training of
Government
Public
health
research
researchers
policy
campaigns
Less
Newspaper
Economic
articles
Other docs,
books
Papers
Pollution
at
output
regulations
conferences
Physical
Popular
environment
opinions
Lifestyle
choices
evaluametrics
Evaluation of research through publication metrics
illness
incidence
OTHER CITATIONS: CLINICAL GUIDELINES - 2
• Example: cancer clinical guidelines (BJC, 98, 1944)
• Sample of 43 guidelines (up to 2006)
• Comparison with world cancer research, 1999-2001
because peak PY of cited refs is then:
400
350
No of references
300
250
200
150
100
50
0
1980
1985
1990
evaluametrics
Year
1995
2000
2005
Evaluation of research through publication metrics
OTHER CITATIONS: CLINICAL GUIDELINES - 3
Ratio of % of guideline refs to % of oncology research
• UK papers are over-cited by x3, but some other
countries’ papers are also over-cited. DE and JP much
under-cited:
3
2.5
2
1.5
1
0.5
0
UK DK AU IE
SE CA IT BE FR NL NO GR US
evaluametrics
FI CH ES AT DE PT JP
Evaluation of research through publication metrics
OTHER CITATIONS: CLINICAL GUIDELINES - 4
• Comparisons can also be made of impact of papers
from other countries, e.g., six Swedish medical schools:
40
% paps
Göteborg (GOT)
papers are much
better cited than
ones from
Karolinska (KAR),
probably because
the latter are more
basic and the
former more
clinical
% CCGs
% of all Sweden
30
20
10
0
GOT
LUN
evaluametrics
UPP
KAR
LIN
UME
Evaluation of research through publication metrics
OTHER CITATIONS: CLINICAL GUIDELINES - 5
What cancer sites are represented in papers cited by UK
guidelines, and do they correlate with disease burden?
20
Data on burden for 2004
from WHO. Some cancers
(e.g., lung - LUN, prostate PRO, pancreatic - PAN)
appear less among guideline
refs than would be expected,
and others (breast - MAM,
melanoma - MEL) appear
more.
MAM
15
% of refs on UK CCGs
COL
10
LUN
STO
MEL
5
LYM
LEU
CER
0
OVA
LIV
UTE
0
OES
BLA
PRO
MOU PAN
5
10
% of UK cancer DALYs
evaluametrics
15
20
Evaluation of research through publication metrics
OTHER CITATIONS: POLICY DOCUMENTS - 2
• Example of erpho report:
January 2006 Issue 16 ISSN 1477-7304
HIV and sexually transmitted
infections in the East of England
Report has 33 references of which 10 journal papers
• Example of report from No 10:
Strategy Unit Alcohol Harm Reduction project. Interim
Analytical Report Nov 2003
This report had over 200 references of which ~70
journal papers: mainly social sciences (NOT
biomedical), and heavily from UK
evaluametrics
Evaluation of research through publication metrics
OTHER CITATIONS: POLICY DOCUMENTS - 3
• UK Department of Health has many reports, e.g.:
Sport and Exercise Medicine
A Fresh Approach
published May 2012, includes
39 refs to journal papers
• some only have a few refs, e.g.:
Improving Outcomes: A Strategy
for Cancer January 2011
has 36 refs of which 6 are journal
papers, but this is a policy paper
signed by Secretary of State so has
more weight
evaluametrics
Evaluation of research through publication metrics
OTHER CITATIONS:TEXTBOOKS - 2
• Three individual chapters analysed, with FR, IT and
UK authors
• FR:The effect of cell-matrix interactions and aging
on the malignant process (2007; basic; RL = 3.4)
• IT: Lung cancer: does age affect treatment strategy?
(2009; clinical; RL = 1.0)
• UK: Supportive, Palliative and End-of-Life Care for
Older Breast Cancer Patients (2010; clinical; RL = 1.1)
• Examination of their references: age, national
distribution, citations
evaluametrics
Evaluation of research through publication metrics
OTHER CITATIONS:TEXTBOOKS - 3
Cumulative %s of papers older than given delay
100
Cumulative percent
80
Some references
cited by French
author are very
old (20% are 16+
years). Ones cited
by UK author are
the youngest –
but result may not
be typical
FR %
IT %
60
UK %
40
20
0
21
16
evaluametrics
11
Delay, years
6
1
Evaluation of research through publication metrics
OTHER CITATIONS:TEXTBOOKS - 4
Percent of cited references from different countries
50
FR
% of references
40
IT
UK
30
20
10
0
FR %
evaluametrics
IT %
Cited country
UK %
US %
Each citing author
gives preference to
own country’s
papers, but this is
most extreme for
France, which overcites by x 8, more
than UK (x 4) and
Italy (x3), relative to
presence in world
cancer research
Evaluation of research through publication metrics
OTHER CITATIONS:TEXTBOOKS - 5
• The references appear to be highly cited papers
• For the clinical papers cited by IT author, mean ACI
in 5 years = 83 cites (cf. world mean in cancer of 22)
100
So the cited
references seem
to be carefully
selected as highimpact papers
% of refs exceeding
80
60
40
20
0
2
5
10
evaluametrics
20
50
100
Cites in 5 yrs
200
500
1000
Evaluation of research through publication metrics
BBC WEBSITE COVERAGE OF CANCER RESEARCH
Story gives the
date, the name of
the journalist, the
institution
carrying out the
work and the
name of the
journal – often
also one of the
leading
researchers – so
that the cited
paper can be
identified easily
evaluametrics
Evaluation of research through publication metrics
COMPARISON OF BBC AND DISEASE BURDEN
Percent of BBC cancer research stories
100
Note log-log scales
Many cancers are
under-reported but
this may reflect the
paucity of research
papers. Note
relative popularity
of breast cancer
stories and those on
skin and cervical
cancer research
breast
10
skin
prostate
lung
colorectal
ovary
leukaemia
cervix
pancreas
bladder
lymphoma
stomach
mouth
oesophagus
1
liver
uterus
0.1
0.1
1
10
100
Percent of UK cancer DALYs (2002)
evaluametrics
Evaluation of research through publication metrics