Transcript New developments in electronic publishing and bibliometrics

New developments in electronic
publishing and bibliometrics
Henk F. Moed
CWTS, Leiden University, Netherlands
Elsevier, Amsterdam, Netherlands
Contents
1
Beyond journal impact factor and H-index
2
How useful are university rankings?
3
Why combine indicators and peer review?
4
Can indicators be manipulated?
5
Does Open Access lead to higher impact?
6
More downloads  more citations?
Contents
1
Beyond journal impact factor and H-index
2
How useful are university rankings?
3
Why combine indicators and peer review
4
Can indicators be manipulated?
5
Does Open Access lead to higher impact?
6
More downloads  more citations?
Journal impact measures
are no good predictors
of an individual paper’s
actual citation impact
Partly based on International Mathematical
Union’s Report ‘Citation Statistics’ (2008)
Length boys vs. adults
35
30
Boys (Mean
length=95 cm)
Players (Mean
length=185 cm)
% Persons
25
20
15
10
5
0
0
15 35 55 75 95 115 135 155 175 195 215
Length (cm)
Citations to P-AMS vs. T-AMS
80
70
60
% Papers
50
40
PAMS (JIF=0.43)
30
TAMS (JIF=0.85)
20
10
0
0
1
2
3
4
Nr Cites
5
6
7
8
Normal vs. skewed distributions
35
80
30
70
Boys (Mean
length=95 cm)
Players (Mean
length=185 cm)
60
50
% Papers
% Persons
25
20
15
40
PAMS (JIF=0.43)
30
10
TAMS (JIF=0.85)
20
5
10
0
0
0
15 35 55 75 95 115 135 155 175 195 215
Length (cm)
0
1
2
3
4
Nr Cites
5
6
7
8
What is the probability that .......
a randomly selected boy
is at least as tall as a
randomly selected player?
a randomly selected PAMS paper
is cited at least as often as a
randomly selected TAMS paper?
Almost zero
62 %
Probabilities still substantial for high JIF
journals
1.0
P-AMS/T-AMS
Case
0.8
PROB
0.6
0.4
MEAN
+- STD
0.2
0.0
0.0
1.0
2.0
3.0
4.0
Journal Impact Factor (midpoint)
5.0
6.0
‘Free’ citations
Thomson/JCR Journal Impact Factor
Citations to all docs
# Citable docs
Citable vs. non-citable docs
Citable documents
“non-citable” documents
Articles
Letters
Reviews
Editorials
Discussion papers
The problem of “free” citations - 1
Cites
+
Docs
+
+
+
+
+
+
+
+
+
The problem of “free” citations - 2
“Free”
Citations
Cites
Docs
+
+
+
+
+
+
+
All three publication lists have a Hirsch Index of 5
1
2
3
4
5
6
7
8
9
Author 1
Author 2
Author 3
30
10
8
6
5
1
0
30
10
8
6
5
4
4
4
4
100
70
8
6
5
1
0
P1
P2
P3
P4
P5
P6
P7
H=? 5
P1
P2
P3
P4
P5
P6
P7
P8
P9
H=? 5
P1
P2
P3
P4
P5
P6
P7
H=? 5
Different bibliometric distributions
have the same H-Index
Indicators are becoming more informative
Feature
Example
Embody ways to put
numbers in context
Field-normalized citation measures
Take into account
“who” is citing
Citations weighted with impact of
citing source
Take into account
Impact outside the own niche;
relationship citing-cited multi-disciplinarity;
author
bridging paradigms
Contents
1
Beyond journal impact factor and H-index
2
How useful are university rankings?
3
Why combine indicators and peer review
4
Can indicators be manipulated?
5
Does Open Access lead to higher impact?
6
More downloads  more citations?
University ranking positions are
primarily marketing tools,
not research management tools
Research assessment methodologies must take
into account… [EC AUBR Expert Group]
1. Inclusive definition of research / output
2. Different types of research and its impacts
3. Differences among research fields
4. Type and mission of institution
5. Proper units of assessment
6. Policy context, purpose and user needs
7. The European dimension
8. Need to be valid, fair and practically feasible
Types of outputs (SSH)
Impacts
Publication/text
Non-publication
Scientificscholarly
Journal paper; book Research data file;
chapter; monograph video of experiment
Educational
Teaching course
book; syllabus
Skilled researchers
Economic
Patent
Product; process;
device; design; image
Cultural
Newspaper article;
Interviews; events;
Performances; exhibits
Top-down institutional analysis
Select an institution’s papers using
author addresses (incl. verification)
Categorize articles into
research fields
Calculate indicators
Compare with benchmarks
Bottom-up institutional analysis (CWTS)
Compile a list of researchers
Compile a list of publications per
researcher (incl. verification)
Aggregate researchers into groups,
departments, fields, etc.
Calculate indicators;
compare with benchmarks
Secondary analyses of ‘ranking’
data are informative
Contents
1
Beyond journal impact factor and H-index
2
How useful are university rankings?
3
Why combine indicators and peer review
4
Can indicators be manipulated?
5
Does Open Access lead to higher impact?
6
More downloads  more citations?
Case study: A national Research Council
• Proposals evaluated by committees covering
a discipline
• Reports from external referees
• Committee members can be applicants
Affinity applicants – Committee
0
Applicants are/were not member of any
Committee
1
Co-applicant is/was member of a Committee,
but not of the one evaluating
2
First applicant is/was member of a Committee,
but not of the one evaluating
3
Co-applicant is member of the Committee(s)
evaluating the proposal
4
First applicant is member of the Committee(s)
evaluating the proposal
For 15 % of applications an applicant is a member of the
evaluating Committee (Affinity=3, 4)
70
% APPLICATIONS
60
50
40
30
20
10
0
Projects
0
1
2
3
4
63.2
10.2
11.5
5.9
9.1
AFFINITY APPLICANTS-COMMITTEE
Probability to be granted increases with
increasing affinity applicants-Committee
% GRANTED APPLICATONS
80
70
60
50
40
30
Projects
0
1
2
3
4
37.0
46.9
60.1
62.6
74.0
AFFINITY APPLICANTS-COMMITTEE
Logistic regression analysis:
Affinity Applicant-Committee has a significant effect
upon the probability to be granted
MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499)
Source
DF
Chi-Square
Prob
------------------------------------------------------------INTERCEPT
1
18.47
0.0000
Publ Impact applicant
3
26.97
0.0000 **
Rel transdisc impact applicant
1
0.29
0.5926
Affinity applicant-Committee
2
112.50
0.0000 **
Sum requested
1
45.47
0.0000 **
Institution applicant
4
25.94
0.0000 **
LIKELIHOOD RATIO
199
230.23
0.0638
The future of research assessment
exercises lies in the intelligent
combination of
metrics and peer review
Contents
1
Beyond journal impact factor and H-index
2
How useful are university rankings?
3
Why combine indicators and peer review
4
Can indicators be manipulated?
5
Does Open Access lead to higher impact?
6
More downloads  more citations?
Effects of editorial self-citations upon journal
impact factors
[Reedijk & Moed, J. Doc., 2008]
•
Editorial self-citations: A journal editor cites in his
editorials papers published in his own journal
•
Focus on ‘consequences’ rather than ‘motives’
Case: ISI/JCR Impact Factor of a Gerontology Journal
(published in the journal itself)
5.0
4.5
CITES PER 'CITABLE' DOC
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
2000
2001
2002
IMPACT FACTOR YEAR
2003
2004
Decomposition of the IF of a Gerontology journal
5.0
4.5
CITES PER 'CITABLE' DOC
4.0
Editorial self citations
Free citations
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
2000
2001
2002
IMPACT FACTOR YEAR
2003
2004
One can identify and correct for the following types of
strategic editorial behavior
•
•
•
•
Publish ‘non-citable’ items
Publish more reviews
Publish ‘top’ papers in January
Publish ‘topical’ papers (with high short term
impact)
• Cite your journal in your own editorials
• Excessive journal self-citing
Contents
1
Beyond journal impact factor and H-index
2
How useful are university rankings?
3
Why combine indicators and peer review
4
Can indicators be manipulated?
5
Does Open Access lead to higher impact?
6
More downloads  more citations?
Journal articles
Deposited
in OA rep
Not deposited
in OA rep.
(o)
(no)
Average
Impact
(CPPo)
?
>
<
=
Average
Impact
(CPPno)
Three effects
[Kurtz et al., 2005]
Open Access ArXiv increases access
Early View
Articles appear earlier in ArXiv
than in Publisher’s Website
Self-Selection Better authors use ArXiv
(Quality bias)
Authors deposit their best
papers in arXiv
ArXiv, Cond Mat Phys
[Moed, JASIST 2007]
ArXiv papers
appear earlier
OA Impact advantage
100
Early
View
Effect
Quality Bias: Better authors use ArXiv
0
96
97
98
99
00
01
Publication Years
02
03
04
05
Age distribution of citations to Arxiv and non-ArXiv papers
0.10
in ArXiv-CM
0.09
Not in ArXiv-CM
0.08
3 per. Mov. Avg. (in
ArXiv-CM)
3 per. Mov. Avg. (Not
in ArXiv-CM)
Cites per Paper
0.07
0.06
0.05
0.04
0.03
Move curve
by 6 months
to the right
0.02
0.01
0.00
0
6
12
18
24
30
36
42
48
54
Months after Publication Date
60
66
72
78
84
Early view effect: Citations to papers deposited in
ArXiv-CM start about 6 months earlier
0.10
in ArXiv 6 months
translated
0.09
Not in ArXiv-CM
0.08
3 per. Mov. Avg. (in
ArXiv 6 months
translated)
3 per. Mov. Avg. (Not
in ArXiv-CM)
Cites per Paper
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0.00
0
6
12
18
24
30
36
42
48
54
Months after Publication Date
60
66
72
78
84
More research questions
• Early view effect also visible in a non-OA
environment?
• Citation impact measured in biased sample?
Contents
1
Beyond journal impact factor and H-index
2
How useful are university rankings?
3
Why combine indicators and peer review
4
Can indicators be manipulated?
5
Does Open Access lead to higher impact?
6
More downloads  more citations?
Downloads vs. Citations
More downloads
more citations
or
More citations
more downloads?
Relation between citations and internet hits for 153
papers in volume 318 of the BMJ (1999)
Figure 1 from: Perneger, TV. BMJ. 2004, 329 (7465): 546–547. Relation between online “hit
counts” and subsequent citations: prospective study of research papers in the BMJ
Analogy Model
Formal use (e.g., SCI)
Informal use (e.g.,SD)
(Collections of)
publishing authors
(Collections of) users
Citing a document
Retrieving the full text
of a document
User session
Article
Author’s institutional User’s account name
affiliation
Number of times cited Number of times
retrieved as full text
Age distribution downloads vs. citations
[Tetrahedron Lett, ScienceDirect; Moed, JASIST, 2005]
20
SD USES
CITATIONS
Downloads
16
12
%
%
Citations
8
4
AGE (MONTHS)
Age (months)
29
27
25
23
21
19
17
15
13
11
9
7
5
3
1
0
Ageing downloads vs. citations:
Two factor vs. single factor model
100
Downloads
Observed
Downloads
Uses
10
%
Downloads
Computed
Downloads
Singular
Points
1
Citations
Observed
0.1
Citations
0.01
0
4
8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88
Age (months)
Age (months)
Citations
Computed
Citations lead to downloads
[Moed, J. Am Soc Inf Sci Techn, 2005]
DOWNLOADS
1000
Paper B
published;
it cites A
A
Paper
C
B (B cites A)
published;
itCcites
(C citesA
A and
and B)B
100
Paper
10
A
published
Download of A
increases
1
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20
AGE PAPER A (MONTHS)
Rank correlations between downloads and
citations
Variables
Spearman
R
Downloads vs. citations
0.22
‘Later’ (>3 months) downloads
vs. citations
0.33
‘Initial’ (<3 months) downloads
vs. citations
0.11
Conclusions
• Positive correlation between downloads and
citations partly due to the effect of citations
upon downloads
• ‘Initial’ downloads and citations hardly correlate,
and relate to distinct phases in processing
relevant scientific information
• ‘Later’ downloads and citations show statistically
similar properties of ageing and frequency
distribution
Downloads and citations
relate to distinct phases in
scientific information processing
.... but (many) more cases must
be studied
Thank you for your
attention!
References
•
•
•
•
•
•
•
•
•
•
•
•
Eysenbach, G. (2006). Citation Advantage of Open Access Articles. PLOS Biology, 4, 692–698.
Garfield, E. (1972). Citation Analysis as a tool in journal evaluation. Science, 178, 471–479.
Garfield, E. (1979). Citation Indexing. New York: Wiley.
Yassine Gargouri, Chawki Hajjem, Vincent Lariviere, Yves Gingras, Les Carr, Tim Brody, Stevan Harnad
(2010). Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research
arXiv:1001.0361v2 [cs.CY]
Harnad, S., & Brody, T. (2004). Comparing the impact of open access (OA) vs. Non-OA articles in the same
journals. D-Lib Magazine, 10, Nr 6.
Kurtz, M.J., Eichhorn, G., Accomazzi, A., Grant, C., Demleitner, M., Henneken, E., & Murray, S.S. (2005). The
effect of use and access on citations. Information Processing & Management, 41, 1395–1402.
Moed, H.F. (2005). Citation Analysis in Research Evaluation. Dordrecht (Netherlands): Springer. ISBN 14020-3713-9, 346 pp.
Moed, H.F., Glänzel, W., and Schmoch, U. (2004) (eds.). Handbook of Quantitative Science and Technology
Research. The Use of Publication and Patent Statistics in Studies of S&T Systems. Dordrecht (the
Netherlands): Kluwer Academic Publishers, 800 pp.
Moed, H.F. (2005). Statistical relationships between downloads and citations at the level of individual
documents within a single journal. Journal of the American Society for Information Science and Technology
56, 1088-1097.
Moed, H.F. (2007). The effect of “Open Access” upon citation impact: An analysis of ArXiv’s Condensed
Matter Section. Journal of the American Society for Information Science and Technology 58, 2047-2054.
Moed, H.F. (2009). New developments in the user of citation analysis in research evaluation. Archivum
Immunologiae et Therapiae Experimentalis (Warszawa) 17, 13-18.
Reedijk, J., Moed, H.F. (2008). Is the impact of journal impact factors decreasing? Journal of
Documentation 64, 183-192.
ISI/JCR Journal Impact Factor
of journal J for year T
Citations in year T to items published in J in
years T-1 and T-2
÷
Number of “citable” items published in J in
years T-1 and T-2