Transcript Document

Author-perceived Quality
Characteristics of Science,
Technology and Medical (STM)
Journals
Dr. John J. Regazzi ([email protected])and
Selenay Aytac, MBA
Scholarly Communication Lab
College of Information and Computer Science
Long Island University, NY
METRO Science Technology Medical Librarians
Special Interest Group
October 12, 2007
Scope of the Research

The aim of this research study is to
explore author-perceived quality
characteristics of STM (Science,
Technology and Medicine) journals.
One of the most, if not the most,
important issue today facing
scholarly communication is what
constitutes quality in the publishing
and dissemination of research
findings.
LIU, CICS Scholarly
Communication Lab, October 12,
2007
2
Research Questions
(1) What quality attributes are most important
to authors for STM journals?
(2) What are some of the underlying differences
among researchers that might account for
certain journals to be called “prestigious” while
others are considered less valuable?
(3) What are the key factors considered in
submitting articles to STM journals by potential
authors?
(4) Which attributes might be more important in
the changing electronic publishing and webbased market?
LIU, CICS Scholarly
Communication Lab, October 12,
2007
3
Literature Review

16 attributes studied and linked to the literature:
1. Online tools
(Tibbitts, 2006; Rowlands and Nicholas, 2005; Mabe, 2003)
2. Society Pub.
(Chressanthis and Chressanthis, 1993; Gorman and Calvert, 2001; Mabe, 2003)
3. Editorial board
(Franke, 1990; Nisonger, 2002; Tibbitts, 2006; Rowlands and Nicholas, 2005; Mabe,
2003)
4. Previous experience(Mabe, 2003)
5. Impact factor
(Garfield, 1955; Yue and Wilson, 2007; Saha, 2003; Tibbitts, 2006; Rowlands and
Nicholas, 2005; Mabe, 03)
6. Reputation
(Frank, 1994; Tibbitts, 2006; Rowlands and Nicholas, 2005; Mabe, 2003)
7. Rejection rate
(Bjork and Holmstorm, 2006); Rowlands et al 2004)
8. Time to publication(Bjork and Holmstorm, 2006; Gleser, 1986; Tibbitts, 2006; Rowlands and Nicholas, 2005;
Mabe, 2003)
9. Price
(Bjork and Holmstorm, 2006; Chressanthis and Chressanthis, 1993; Rowlands et al,
2004))
10. Publisher
(Mabe, 2003)
11. Readership
(Franke and et all., 1990; Tibbitts, 2006; Rowlands and Nicholas, 2005)
12. Colleague recs. (Mabe, 2003)
13. Journal online
(King and et all, 2006)
14. Copyright
(Grimby, 2005; Rowlands et al, 2004))
15. Open access
(Schroter, 2005; Regazzi and Caliguiri, 2006; Rowlands et al, 2004))
16. Design
(Gorman and Calvert, 2001; Joseph, 2006; Erdman, 2006; Mabe, 2003)
LIU, CICS Scholarly
Communication Lab, October 12,
2007
4
Methodology



Data collected from the volunteer full-time faculty of
Long Island University.
Subjects were invited by mail. Letters of invitation to
the research study were mailed to the faculty members
of two colleges: (1) College of Information and
Computer Science (CICS), and (2) School of Health
Profession and Nursing Science (SHS). These 2 schools
were selected represent much of the publishing interest
of ALPSP members
Pre-test for the survey instruments was undertaken
among 3 subjects prior to the experiments
LIU, CICS Scholarly
Communication Lab, October 12,
2007
5
Methodology
3 Research Methods used:



(1) Survey: ranking importance of 16
attributes
(2) Focus groups and
(3) Semi structured face-to-face
interviews conducted for this study.
LIU, CICS Scholarly
Communication Lab, October 12,
2007
6
Data Collection


13 Surveys completed
2 Focus Group(s)



CICS Focus Group with 7 subjects
SHS Focus Group with 6 subjects
5 Face-to-face semi-structured
interview(s)


2 CICS Subjects
3 SHS Subjects
LIU, CICS Scholarly
Communication Lab, October 12,
2007
7
Survey: Prior to the each Focus
Group


Each subjects first asked for the key
demographics such as: (1) academic title,
(2) tenured or not, (3) number of
publications in last 7 years, and (4) gender.
And the subjects have been asked to circle
the number that represents how they would
rate the importance of the pre-defined 16
journal attributes (5 point Likert scale was
used to measure the perception)
LIU, CICS Scholarly
Communication Lab, October 12,
2007
8
Survey: 16 Pre-defined Quality
Attributes were Ranked
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
Availability of Online Manuscript Tools
Journal Published by a Society or Not-for-profit
Having an Editorial Board to Oversee the Journal
Previous Experience with the Journal
The Impact Factor of the Journal
The Reputation of the Journal
The Rejection Rate of the Journal
The Estimated Length of time to Article Publication
The Price of the Journal
The Specific Publisher of the Journal
The Readership of the Journal
Recommendations form Colleagues about the Journal
The Availability of the Journal Online
Copyright Restrictions on you
Open Access, Public Access and Web posting options and policies of
the Journal
Design: quality of images, typesetting, etc.
LIU, CICS Scholarly
Communication Lab, October 12,
2007
9
Focus Groups

Focus groups were divided into 3
broad areas of discussion:



a) the most important attributes
b)the least important attributes
c) other attributes that the group wish
to discuss
LIU, CICS Scholarly
Communication Lab, October 12,
2007
10
Face-to-Face Interviews


5 in-depth interviews were conducted the
following days to discuss these
aforementioned quality attributes with the
volunteer subjects from the focus group
Each interviewee was asked to describe
the process from the research to article
submission and when and why a journal
is considered.
LIU, CICS Scholarly
Communication Lab, October 12,
2007
11
Findings: CICS Survey
Descriptive Statistics
N
Onl ine M anuscri pt Tools
By a Soci ety/Non-Profi t
Editorial board oversee
Previous Experience
Impact Factor
Reputation
Rej ection rate
Est. lenght of time
The price
The specifi c Publisher
Readership
Recommendation
Availabi lity of online
Copyright restri ctions
Design: q uality of
images etc.
Open access, etc, web
posting pol ici es
Valid N (li stwise)
7
7
7
7
7
7
7
7
7
7
7
7
7
7
Mi ni mum
1
1
1
3
4
4
3
3
1
3
3
3
1
2
Maximum
4
5
5
5
5
5
4
5
4
4
5
5
5
5
Mean
2.71
3.00
3.71
3.86
4.29
4.57
3.29
4.43
3.29
3.57
4.29
4.14
4.00
3.86
Std. Deviation
1.254
1.414
1.380
.900
.488
.535
.488
.787
1.113
.535
.951
.690
1.414
1.069
7
2
5
3.71
.951
7
3
5
4.14
.690
7
LIU, CICS Scholarly
Communication Lab, October 12,
2007
12
CICS – Ranking by Importance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Reputation
4.57
T ime to publication 4.43
Impact factor
4.29
Readership
4.29
Colleague recs.
4.14
Open access
4.14
Journal online
4.00
Previous experience 3.86
Copyright
3.86
Editorial board
3.71
Design
3.71
Publisher
3.57
Rejection rate
3.29
Price
3.29
Society Pub.
3.00
CICS Scholarly
OnlineLIU,
tools
2.71
Communication Lab, October 12,
2007
13
Findings: SHS Survey
Descriptive Statistics
N
Onl ine M anuscri pt Tools
By a Soci ety/Non-Profi t
Editorial board oversee
Previous Experience
Impact Factor
Reputation
Rej ection rate
Est. lenght of time
The price
The specifi c Publisher
Readership
Recommendation
Availabi lity of online
Copyright restri ctions
Open access, etc, web
posting pol ici es
Design: q uality of
images etc.
Valid N (li stwise)
6
6
6
6
6
6
6
6
6
6
6
6
6
6
Mi ni mum
4
3
3
4
2
3
3
2
2
3
3
3
3
3
Maximum
5
5
5
5
5
5
5
5
5
4
5
5
5
5
Mean
4.33
4.00
4.17
4.50
4.00
4.50
4.00
4.33
3.17
3.50
4.67
4.33
4.33
4.17
Std. Deviation
.516
.632
.753
.548
1.095
.837
.632
1.211
.983
.548
.816
1.033
.816
.753
6
3
5
4.17
.983
6
3
5
4.17
.753
6
LIU, CICS Scholarly
Communication Lab, October 12,
2007
14
SHS – Ranking by Importance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Quality Attributes
Readership
Previous experience
Reputation
Colleague recs.
Journal online
Online tools
Time to publication
Copyright
Design
Editorial board
Open access
Impact factor
Rejection rate
Society Pub.
Publisher
Price
SHS Mean
4.67
4.50
4.50
4.33
4.33
4.33
4.33
4.17
4.17
4.17
4.17
4.00
4.00
4.00
3.50
3.17
LIU, CICS Scholarly
Communication Lab, October 12,
2007
15
Findings: Rankings by Group
and Attributes
Series 1 SHS and Series 2 CICS
5
Mean
4
3
2
1
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16
Atributes
LIU, CICS Scholarly
Communication Lab, October 12,
2007
16
Statistical Analysis

Are there a group differences?



Overall by Discipline (CICS vs. SHS)
Gender
Tenure (Tenured vs. not Tenured)
(Reliability of Survey Instrument: Highly
reliable - Chronbach’s alpha = .8)
LIU, CICS Scholarly
Communication Lab, October 12,
2007
17
Statistical Analysis– Overall
Differences by Discipline
Online Manuscript Tools
CICS
Mean 2.71
SHS
Mean 4.33
Rejection rate
CICS
Mean 3.29
SHS
Mean 4.00
“Online Manuscript Tools and Rejection Rate
Groups are significantly different at the p=.013
and p=.042 levels respectively.”
LIU, CICS Scholarly
Communication Lab, October 12,
2007
18
Statistical Analysis– Overall
Differences by Gender
Online Manuscript Tools
Male
Mean 2.60
Female Mean 4.00
Recommendations
Male
Mean 3.60
Female Mean 4.63
Society Publisher
Male
Mean 2.40
Female Mean 4.13
“Online Manuscript Tools and Recommendations
and Society Publisher are significantly different
at the p=.047 and p=.023, and p=.005 levels
respectively.”LIU, CICS Scholarly
Communication Lab, October 12,
2007
19
Statistical Analysis– Overall
Differences by Tenure vs. Non-Tenure
Copyright
Tenured
Non-Tenured
Mean 4.30
Mean 3.00
“Copyright restrictions is significantly
different at p=.023 level”
LIU, CICS Scholarly
Communication Lab, October 12,
2007
20
Content Analysis-Focus GroupsCoding


We had 16 pre-defined set of
concepts for these experiments,
however, some new attributes
emerged during the focus groups
and interviews as well
Each script was coded for implicit
and explicit terms for conceptual
analysis
LIU, CICS Scholarly
Communication Lab, October 12,
2007
21
Focus Group-Coded Data
FOCUS GROUP DATA
CICS
Readership
Estimated length of time
Reputation
Copyright restrictions
Recommendation
Online Manuscript Tools
Availability of Online Journal
Open access
Impact Factor
Editorial board oversee
By a Society/Non-Profit
Publisher Name
Design
Rejection rate
Price
Previous Experience
SHS
12
6
8
8
8
6
7
4
4
2
2
1
1
0
0
0
TOTAL
12
8
8
2
2
2
1
4
4
4
2
1
1
1
1
1
LIU, CICS Scholarly
Communication Lab, October 12,
2007
MEAN
24
17
16
10
10
8
8
8
8
6
4
2
2
1
1
1
12
8.5
8
5
5
4
4
4
4
3
2
1
1
0.5
0.5
0.5
22
Analysis: by Attribute
Importance and Awareness
Differatiating Journal Quality Attributes for Authors
1. Online tools
4. Previous experience
7. Rejection rate
10. Publisher
13. Journal online
16. Design
2. Society Pub.
5. Impact factor
8. Time to publication
11. Readership
14. Copyright rights
3. Editorial board
6. Reputation
9. Price
12. Colleague recs.
15. Open access
Emerging
Leading
11
6
Awareness
8
12
1
14
5
15
2
16
10
9
Undifferentiating
13
7
LIU, CICS Scholarly
Importance
Communication Lab, October 12,
2007
4
4
Prerequisities
23
Interviews
Key Questions:



When do you start thinking about
journals?
How many journals do you consider
submitting your article?
How do you think about journals at
different points of research process?
LIU, CICS Scholarly
Communication Lab, October 12,
2007
24
Interviews – Journal Identification
by Research Stage
Research
Idea
2-3
Journals(1)
1
Journal(2)
1-2
Journals(3)
Literature
Review
1
Journal(4)
Data
Collection
Write
Up
Submit
Article
2
Journals(5)
The 5 subjects interviewed
identified the stage of research
they begin to consider a journal,
and the number of journals they
are considering at that point.
LIU, CICS Scholarly
Communication Lab, October 12,
2007
25
Conclusion
1. Group differences by discipline (Computer Science vs. Allied
Health) – significant differences on the perceived value of online
tools and rejection rate, with SHS faculty rating these attributes
significantly higher than CICS faculty.
2. Gender differences– significant differences for recommendations
from colleagues, online manuscript tools, and society as
publisher, with female faculty rating these attributes higher than
male faculty.
3. Tenure vs. non-tenured differences– significant differences on
copyright, with tenured faculty rating the importance of this
attribute significantly higher than non tenured faculty.
4. The leading perceived attributes were: (a) the reputation of
the journal, (b) the estimated length of time to article
publication, and (c) the readership of the journal, as highly
leading; others were identified as important as “emerging”
issues/attributes: (d) recommendations from colleagues about
the journal, (e) copyright restrictions, and (f) open access.
LIU, CICS Scholarly
Communication Lab, October 12,
2007
26
Implications for the Future


Monitoring these attributes as more
articles become available on the Web,
through search engines, and with the
emerging new functionality of Web 2.0.
Further studies:




Larger sample size
Different disciplines
Deeper analysis of differences of group,
gender, and publications
Role of ‘social responsibility’ in the
development of journal brand equity
LIU, CICS Scholarly
Communication Lab, October 12,
2007
27