Transcript From Data to Assessment: Collecting

FROM DATA TO ASSESSMENT
Collecting, Presenting, and
Using Statistics
Jim Self
University of Virginia
VIVA Stats Workshop
February 2, 2010
THIS MORNING’S PROGRAM
A
few basic statistical concepts
 Displaying the data
 Examples from U.Va. and elsewhere
 Moving toward assessment
 Questions, comments
WHY MEASURE OR COUNT?
 To
evaluate
 To compare
 To improve
WHEN NOT TO COLLECT DATA
 When
someone has already done it
 When the data will not be useful
 When the results do not justify the costs
MAKING DATA MEANINGFUL
 Summarize
 Compare
 Analyze
 Present
MEASURES OF CENTRAL TENDENCY
 Median
 Mean
 Mode
 Midrange
MEASURES OF DISPERSION
 Range
 Variance
 Standard
Deviation
 Interquartile Range
SAMPLING
 Selection


Strive for an unbiased sample
Unambiguous, pre-set selection process
 Sample


of members is crucial
size
Large enough for appropriate precision
Small enough to be economical
DETERMINING SAMPLE SIZE
 Confidence
 Level
Interval
of confidence
U.VA. STUDENT SURVEY
 Question:
U.Va?

First Year Students


Do you own a computer at
Yes 99.0%
Graduate Students

 Do
Yes 88.2%
we know if these two groups are
different in terms of computer
ownership?
95% LEVEL OF CONFIDENCE
Population
Size
Sample
Size
Percent in
Sample
Answering
“Yes”
Plus or
Minus
Sample
Error
Grads 5,400
443
88.2%
2.9% 85.3% 91.1%
1st
Years
102
99.0%
1.9% 97.1% 99.9+%
2,900
Min.
Value
Max.
Value
PREPARATION FOR
DATA COLLECTION
Clarify
your objectives
Prefer the simplest procedures
Choose the appropriate technique
Collect only data you can use
FOUR USEFUL ASSUMPTIONS
 Your
problem is not unique.
 You
have more data than you think.
 You
need less data than you think.
 There
is a useful measurement that is much
simpler than you think.

Hubbard, D. (2007) “How to Measure Anything.”
PROCESSING THE DATA
 Check
and re-check the numbers
 Summarize
 Compare
 Find the interesting points
 Look for explanations
COMMON PITFALLS
 Sampling
with bias
 Surveys with a message
 Graphs with distorted figures
 Graphs with varying time frames
 Atypical projections
 Inappropriate grouping of data
 False precision
 Confusion of correlation and causality
“…BUT TO SUPPOSE THAT THE
FACTS, ONCE ESTABLISHED IN ALL
THEIR FULLNESS, WILL ‘SPEAK FOR
THEMSELVES’ IS AN ILLUSION.”
Carl Becker
Annual Address of the President of the American
Historical Association, 1931
DATA REPORTING
 Integrate
data, graphs, and narrative to tell
a succinct and accurate story.
 Show the data
 Give the source of the data
 Do not decorate graphics, let the data speak
 Put your name on the report
UVA LIBRARY SURVEYS
 Faculty


1993, 1996, 2000, 2004
Response rates 59% to 70%
 Students



1994, 1998, 2001, 2005
Separate analysis for grads and undergrads
Response rates 43% to 63%
 LibQual+

2006
Response rates 14% to 24%
 Annual
Surveys 2008+
Student samples
 One third of faculty
 Response rates 29% to 47%

FINAL QUESTION
U.VA. LIBRARY SURVEY
Please rate your overall satisfaction with the library.
Not at all
Satisfied
1
Very
Satisfied
2
3
4
5
No
Opinion
X
2000 U.VA. FACULTY SURVEY
OVERALL RATING
1
2
3
4
5
X
0
5
19
126
154
11
U.VA. FACULTY SURVEYS
OVERALL RATING
1
2
3
4
5
X
Total
1993
1
5
52
174
93
24
349
1996
1
8
36
146
136
12
339
2000
0
5
19
126
154
11
315
OVERALL SATISFACTION
U.VA. LIBRARY SURVEYS
Category
Mean
1993
Faculty
4.09
1994
Undergraduates
4.01
Graduate Students
3.96
1996
Faculty
4.26
1998
Undergraduates
3.90
Graduate Students
3.97
2000
Faculty
4.41
2001
Undergraduates
4.07
Graduate Students
4. 02
2004
Faculty
4.20
2005
Undergraduates
3.94
Graduate Students
4.02
OVERALL SATISFACTION
UNIVERSITY OF VIRGINIA LIBRARY
1993/94 1996/98 2000/01
2004/05
2008/09
Faculty
4.09
4.26
4.41
4.20
4.23
Graduate
Students
3.96
3.97
3.97
4.02
4.17
Undergrads
4.01
3.90
3.97
3.94
4.12
OVERALL SATISFACTION
UNIVERSITY OF VIRGINIA LIBRARY
1993-2009
4.60
4.40
Faculty
4.20
Grads
4.09
4.00
4.23
4.17
4.12
Undergrads
4.01
3.96
3.80
1993/94
1996/98
U.Va. Library Surveys 1993-2009
2000/01
2004/05
http://www2.lib.virginia.edu/mis/reports/
2008/09
U.VA. FACULTY AND STUDENTS
WHO PHYSICALLY VISIT A LIBRARY
AT LEAST ONCE A WEEK
100%
84%
80%
Undergrads
79%
72%
69%
60%
Grads
43%
40%
Faculty
26%
20%
0%
1993/94
1996/98
2000/01
2004/05
2008/09
DATA PRESENTATION ACCORDING TO
DILBERT
THE LIBRARY BUDGET AS PIE
Sources of Funding FY09
SR,SS
0%
DR
2%
EI, DI
4%
EF, ER, EU
9%
FA
2%
GF
1%
LG
2% LS
1%
SE
2%
SG
77%
CORROBORATION
 Data
are more credible if they are
supported by other information
 John Le Carre’s two proofs
ANALYZING SURVEY RESULTS
 Two


Scores for Resources, Services, Facilities
Satisfaction = Mean Rating (1 to 5)
Visibility = Percentage Answering the Question
 Permits
comparison over time and among groups
 Identifies areas that need more attention
UVA REFERENCE ACTIVITY AND
REFERENCE VISIBILITY IN STUDENT SURVEYS
7,000
6,008
Reference Questions
Recorded per Week
in Annual Sample
64%
Visibility
39%
Visibility
34%
Visibility
Reference Visibility
among Undergraduate
75%
Visibililty
1,756
10%
1,000
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
COLLECTING AND USING DATA
AT THE U.VA. LIBRARY
Customer Surveys
 Staff Surveys
 Mining Existing Records
 Comparisons with peers
 Qualitative techniques
 Statistical compilations
 Revealing the budget process and issues

DATA MINING
 Acquisitions
 Circulation
 Finance
 University
www.arl.org
Records
COLLECTIONS EXPENDITURES BY FORMAT
University of Virginia Library
$4,000,000
Electronic Resources
Print Monographs
$3,500,000
Print Serials
Manuscripts
Miscellaneous
$3,000,000
Microforms
Expenditures
Video or Film
$2,500,000
$2,000,000
$1,500,000
$1,000,000
$500,000
$0
Fiscal Year
USAGE DATA AT UVA – CIRC, REF, E-JRNLS
2,500,000
Circulation
Reference
2,000,000
1,500,000
1,000,000
500,000
0
E-Journal Articles
INVESTMENT AND CUSTOMER ACTIVITY
UNIVERSITY OF VIRGINIA LIBRARY
1993-2006
Acquisitions Expenditures by Format
Customer Activities
$3,000,000
2,400,000
1,800,000
$2,000,000
E-Journals
Circulation
Reference
1,200,000
$1,000,000
600,000
Electronic Resources
Print Monographs
Print Serials
$0
FY93
FY94
FY95
FY96
FY97
FY98
FY99
FY00
FY01
FY02
FY03
FY04
FY05
FY06
0
FY93
FY94
FY95
FY96
FY97
FY98
FY99
FY00
FY01
FY02
FY03
FY04
FY05
FY06
TOTAL EXPENDITURES AT UVA
1989—2003
300%
% Change since 1989
Research (+219%)
Other Academic Support
(+200%)
Total Academic Division
(+140%)
Libraries (+81%)
Instruction (+80%)
19
89
19
90
19
91
19
92
19
93
19
94
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
-50%
Fiscal Year
LOOKING AT LIBQUAL+
 22


core questions
1-9 scale
Ratings of minimum, desired, perceived
 Locally
selected questions
 General satisfaction ratings
SELECTING AND DISPLAYING
LIBQUAL+
 Choose

the meaningful data
Ignore the chaff
 Summarize
those data
 Compare



By category and subgroups
Over time
With peers
 Present
findings transparently
LibQUAL+ 2006
ARL Composite Faculty
9
8
7
6
5
AS-1
AS-2
AS-3 AS-4
AS-5
AS-6
AS-7
AS-8
AS-9
IC-1
IC-2
IC-3
IC-4
IC-5
IC-6
IC-7
IC-8
LP-1
LP-2
LP-3
LP-4
LP-5
LibQUAL+ 2006
University of Virginia Faculty
9
8
7
6
5
AS-1
AS-2
AS-3
AS-4
AS-5
AS-6
AS-7
AS-8
AS-9
IC-1
IC-2
IC-3
IC-4
IC-5
IC-6
IC-7
IC-8
LP-1
LP-2
LP-3
LP-4
LP-5
LibQUAL+ 2006
Ratings of AS-3:
Employees who are consistently courteous
9
Library Staff
Faculty
Grads
Undergrads
8
7
6
5
Faculty AS-3
Grad AS-3
Undergrad AS-3
Library Staff AS-3
USING DATA AT UVA
 Additional
resources for the science libraries (1994+)
 Redefinition of collection development (1996)
 Initiative to improve shelving (1999)
 Clemons Library open 24 hours (2000)
 Additional resources for the Fine Arts Library (2000)
 Support for transition from print to e-journals (2004)
 New and improved study space (2005-06)
 Increased appreciation of the role of journals (2007)
 Re-design of main floor of Clemons Library (2008)
USING DATA:
FROM STATS TO ASSESSMENT
 Counting
Traditional statistics emphasize inputs:
expenditures, acquisitions, holdings
 Assessing
 More recent efforts assess outputs:
service quality

WHY ASSESS?
 Improve
services
 Justify resources
 Compare with others
 Identify changing needs
 Identify questionable services
 Promote services
PARALLEL TRENDS
Customer-Centered
Library
All services and
activities are viewed
through the eyes of
the customers
 Customers determine
quality
 Library services and
resources add value
to the customer

Culture of
Assessment
Organizational
environment in which
decisions are based
on facts, research
and analysis.
 Services are planned
and delivered to
maximize positive
customer outcomes

EFFECTIVE ASSESSMENT
 Is
customer-focused
 Uses representative groups of users
 Asks fair and unbiased queries
 Develops criteria for success
 Uses multiple assessment methods
 Uses corroboration from other sources
 Provides results that can be used to
improve libraries
A CULTURE OF ASSESSMENT
 Decisions
based on facts, research,
and analysis
 Wide recognition and acceptance of
the importance of assessment
 Expertise and staff development
related to assessment
QUESTIONS AND COMMENTS?
 More
information:
 Jim Self
 [email protected]
 www.lib.virginia.edu/mis