Transcript Document

User needs
Vipavc Brvar Irena, ADP, Slovenia
IASSIST/IFDO 2005, 24 - 27 May, 2005
Purpose – EVALUATION
The most general definition of evaluation is that it is a
process of assessment of the value of a certain activity or
objects.
In the case of libraries they evaluate how well the library
satisfy the needs of its users and basic limitations and
weaknesses in the library’s operation. The standard ISO
11620 (1998) defines evaluation as a process of assessing
effectiveness, efficiency, usefulness and relevancy of
a certain services, equipment or facilities.
Lancaster (1977) states that services (in our case also
data archives services), can be evaluated from
different viewpoints and that in the process we find:
- How well a service realises the goals set, which usually
means how well it satisfies the needs of the users.
- How efficiently a service, system realises the goals
set (operating costs).
- Does the data archive justify its existence, that is
what is the usefulness of the data archive service in light
of operating costs (measuring cost-performancebenefits)
Evaluation processes are important for decision
making.
Evaluation is a part of strategic planning process and has
practical significance.
It is in some cases also necessary because of political
reasons (to justify the expenditure of public funding
and to show sense in its spending).
- > The future of data archives does not lie in
supporting generic, broadly useful services such as
information access to large collections, but in supporting
“customization by community”, i.e. the development of
services tailored to support the specific, and real, practices of
different user constituencies.
In data archive area we do not have indicators that
would measure our performance nor do we have
standard questionnaire/questions that would measure
satisfaction of our users with our services.
There were some attempts to measure user satisfactions
(UK DA, ICPSR, University of Tennessee, etc.).
-> Changes in IT brought to us new expectations from
users. We experience change of practice in data
distribution and collection, change of behaviour of users,
and usability. Users want codebooks and data files to be
easy available.
Establishment of data arhive and launching of Web
page/Nesstar (Situation in 2005)
NO DK
2006
DE
NL
AT
CH
2004
SI
GB
2002
IT IR RO
RO
EE
2000
SI
AT
CH
GB
1994
FI
EE
NL
1996
GR
CZ
ES
1998
FI
DK
DE
HU
NO
SE
1992
1990
1955
1960
1965
1970
1975
1980
1985
1990
Year data archive was established
Web page
Nesstar
1995
2000
2005
Decision was made to conduct a survey and evaluate users
needs and their satisfaction with offered services.
We do want responses from different groups of users
(researchers, users and potential users)
Registered Slovene
researchers in the
field of social
science.
Cover letter by email
Banner on
Faculties
Web pages
Known users- in
our user
database
How to measure usability
- Analytically: counter – log analyzer, tracing paths,
Web site usability evaluation, etc.
- Empirically: survey...
Information from Web statistics (log analysis):
* Number of visits, and number of unique visitors,
* Visits duration and last visits,
* Days of week and rush hours,
* Domains/countries of hosts visitors (pages, hits, KB,
269 domains/countries detected, GeoIp detection),
* Hosts list, last visits and unresolved IP addresses list,
* Most viewed, entry and exit pages,
* OS used,
* Browsers used ,
* Search engines, keyphrases and keywords used to find
your site,
* Number of times your site is "added to favourites
bookmarks".
* Screen size
* ......
• Analysis of log file from Nesstar
• Analysing user paths
Colleting
Study. Info.
news
event
Particular
study
EN
List of
studies
SLO
ADP
Price list
Mail-contact
Order
form
depositors
Survey on user satisfaction
• Pretest (current stage)
• Define Sampling Frame and Contact Design
• Decide on data collecting method
Web based survey
Create separate URL addresses for each email
address given. Demands for anonymity are satisfied.
Web survey with PHPSurveyor
PHPSurveyor is a multi-question surveying tool that
allows you to develop, publish and manage surveys.
PHPSurveyor script allows you to display surveys as single
questions, group by group or all in one page or use a
dataentry system for administration of paper-based
versions of the survey. PHP Surveyor can produce
"branching" surveys (set conditions on whether individual
questions will display), can vary the look and feel of your
survey through a templating system, and can provide
basic statistical analysis of your survey results.
-Conversion to SPSS data file was written.
Most of current user satisfaction surveys ask users to tell them
how much they are satisfied and how important particular
mentioned topic is (5 point scale) .
To shorten the questionnaire we decided to do it a little bit
different.
We inquire about overall satisfaction (on general, and about
web page) and to evaluate particular services.
Discriminant analysis!!
-> create indexes that measure satisfaction and importance
Relation between satisfaction with and
importance of particular element
Satisfaction
Importance
This need to be
changed!!!!
Questionnaire available at
http://www.adp.fdv.uni-lj.si/publikacije/userneeds.html
-
Literature
AMBROŽIČ, Melita (2003): A few countries measure impact and outcomes –
most want to measure at least something. In: Performance Measurement
and metrics. Vol.4, No.2, p. 64-78.
BARNES, Stuart J. And VIDGEN, Richard (2001): An Evaluation of CyberBookshops: The WebQual Method. In: International Journal of Electronic
Commerce. Fall 2001, Vol.6, No.1., pp.11-30.
BOLLEN, Johan and LUCE, Rick (2002): Evaluation of Digital Library Impact
and User Communities by Analysis of Usage Patterns. In: D-Lib Magazine.
Vol. 8, No.6.
COLEMAN, Anita and SUMNER, Tamara (2004): Digital Libraries and User
Nees: Negotiating the Future. In: Journal of Digital Information. Vol 5, Issue
3.
HERNON, Peter and ALTMAN, Ellen (1998): Assessing service quality:
satisfying the expectations of library customers. Chicago, London:
American Library Association.
NORLIN, Elaina (2002): Usability testing for library Web Sites. A hands-on
guide. Chicago, London: American Library Association.
RUBIN, Jeffrey (1994): Handbook of usability testing. How to plan, design,
and conduct effective tests. John Wiley & Sons, USA.