Transcript MS PowerPoint 97/2000 format

Evidence, Impact, Metrics:
Proposal For Further Work
Brian Kelly
UKOLN
University of Bath
Bath, UK
Acceptable Use Policy
Recording this talk, taking photos,
discussing the talk using Twitter,
blogs, etc. is permitted but please try
to minimise distractions to others.
Blog:
http://ukwebfocus.wordpress.com/
Twitter:
@briankelly
UKOLN is supported by:
This work is licensed under a AttributionNonCommercial-ShareAlike 2.0 licence
(but note caveat)
Timetable
UKOLN’s EIM Work 2010/11
 Context
 Examples
 Additional documents
Discursive
• That’s interesting
• I didn’t know that
• I disagree with that
Tangible deliverables,
Proposals for 2011/12
identified owners &
 Work for DI programmes
dates
 Work relevant to
• I want a piece of the
DI programmes
action
• That will help my
 Work for sector
programme
• This is already being
done
• It’s too hard
2
Discursive
Background
Evidence, Impact and Metrics Work
• Engage with community in order to explore
ways of gathering evidence of the value of
online services and explore appropriate
metrics to facilitate comparisons
• Scoped to institutional use of web services
and professional/institutional use of social
web services
• Three 1-day workshops (~100 participants);
6 additional talks and ~24 evidence-based
blog posts (21,800+ views)
3
Discursive
Changes In Cultural
Some initial scepticism and mistrust
• We know we’re doing well
• Bean-counters
• You’ll use evidence against us
• Metrics are flawed
• Systems can be gamed
• ….
Fortunately we had external support in
changing the culture …
4
Discursive
The Context
•The need to demonstrate value in light of
Government cuts
NB 1D view:
no comments
on ROI,
comparisons
with print, TV,
… costs
5
Discursive
Daily Telegraph
Daily Telegraph data
journalist used FOI to
gather figures on Web
site redevelopment
costs.
UK Web Focus & SiD
blog posts pointed out
flaws in interpretation.
Sector realised that
they couldn’t ignore
such threats.
6
Discursive
Open Data
Government to ‘force’
Unis to publish data
about courses, student
experiences, …
• Builds on Labour’s
Open Gov initiative
• An opportunity to be
welcomed?
• Can take initiative in
publishing open data
about uses of online
services
7
Discursive
Numbers Matter
“Sales of 25 million
iPads in just 14
months. 200
million iOS devices
in total. 15 billion
songs downloaded
since 2003. 130
million books. 14
billion apps
downloaded from a
store that now runs
to 425,000 apps.”
JISC-funded OII Impact report: “The
media and the public are influenced
by numbers and metrics”
8
BBC News
7 Jun 11
Discursive
Bath University Cares
• Bath University is the “”
• The Sunday Times (and THE) University
League Tables are based on metrics
• Ignoring the reality can be dangerous
9
Evidence
Evidence-Gathering
Evidence-based surveys and discussions
published on UK Web Focus blog
Aims:
• Explore methodologies (strengths &
weaknesses)
• Provide benchmarks for future comparisons
• Identify patterns of uses which can help to
inform practice & policy
• Invite & encourage discussions on survey
methodologies and interpretations and
sense-making processes
10
Evidence
Examples
• The Nudge Principle (cf Michael Neilson’s
talk at SOLO11) applied to RSS use across
Scottish Unis
• Use of HTML and RDFa by Welsh Unis
• Evidence of JISC PoWR’s impact
• Cookie usage by Russell Group Unis
• Mobile-readiness of Russell Group Unis
• Effectiveness of comms channels (e.g.
JISCMail, Slideshare) and how evidence can
inform policy
11
Proposal
Proposal 1
Evidence-gathering to support Linked U call
• Snapshot of existing approaches to HTML across
selected HEIs to act as benchmark (Nov 2011)
• More comprehensive snapshot in Jun 2012
• Report to inform JISC
• Presentation at IWMW 2012 aimed at engaging
sector
Audiences: JISC PM, Web managers
Activity: Survey, interpretation and open discussion,
presentation.
Times: Part 1 in Nov 2011 Part 2 in Jun/Jul 2012
12
Proposal
Proposal 2
Evidence-gathering on use of Cookies
• Snapshot of existing uses of cookies and privacy
policies across selected HEIs and other sectors
to act as benchmark (Dec 2011)
• More comprehensive snapshot in Jun 2012
• Analysis to inform discussions with JISC Legal
• Presentation at IWMW 2012 aimed at engaging
sector
• Report on recommendations
Audiences: Institutions, Web managers, JISC PMs
Activity: Survey, interpretation and open discussion,
presentation.
Times: Part 1 in Dec 2011 Part 2 in Jun/Jul 2012
13
Proposal
Proposal 3
Consultancy to JISC PMs
• Advice provided to JISC PMs on possible ways of
gathering quantitative evidence related to their
programmes
Audiences: JISC PMs
Activity: Discussions and, if required, suggestions
Times: As appropriate
14
Proposal
Proposal 4
Evidence-gathering to support xxxx call
• Snapshot across additional JISC Call in area
relating to UKOLN expertise (Libraries, mobiles?)
• Report to inform JISC and sector
Audiences: JISC PMs, xxx stakeholders
Activity: Survey, interpretation and open discussion,
presentation.
Times: To be negotiated
15
Discussion
16