Transcript Online, Remote Usability Testing

Online, Remote Usability Testing


Use web to carry out usability evaluations
Two main approaches


agent-based evaluation (e.g., WebCritera)
 model automatically evaluates UI (web site)
 WebCritiera uses a modified GOMS model
remote usability testing (e.g., NetRaker & Vividence)
 combines usability testing + market research techniques
 automatic logging & some analysis of usage
Online Useability Testing
Max – WebCriteria’s Agent


Predicts how long information seeking
tasks would take on a web site
Automated procedure:


seed with start page & goal page
procedure
1) reads page
2) model predicts how long to find & click proper link
load time, scan time, & mouse movement time
3)

repeat until find goal page
Claim: time is directly related to usability
Sample of Max’s Reports
Tradeoffs of Max-style Model

Advantages




very inexpensive (no participants needed)
fast turnaround (hours)
can run on many sites & compare -> benchmarks
Disadvantages

focus on time (much of it download time)




only 3rd in important factors driving repeat web visits
can’t tell you anything about your content
doesn’t say anything directly about usability problems
robots aren’t humans

doesn’t make mistakes




remember, GOMS assumes expert behavior!
doesn’t account for understanding text
only tries the best path – users will use many
major flaw is the lack of real users in the process
Remote Usability Testing

Move usability testing online





research participants access “lab” via web
answer questions & complete tasks in “survey”
system records actions or screens for playback
can test many users & tasks -> good coverage
Analyze data in aggregate or individually



find general problem areas
 use average task times or completion rates
playback individual sessions
focus on problems w/ traditional usability testing
Warning
I am a founder of the
following company –
watch for bias!

NetRaker: Web Experience Evaluation

NetRaker Index



Market Research & Usability Templates




short pop-up survey shown to 1 in n visitors
on-going tracking & evaluation data
surveys & task testing
records clickstreams as well
invite delivered through email, links, or pop-ups
NetRaker Experience Recording


captures “video” of remote participants screen
indexed by survey data or task performance
NetRaker Index:
On-going customer intelligence gathering

Small number of rotated questions increases response rate
NetRaker Index:
On-going customer intelligence gathering

Small number of rotated questions increases response rate
NetRaker Index:
On-going customer intelligence gathering

Increasing these indices (e.g., retention) moderately (5%)
leads to a large increase in revenue growth
NetRaker Usability Research:
See how customers accomplish real tasks on site
NetRaker Usability Research:
See how customers accomplish real tasks on site
NetRaker Usability Research:
See how customers accomplish real tasks on site
Advantages of Remote Usability Testing

Fast



can set up research in 3-4 hours
get results in 24 hours
More accurate

can run with large sample sizes




uses real people (customers) performing tasks
natural environment (home/work/machine)
Easy-to-use


50-200 users -> reliable bottom-line data (stat. sig.)
templates make setting up easy for non-specialists
Can compare with competitors

indexed to national norms
Disadvantages of Remote Usability

Miss observational feedback



Need to involve human participants


facial expressions
verbal feedback (critical incidents)
costs money (typically $20$50/person)
People often do not like pop-ups

need to be careful when using them