Transcript chp12

Ch. 12 Observing Users
Reece, Rogers, Sharp. Beyond human
computer interaction.
Team 1:Andy, Nikhil, Vladimir, Sajay
12.1 Introduction
 Learning Goals
 Benefits and challenges
of different methods
 How to observe
 How to deal with data
 Key issues in think-aloud,
diary studies, and
interaction logging
 Selecting and doing
observations
 Additional Content
 Critical Incident
Reporting: A type of diary
observation
 CIF: common industry
format
 Summative vs Formative
 Interaction Logging for the
Web
 Uzilla: A tool for web
usability testing by Andy
12.2 Observing in a controlled
environment
i.e., usability testing
Plan user and equipment location.
Test the setup.
Provide participants with a written
consent form and script of the study.
Use think-aloud technique (p. 365) to find
out what participants are thinking.
Observing as an outsider
 More objective than participant observation
 Lab equipment is in place
 Recording is continuous
 Analysis & observation almost simultaneous
 Care needed to avoid drowning in data
 Analysis can be coarse or fine grained
 Video clips can be powerful for telling story
Frameworks to guide observation
 The Goetz and LeCompte (1984)
framework:
- Who is present?
- What is their role?
- What is happening?
- When does the activity occur?
- Where is it happening?
- Why is it happening?
- How is the activity organized?
Frameworks to guide observation
The Robinson (1993) framework
Space. What is the physical space like?
Actors. Who is involved?
Activities. What are they doing?
Objects. What objects are present?
Acts. What are individuals doing?
Events. What kind of event is it?
Goals. What do they to accomplish?
Feelings. What is the mood of the group and
of individuals?
Participant observation &
ethnography
Must get co-operation of people observed
Informants are useful
Data analysis is continuous
Move backwards and forwards from broad
to specific questions
Collect variety of data
Holistic approach – broad analysis
You need to consider
Goals & questions
Which framework & techniques
How to collect data
Which equipment to use
How to gain acceptance
How to handle sensitive issues
Whether and how to involve informants
How to analyze the data
Whether to triangulate
12.4 Data collection techniques
Notes & still camera
Audio & still camera
Video
Tracking users:
- diaries
- interaction logging
Notes plus still camera
 Taking notes
Least technical way
Most feasible way
Could prove a challenge as one has to observe and
listen while taking notes.
Writing skills such as writing speed, use of short
hand could prove useful
One needs to translate and transcribe the written
content into useful information
 Other alternative are include another person
in the data collection process; use a laptop
 Use a camera to record images
Audio recording plus still camera
Audio recording
Light and easy to carry
Less intrusive than a video camera
Transcribing long interview hours could be a
challenge
Lack of a visual record
Cannot be used in a noisy environment
Video
 Captures both visual and audio data
 Extremely intrusive
 Instruments could be expensive but
digicams/handheld camera are a cheaper
option
 One may tend to focus only on what is seen
through the lens and ignore what has been
missed out.
 Sound may get muffled when recording under
noisy conditions
 Analysis of video data could be very time
consuming.
Lets compare the Data-collection techniques – Table 12.2 page 376
12.5 Indirect Observation
Done when direct observation is not
possible (obtrusive or evaluators cannot
be present)
Tracking users
Diaries
Critical Incidents
Interaction logging
Diaries
 Classical paper based or using online forms
linked to database etc.
 Participant records what she did and experience
she had while doing the task
 Diaries collected from all the participants are
then analyzed
Barry Brown’s research using diarys’ for HP design
framework for devices used to capture or record
information.
http://www.dcs.gla.ac.uk/~barry/papers/Diary%20studychi%20paper.pdf
Advantages
Inexpensive
Subjects could be tested location
independent
Experimenter not required to be present
Large amount of data could be gathered
Disadvantages
Relies on the reliability of the participant
Problem of participant not remembering
actual experience of event (better or
worse)
Subject is the experimenter and hence
variability in data – not much control other
than structures templates to collect
information.
Interaction logging
Record all the interactions of the subject
with the system (key presses, mouse
movements, eye tracking, etc.) combined
with audio and video logs.
Log is time stamped.
Data can be analyzed to understand
cognitive aspects, user behaviors and
reactions to variations in task etc.
Interaction logging - Uses
Useful for web studies.
Testing Web layouts, displays, GUI
interfaces etc.
Understanding the users’ reactions to
variations in tasks and layout
combinations.
Testing the efficiency of the interface and
its ease of use.
Interaction Logging on the Web
 Server log file miss
cached pages
 To circumvent
Web bugs – useful for
visitors at large
Proxies – ala WebQuilt
 http://guir.berkeley.edu
/projects/webquilt/
Instrumented Browsers
 Andy’s Uzilla.net
 PARC’s WebLogger
 Challenges
Inferring user goals
Analyzing paths
 Triangulation
Establishing the
magnitude of problems
observed in testing from
log files
12.6 Analyzing, Interpreting, and
Presenting Data
 12.6.1 Qualitative analysis to tell a story
Hilight tapes, effective communication
 12.6.2 Qualitative analysis for categorization
Advanced linguistic analysis tools
 12.6.3 Quantitative Data Analysis
Measure learning, efficiency
 12.6.4 Feeding the findings back into design
Picture in a picture highlight tapes
Common Industry Format (CIF)
For summative testing
(later stages in software development)
Subjective and Quantitative Measures
On key tasks
Intended for vendors to communicate
usability
Boeing has adopted for all products and
suppliers
Uzilla: A product suite for web usability
testing
Commercial venture by Andy Edmonds
Overview talk SCIP 2002
 Edmonds, K. A. (in press). Uzilla: A new tool for web usability testing. Behavior Research
Methods, Instrumentation and Computers. May, Special Issue of Soc. For Computers in
Psychology.
Automates test design
Instruments data collection
Aggregates results
Free for student projects
Sarah Bly Interview
 Unfinished book on non-speech audio in UI
design
 At Chi ’03
 Visual Support for Conversations: People's Choice
Tutorial, Steve Harrison (Dyxsis D/R Consulting),
Sara Bly (Sara Bly Consulting)
To familiarize the participant with the underlying ideas
and subtle design considerations that make shared
drawing systems essential in many conversational
settings.