Transcript Full Usability Testing - United International College

Creating effective web presence
1
Strategic Premise
Building a Web Site for an Enterprise or
Non-Profit is Not an Exercise
in Either Technology Or Aesthetics.
It is an Exercise in Creating Satisfying
Customer Experience in a Way that Leads to
Cost-Effective Execution of Marketing Strategy.
COMP7880-web-2
Effective Execution of Marketing Strategy





Building Web sites that deliver satisfying
customer experience and do so in a way that fits
both strategy and budget.
Visual attractiveness is a plus, but not a
necessity.
If technology gets in trouble, it is a negative, not
a plus, e.g., flash intros/demos.
Web sites is an important marketing strategy but
not the only one - that must be cost-effective.
Most Web sites should eventually be expected to
produce a reasonable return on investment.
COMP7880-web-3
Web Site Development Process




Not too different from marketing / communications planning
Planning a Web site should be first
Foremost a business/marketing planning process
Good business sense should take precedence
COMP7880-web-4
Establishing Site Objectives

Enterprise/Unit Marketing Objectives



cascade with levels
The Interactive Nature of the Internet

Communications/Branding Objectives

Behavioral Objectives
Role of Web Site on Overall Marketing/ Marketing
Communications Strategy

integration of online and offline strategy for multichannel marketers
COMP7880-web-5
Identify / Describe Target Market

Demographics, Life Styles

Motives for using the site

Tasks they wish to perform

Consider utility and customization

Stepwise scenario development / use case
analysis
COMP7880-web-6
Site Content / Navigation Structure

What content do visitors need/expect?

How do they access content?


More than just a straightforward replication of offline content

Interactions

Marketing research
What role should visuals/graphics play?

Simple and Usable

Content and structure more important
COMP7880-web-7
Typical Site Hierarchy



Enough second-level pages to clearly categorize content but not create
confusion
Visitor should never be more than 2-3 intuitive mouse clicks away
Avoid dead ends 
COMP7880-web-8
Site Design Issues
Content
 Navigation
 Color (especially background)
 Font
 Minimize Scrolling
 Artwork
 Animation/Graphics/Rich Media
Demo Case: http://www.cnet.com/

COMP7880-web-9
Deployment and Tuning




Uploading site server / hosting service raises
many technical issues
Calibrating and fine tuning for best site
performance is highly technical
Reliability and scalability issues
Test at your target customers’ regions and
environment!
COMP7880-web-10
Measure / Evaluate / Improve
Performance & Effectiveness


Measuring and Improving site performance is
a technical task
Measuring and improving the business
effectiveness of site is a marketing task
COMP7880-web-11
Measurement Techniques

Concept Tests - basic marketing research techniques

Prototype Tests - tests of sites with limited functionality,
usually done in a laboratory setting.

Beta (Functional) Tests –

testing a fully-functioning Web site.

can be done in a laboratory setting or by giving trusted
users/customers access to a site on a server, before the site has been
deployed to the Web.

Customer Usability/Satisfaction Feedback - uses conventional
marketing research techniques to measure satisfaction with
the Web site experience.
COMP7880-web-12
Measuring Web Customer Satisfaction



Employs research methods developed offline
Adapted for the online environment
Single measures vs multiple measures
COMP7880-web-13
Satisfaction with Content
COMP7880-web-14
Satisfaction with Transactional
Experiences
COMP7880-web-15
E-Commerce Satisfaction Drivers
COMP7880-web-16
What Should The Marketer Do
To Create Good
Customer Experience
On The Web Site?
COMP7880-web-17
Stages/Elements of Customer Experience
COMP7880-web-18
Continuous Improvement Essential
Figure 9.12
TOP Image Only
Goes Here
Figure 9.13
TOP Image Only
Goes There
COMP7880-web-19
Elements WSJOnline Offers


Usable Site
Navigation Made Easier By Familiarity With
Print Version

Personalization Options

A Trusted Brand Name

E-Mail Notices—Features, Breaking News

Community Through Feedback/Discussions
COMP7880-web-20
Web Site Costs

2 to 3 times as much to maintain a site as it
costs to develop it initially!
COMP7880-web-21
More on website evaluation
COMP7880-W2-22
The Power of Clickstream to Produce
Internet Metrics


Tremendous amount of data produced on the
Internet.
One of the main challenges for the Internet marketer
is to control this data to improve existing marketing
programs and to gain insights into additional
marketing efforts that have a high probability of
being productive
COMP7880-web-23
Purpose of Usability Testing
Visitors expect smooth navigation suiting their
need
 To be pleased and not frustrated
 The fundamental basis of Web site usability is
user task performance.
• Visitors come to the Web site motivated to
accomplish some goal, to perform some task.
• Usability testing is designed to ensure that task
performance is not only possible, but hopefully
efficient and entirely satisfactory.

COMP7880-web-24
Types/Stages of Usability Testing



Concept Testing - testing site design concepts
to see if they make sense. This is primarily site
structure, not design approaches.
Prototype Testing - testing prototypes to see if
they fit the manner in which users expect the
site to be organized and laid out in order for
them to complete tasks in an orderly fashion.
Full Usability Testing - Testing the full usability
of the site when it is functionally complete and
most if not all of the content is there.
COMP7880-web-25
Pareto Curve for Usability Testing

Over 75% of the problems can be identified with 5
user tests; only 15 are need to find 100% 
COMP7880-web-26
No Website can Ignore the Need for Usability
Testing



Usability testing is critically
important.
A careful marketer can
learn to do it, especially
one who has had focus
group training or
experience.
It can be outsourced to
interactive marketing
agency or a specialized
marketing services firm.
COMP7880-web-27
Site Performance Metrics
COMP7880-web-28
Traffics & Audience Metrics

Site Administered


Hit Counters
Purchased Services

Server Request Log Data

Coded Web Pages

Customer Panel Data
COMP7880-web-29
Hit Counters are Often Free
They Provide Simple But Useful Reports
COMP7880-web-30
What is a Server Log?



Server logs record every hit (every file
requested) and most pages have many files.
This is a necessary lead-in to understanding
that the IT people use server log data to run
the site and marketers use it (after much
processing) to understand the performance of
their marketing programs.
Includes, e.g. IP address, date and time of
request
COMP7880-web-31
Basic Metrics – Site Traffic

Hits



Impressions.



recorded each time a file is requested
little value in measuring site effectiveness.
Typical advertising usage applies here.
Each time a visitor has an opportunity to view an
item, an impression is recorded.
Page views (page impressions).

recorded each time a page is requested.
COMP7880-web-32
Basic Metrics – Site Audience





Visitors - simple count of the number of people who visit
a site
Unique visitors - the number of different people
Identified visitors - the next step up; now we know who
they are
Unduplicated audience - the number of unique
visits/exposures in a specified time frame.
Traffic and audience measures are obviously related.


Traffic simply measures the activity on the site.
Audience measures are of more interest to marketers who need
information about the composition of that traffic.
COMP7880-web-33
Need for Ratings





Management to assess performance
Investors to assess potential & returns
Advertisers for traffic numbers
Must be accurate & verifiable
External audit is the preferred option
COMP7880-web-34
Television Ratings






Create-once-sell-many medium
Production costs don’t change with number of
audience
Producers cannot directly count their audience
Independent panel-based measurement companies
are preferred, e.g. ACNielsen
Survey a representative sample of viewers and the
TV channels to which they tuned
Set-top box is used to record the viewing behavior
COMP7880-web-35
Magazine Ratings






Create-many-sell-many medium
They can count how many copies printed & sold
No way to count magazines actually opened and read
Independent companies verify circulation numbers
based on audits of financial documents, mailing lists,
postal receipts, and printing bills
Survey is much harder than TV as the number of
magazines is much higher than TV channels
No mechanism similar to set-top box
COMP7880-web-36
Web Ratings







Create-once-sell-many medium
Supply side resembles magazines but demand side
resembles TV
Can count using server logs how many pages were
“printed”
Can install set-top box like software to record viewing
behavior
Millions of web sites with billions of web pages require
prohibitively large samples
Representative samples are impractical to put together in
addition to difficulty of installing measurement software
Don’t guess but count!
COMP7880-web-37
Something You Can Know




The referral links let you know how much
traffic is coming from where?
Also captured are the search terms visitors
typed into portals like Yahoo!
Can discover the most-used entry and exit
pages
How long did they stay on each page?
COMP7880-web-38
IAB Online Measurement Study
39
Background & Objectives

Online Advertising Measurement Study







Interactive Advertising Bureau (IAB)
Media Rating Council (MRC)
Advertising Research Foundation (ARF)
Conducted by PricewaterhouseCoppers (PwC)
Review measurement criteria & practices for online
advertising and audience measurement reporting
Document and report the comparability of existing
metrics used by the industry
Propose a common set of industry definitions and
guidelines for data analysis and reporting
COMP7880-web-40
Scope

11 participating companies selected by IAB





Participating companies represented 2/3 of total
industry revenue
Interviews




portals (e.g., AOL, MSN, Yahoo!)
destination sites (e.g., CNET, Forbes.com)
third party ad networks / servers (e.g., Avenue A, DoubleClick)
what types of audience and advertising data are measured
how the data is measured and how it is reported
Verified collection methods & definitions using scripted
testing
Identified discrepancies between definitions, editing
procedures, and reporting
COMP7880-web-41
The Top Five Metrics








Ad Impressions
Clicks
Unique Visitors
Total Visits
Page Impressions
Time Spent on Page
Number of Completed
User Registrations
Conversions
# Participants
12
10
8
6
4
2
0
Ad
Impressions
Clicks
Unique
Visitors
Total
Visitors
Page
Impressions
COMP7880-web-42
Ad Impression



A measurement of responses from an ad
delivery system to an ad request from the
user browser
filtered from robotic activity
recorded at a point as close as possible to the
actual viewing of the creative material by the
user browser.
COMP7880-web-43
Ad Impression Measurement

Server Initiated Measurement




prior to serving a web page to a user agent
request
the page is built with links to an ad
resource
ad impression transaction is recorded in a
log
Client Initiated Measurement


direct connection between a user agent and
the ad server via advanced HTML tags
recorded via an independent request to a
special ad transaction logging server
# Participants
7
6
5
4
3
2
1
0
Server
Client
COMP7880-web-44
Page Impressions

A measurement of responses from a web
server to a page request from the user
browser, which is filtered from robotic activity
and error codes, and is recorded at a point as
close as possible to the actual viewing of the
page by the user browser.
COMP7880-web-45
Clicks



A measurement of the user-initiated action of clicking
on an ad element, causing a re-direct to another web
location.
Tracked and reported as a 302 redirect at the ad
server.
This measurement is filtered for robotic activity and is
recorded at a point as close as possible to the actual
viewing of the destination web location by the user
browser.
COMP7880-web-46
Uniform Use of 302 Redirects




All 11 participants track clicks &
share a common definition
A click is a user-initiated action of
clicking on an ad element causing
a redirect to another web location
A click does not include
information on whether or not the
user completed the redirect
transaction
All base click metric on 302
redirects (or transfers)
successfully processed by the ad
server
# Participant
12
10
8
6
4
2
0
Tracking via 302 No Tracking via
redirects
302 redirects
COMP7880-web-47
Unique Visitors


10 out of 11 participants
track unique visitors
Cookie Based



8 use cookies with 2 using
also IP address
recurring vs new cookies
Registration Based

2 use registered users or
user login counts
# Participants
8
7
6
5
4
3
2
1
0
Cookie
Registration
None
COMP7880-web-48
Cookie Based Unique Visitors



Should new cookie be
counted?
Count all new cookies
Exclude all new cookies


a unique cookie must visit
the site at least twice to be
considered a new visitor
Exclude some new cookies
based on historical data


using known user data
estimate of new cookies
representing repeat visitors
that do not accept cookie
# Participants
4
3.5
3
2.5
2
1.5
1
0.5
0
Count All
New
Cookies
Count No
Count
New
Some New
Cookies
Cookies
COMP7880-web-49
Total Visitors




10 participants calculate
total visitors
Definitions vary among
participants
Actual
Sampling


sample user activity (e.g.
several days over a period)
Statistical Analysis

to perform statistical analysis
to estimate total visitors
# Participants
6
5
4
3
2
1
0
Actual
Sampling Statistical
COMP7880-web-50
Web vs Ad Server Tracking


8 participants track page
impressions
6 use standard web server logs



2 use web beacon technology
(see Yahoo)


with successful HTML status codes
filter from robotic activity
# Participants
6
5
4
3
A Web beacon is an object that is
embedded and invisible but allows 2
checking that a user has viewed the
1
page or e-mail.
no third party entity does the
tracking
0
Web Server Ad Server
COMP7880-web-51
Robotic Activity Filtering


All participants perform some
such filtering
Basic



List of Known Robots



prevent robots from scanning
the ad server
exclude transactions from
empty agents or “bot” agents
based on User Agent Strings or
IP address
varied from 10 to 700
Behavioral Filtering

define business rules to identify
robotic behaviors
# Participants
12
10
8
6
4
2
0
Basic
Filtering
List
Filtering
Behavioral
Filtering
COMP7880-web-52
Internal IP Address Filtering



Eliminate any activity
generated by internal
monitoring tools
Demographic of
company users may not
be representative
Eliminate any activity
generated by internal
testing
# Participants
7
6
5
4
3
2
1
0
Remove Internal Include Internal
IP Addresses
IP Addresses
COMP7880-web-53
Independent Verification
# Participants
7
6
5
4
3
2
1
0
Process
Audit
Activity
Audit
Process & No Audit
Activity
Audit
COMP7880-web-54
Resources


Web Metrics: Proven Methods for Measuring Web Site Success,
Jim Sterne
John Wiley & Sons, Inc., 2002
IAB Online Ad Measurement Study
PricewaterhouseCoppers, 2001
COMP7880-web-55