Transcript New Technologies and Methods

ND-AMA School of Marketing Research 2003
New Techniques and Technologies:
Best Practices and
the Future of Online Research
Socratic Technologies, Inc. © 2003
1
ND-AMA School of Marketing Research 2003
Agenda
•
Overview of Research on the Web
¬ State of the Online Research Industry
¬ Evolving Online Populations
¬ Dealing with the Non-response Crisis in
Research
¬ Salience  Web Cooperation and Satisfaction
•
Best Practices in Online Research Design
•
Techniques and Methods
¬ Historical Technological Perspective
¬ New Techniques
•
Best Practices in Online Research Recruitment and Sampling
Socratic Technologies, Inc. © 2003
2
ND-AMA School of Marketing Research 2003
Evolution of Web-based Surveying
•
The research industry in the U.S. began actively working with Webbased surveying around 1995
•
In the early years, online research was met with skepticism from
academic, business and traditional research agencies
•
Key Concerns:
¬ Web penetration was too low to be “representative” and
“projectable”
¬ Identity of online participants was difficult to “verify”
¬ Significant minority communities of the population were not
“online”
¬ Ethnic Groups
¬ Low Income
¬ Low Education
¬ Fear of changing from “traditional” methods
Socratic Technologies, Inc. © 2003
3
ND-AMA School of Marketing Research 2003
Benefits Now Outweighing Concerns
•
Over time, many of the issues causing concern have been either
corrected or determined to be less important
•
Numerous studies have shown that Web-based surveys (when done
with similar demographic groups) yield virtually the same results as
traditional methods
•
Other benefits have superceded procedural concerns
¬ Cost
¬ Speed
¬ Cooperation Rates (>45%)
¬ Quality of Data
•
At the same time, the response rates to traditional requests have
fallen sharply
¬ Random telephone response rates (<12%)
¬ Cooperation rates (<40%)
Socratic Technologies, Inc. © 2003
4
ND-AMA School of Marketing Research 2003
Model of U.S. Adoption of Web Research
Large number of U.S.
companies projecting
more than 60% of
research to be switched
from “traditional” to
online
Socratic Technologies, Inc. © 2003
5
ND-AMA School of Marketing Research 2003
Growth in U.S. Web Research Revenue
•
Web-based research is expected to grow at an average of 30%
per annum over the next two years to 45% of U.S. research
revenue in 2005
1,000
Revenue in US$ Millions
900
18% of US Research Spending
800
700
24%
600
500
400
300
200
100
1995
1996
1997
1998
1999
2000
2001
2002
2003
Source: Inside Research, March 2002
Socratic Technologies, Inc. © 2003
6
ND-AMA School of Marketing Research 2003
Online Research Economics
Hours to Program 20-minute
Complex Interactive Survey
90
80
70
Hours
60
50
40
30
20
<4 Hours
10
0
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
Cost per Hour
for Programming
2003
140
120
100
US$
Productivity enhancements
drive down costs and
time-to-field.
80
60
40
20
0
1993
Socratic Technologies, Inc. © 2003
$32/hour
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
7
ND-AMA School of Marketing Research 2003
Online Research Economics
Programming & Field Costs
60,000
Most traditional research
costs are driven by
marginal, variable costs:
Interviewers
•
Online research costs are
mostly fixed, up-front
charges with low variable
costs
CATI
50,000
US$ (Average)
•
n=575
Web = ½ CATI
40,000
30,000
20,000
10,000
Web
0
0
200
400
600
800
1000
1200
1400
1600
Sample Size Breakeven with CATI
Sample Size
450
400
For larger sample sizes,
Web can be half the cost
of phone.
350
300
250
200
n=75
150
100
50
0
1993
Socratic Technologies, Inc. © 2003
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
8
ND-AMA School of Marketing Research 2003
Online Research Industry Profile
•
Three Major Trends:
¬ Diffusion of Technology
¬ Consolidation  Absorption
¬ Hyper-competition
Average Revenue per Project
Industry profits are derived
almost exclusively from
marginal productivity gains.
# of Projects
1996
Socratic Technologies, Inc. © 2003
1997
1998
1999
2000
2001
2002
2003
9
ND-AMA School of Marketing Research 2003
Growth in Size of Internet
•
By year end 2002, 513 million people around the world will have
Internet access; this number will approach 1 billion by 2005
(eTForecasts)
¬ The top 15 countries will account for nearly 82 percent of Internet
users
¬ 79.4 per 1,000 people worldwide 2002
¬ 118 people
per 1,000 by
year-end
2005
Socratic Technologies, Inc. © 2003
10
ND-AMA School of Marketing Research 2003
Number of Internet Users by Region
World Total
612.2 Million
%
Europe
190.91
31%
Asia/Pacific
187.24
31%
USA/Canada
189.3
31%
Latin/South America
33.35
6%
Africa
6.31
1%
Middle East/ROW
5.12
1%
September 2002
Source: http://www.nua.ie/surveys/how_many_online/index.html
Socratic Technologies, Inc. © 2003
11
ND-AMA School of Marketing Research 2003
Barriers to Internet Use
•
Lack of Local Content
¬ While information and content from multi-national sources
may be useful, Internet users may also desire local
information
•
Literacy
¬ In nations where the literacy rate is low, a text-heavy Web
has little use
Language
Language
Online Language
Population1 (2002)
Public Sites in that
Language2 (2002)
Over Represented
English
36.5 %
72 %
Under Represented
Chinese
10.9 %
<1 %
Japanese
9.7 %
6%
Spanish
7.2 %
3%
German
6.7 %
7%
Korean
4.5 %
<1 %
Italian
3.8 %
2%
French
3.5 %
3%
Under Represented
Under Represented
1:http://www.glreach.com/globstats/, 2:http://wcp.oclc.org/
Socratic Technologies, Inc. © 2003
12
ND-AMA School of Marketing Research 2003
Corporate Cultures & Web Research
•
Separate from simple Web penetration figures, some national
business cultures cause companies to resist the adoption of
the Internet as a research vehicle
•
The U.S. has moved beyond most of the early objections to
Web adoption and is converting large proportions of
traditional research to the Web (e.g., General Mills, Kraft
Foods) and experimenting with rich media formats
¬ Continental European and South American companies in
particular tend to be less willing to shift research projects
to the Web
¬ Resistance is strongest in countries with a national
business culture built on personal contact and face-toface meetings (Italy, Spain, Brazil)
Socratic Technologies, Inc. © 2003
13
ND-AMA School of Marketing Research 2003
Global Web Research Adoption Curve
Socratic Technologies, Inc. © 2003
14
ND-AMA School of Marketing Research 2003
Staying Ahead of the Curve
Evolving Online Populations
Socratic Technologies, Inc. © 2003
15
ND-AMA School of Marketing Research 2003
U.S. Online Populations Now
Similar to Offline
•
U.S. Online Populations are beginning to match broader population
statistics
¬ Consumer Demographics Similar
¬ Gender
¬ Race/Origin
¬ Age Groups (<50)
¬ Geographic Dispersion
¬ Household Type
¬ Business Firmographics Similar
¬ 86% Web penetration
¬ Equal distribution across major SIC categories
¬ E-mail/Web preferred method of communication by Business
Decision Makers
Recommendation: http://www.ntia.doc.gov/ntiahome/dn/index.html
Socratic Technologies, Inc. © 2003
16
ND-AMA School of Marketing Research 2003
Web Still Has Important Differences
•
Psychographic Differences between online and offline respondents
affect survey results
•
People appear to be “in a different place” mentally when responding
to Web surveys
•
Online users’ attitudes change rapidly through experience and gain
media-based savvy more quickly than other communication modes
Socratic Technologies, Inc. © 2003
17
ND-AMA School of Marketing Research 2003
Psychographic Differences
•
There are some differences between
data collected on line compared to
interviewer administered methods
•
Web vs. Telephone
e-Personality
Effect Pattern
e-Personality
¬ Aggressive Behavior
¬ Stronger Opinion Positions
¬ Intense Candor
¬ More Participatory
¬ Cynicism
Socratic Technologies, Inc. © 2003
18
ND-AMA School of Marketing Research 2003
Anonymity Gradient
E-Personality Driven by
Anonymity Gradient
Less Human Interaction =
Greater Candor
¬ Effect is intensified
in Asian cultures
¬ Effect is less
pronounced
in No. Europe
Socratic Technologies, Inc. © 2003
19
ND-AMA School of Marketing Research 2003
Comparing “Threats to Validity”
Phone vs. Web
Internal Threats (Design and Implementation)
•
Extraneous Factors
No difference
•
Changes in Subjects
Web may be better (speed)
•
Measurement Changes
Web better (<Interview bias)
•
Subject Guessing
No difference
•
Equivalent Groups
No difference
•
Dropout Rate Over Time
Phone may be better
(email address perishability)
Burns, Bush 2003, pg 137
Socratic Technologies, Inc. © 2003
20
ND-AMA School of Marketing Research 2003
Comparing “Threats to Validity”
Phone vs. Web
External Threats (Projectability)
•
Representativeness of Sample
No difference
•
Realism
Web may be better (visual)
•
Gerneralizability
No difference
Burns, Bush 2003, pg 137
Socratic Technologies, Inc. © 2003
21
ND-AMA School of Marketing Research 2003
Web vs. Traditional Phone
Socratic Technologies, Inc. © 2003
22
ND-AMA School of Marketing Research 2003
Implications of Web Drawbacks
•
Phone may be better if…
¬ Answers are likely to be broad ranging and/or unclear
(need to probe)
¬ Survey requires “persuasion” to get a full set of answers
(need to coax responses)
¬ Audience is not likely to be online (Web access is not
representative of study population)
•
Rule of thumb  If 60% of the population is not “online” use
alternative method or hybrid approach
•
All other areas, Web is generally as good as or better than
traditional phone
Socratic Technologies, Inc. © 2003
23
ND-AMA School of Marketing Research 2003
Types of Online Research
•
Proportion of study types from top 27 Online MR Firms
Concept/Product Testing
Customer Sat
A&U Studies
Advertising/Brand Tracking
Site Evaluation/User Profiles
Sales Treacking
Copy Testing
Qualitative/Focus Groups
Opinion Polling
Other
0%
5%
10%
15%
20%
25%
30%
35%
Inside Research, Jan. 2003
Socratic Technologies, Inc. © 2003
24
ND-AMA School of Marketing Research 2003
Web Candor = Data Quality
•
“Self-reported data” collected via the Web more closely
approximates known behavioral data
12
10
Case Study:
CATI – consistently
understated problems
overstated satisfaction
8
Phone
6
Web
Known
4
2
Web – much closer to
known behavioral data
Socratic Technologies, Inc. © 2003
0
Avg # of
Avg Length of Avg # Calls
Issues 1999
Wait
to Resolve
Avg # of
Reps to
Solve
Total Days to Mean Sat
Resolve
with Process
(5-Pt)
25
ND-AMA School of Marketing Research 2003
Web Candor = Data Quality
•
Most Purchase Intent and Projected Behavior data must be
“deflated” for overly optimistic estimates
CATI Purchase Likelihood Deflator
Proportion where Stated = Actual
90%
80%
Proportion
Stating
25%
30%
22%
10%
13%
Stated Purchase Likelihood
Definitely Would (5)
Probably Would (4)
Might or Might Not (3)
Probably Would Not (2)
Definitely Would Not (1)
Deflator
75%
50%
35%
25%
10%
Product
19%
15%
8%
3%
1%
70%
60%
50%
CATI Deflator
Web Deflator
40%
30%
20%
46%
Total Projected Actual Purchases
10%
0%
5
CATI
80%
70%
3
2
1
Web
90%
80%
70%
60%
60%
50%
Rating
40%
Deflator
Projected Actual
30%
Rating
50%
Deflator
40%
Projected Actual
30%
Total Actual = 46%
20%
Total Actual = 46%
20%
10%
10%
0%
4
0%
5
4
3
2
Socratic Technologies, Inc. © 2003
1
5
4
3
2
1
26
ND-AMA School of Marketing Research 2003
Rapid Evolution of Sophistication
•
Online population “matures” faster than offline
•
Offline customer tastes and demands can take years to
change
¬ Brand positioning,
messaging, and
effective promotional
strategies can last for decades
•
Online customer tastes change rapidly
as a by-product of experience
¬ Levels of "satisfaction" with an Online process can
change in a matter of months
•
Large differences between B2B and B2C customer rate of
evolution
Socratic Technologies, Inc. © 2003
27
ND-AMA School of Marketing Research 2003
Rapid Evolution of Sophistication
•
Model of customer evolution
Results from
21 studies 1995
through 2001
Socratic Technologies, Inc. © 2003
28
ND-AMA School of Marketing Research 2003
Online Usage Behavioral Profiles
Socratic Technologies, Inc. © 2003
29
ND-AMA School of Marketing Research 2003
Online Usage Behavioral Profile Proportions
80%
70%
70%
60%
50%
50%
40%
30%
30%
70%
60%
60%
50%
50%
50%
25%
20%
20%
10%
50%
0%
40%
Men
Women
US
German Japanese
70%
60%
30%
Fast Adopters
25%
60%
20%
50%
40%
10%
40%
0%
30%
25%
25%
Men
Women
US
German
Japanese
20%
20%
Utilitarians
10%
0%
Men
Women
US
German Japanese
Abandoners
Socratic Technologies, Inc. © 2003
30
ND-AMA School of Marketing Research 2003
Staying Ahead of the Curve
Best Practices in Research Design
Socratic Technologies, Inc. © 2003
31
ND-AMA School of Marketing Research 2003
Email Invitations:
10 Steps to Greater Cooperation
Socratic Technologies, Inc. © 2003
32
ND-AMA School of Marketing Research 2003
1. Don’t look like spam!
•
#1 reason that people do not respond to email invitations is
"I thought it was spam“
•
Send customized text messages, not HTML
•
Use the full, correct email address for each invitation. Do NOT
use BCC or bulk mailing options
•
Familiarity with the sender is key: don't constantly change
domain names from which invites are sent
¬ Avoid domain name elements that "spam filter rules" will
eliminate:
¬ Offer, Free, Blast, Private, Bargain, Discounts, Daily, Deals,
Promo, Win/Winner, Shop, Lotto, Marketing, Rewards,
Wholesale, Unique, Thrifty, Value, Direct, Buy
Don’t BE
spam
Socratic Technologies, Inc. © 2003
33
ND-AMA School of Marketing Research 2003
2. Message Line: “Sponsor” “Topic” "Survey"
•
Dealing with a "known and trusted" source is also a key to
getting your email invitation opened
•
Trial-and-error has shown that a combination of "source and
topic" is important
•
Coupling the source and topic with the specific request word:
"survey" usually produces the best effect
Socratic Technologies, Inc. © 2003
34
ND-AMA School of Marketing Research 2003
3. First Statement
•
Once opened, you have between only 2 and 5 seconds worth of
reading time to get the person's decision to toss or to
participate
¬ (Compare this to 7 seconds of a spoken telephone
solicitation)
¬ About one sentence's worth of content
¬ Our recommendation: Distance research from direct
sales/spam
•
Recommended wording:
¬ "This invitation to take part in research and is not a sales
solicitation."
Socratic Technologies, Inc. © 2003
35
ND-AMA School of Marketing Research 2003
4. Salience Points
•
Different factors will act most strongly to convince someone
whether or not to participate.
¬ These factors are difficult to predict because they differ by
individual and by study over time
•
Suggest a bullet-point list of all four key salience drivers:
¬ What are we researching?
¬ How much are we offering for their time?
¬ Who is the sponsor (or affiliated industry)?
¬ How much time/effort will this take?
Socratic Technologies, Inc. © 2003
36
ND-AMA School of Marketing Research 2003
5. Non-Qualified Terminates Recognition
•
If your sample must be screened for demographic or other
characteristics, give an indication of what will happen if they
don't qualify
•
People who terminate without recognition of their effort are
less likely to participate in future research
•
Recommended wording:
¬ "If you do not qualify to participate in the entire survey,
you will still be entered into a drawing to win $XXX just for
trying."
Socratic Technologies, Inc. © 2003
37
ND-AMA School of Marketing Research 2003
6. Contact for Help/More Information
•
Today's climate of spam and "sifting" has created respondent
anxiety about "legitimacy"
¬ Respondents do not trust "third party" researchers as
much as they used to (particularly in B2B)
•
Need to provide a "contactable" human being live or online,
who can assure potential participants that the study is
authorized and legitimate
¬ Include an email (alias) for online contact and a toll-free
number for "questions you may have" about the study
•
Recommended wording:
¬ "Please contact [email protected], our member
services manager, if you have any questions and reference
project number 123-1234. Antonio also can be reached at
800-555-1234."
Socratic Technologies, Inc. © 2003
38
ND-AMA School of Marketing Research 2003
7. Privacy Policy/Confidentiality Statement
•
Privacy concerns center primarily around
¬ Are you going to sell my telephone number?
¬ Are you going to sell my email address?
•
Confidentiality concerns center on
¬ Who is going to see my answers?
¬ Will I be confronted by someone if my reactions are
negative
•
Recommended wording:
¬ "We are a market research firm that values your privacy.
All of your responses will be kept strictly confidential and
reported only in aggregate. Your personal information will
not be sold or traded to any direct-mail companies."
Socratic Technologies, Inc. © 2003
39
ND-AMA School of Marketing Research 2003
8. Privacy Policy Link
•
Over time, more and more people are checking privacy policies
prior to entering any type of personally identifiable
information
•
A link to your formal privacy policy is important
•
Base your privacy policy on one of the many existing
organizations publishing codes of ethics and stick to it!
•
Sources for Privacy Statements…
¬ IMRO (www.imro.org)
¬ CMOR (www.cmor.org)
¬ CASRO (www.casro.org)
¬ AMA (www.marketingpower.com)
¬ MRA (www.mra-net.org)
Socratic Technologies, Inc. © 2003
40
ND-AMA School of Marketing Research 2003
9. Opt-out email address and 800 #
•
Increasingly, state laws are requiring an opt-out mechanism
for "list removal"
¬ 14 states require a toll-free number in addition to an email
reply system
¬ $5,000-per-incident fine for not providing this information
•
If complaints arise, spam filter policy makers also check to see
if email/telephone numbers work
¬ Domains that do not comply will often be blocked at the
ISP level for multiple systems
•
Recommended wording:
¬ "If you would like to be removed from our contact list,
please reply to this email and type 'Remove' in the subject
line, or call 1-800-555-1234 and reference project number
123-1234."
Socratic Technologies, Inc. © 2003
41
ND-AMA School of Marketing Research 2003
10. Industry affiliations and ethics standards
•
If you are a member of a national or international industry
group, publish
¬ the name (with a hyperlink) in your text-based email
invitation and
¬ the logo on your privacy page and on the first page of your
Web survey
¬ If you use a hyperlink within your live survey, be sure it
opens the link in a new window
•
Recommended wording:
¬ "XYZ company is a member of IMRO, the Interactive
Marketing Research Organization, and we subscribe to the
privacy policies and code of research ethics published by
that group."
Socratic Technologies, Inc. © 2003
42
ND-AMA School of Marketing Research 2003
Incentive Programs:
Types and Applications
Socratic Technologies, Inc. © 2003
43
ND-AMA School of Marketing Research 2003
Incentivization Programs
•
Important to match survey types with appropriate levels of
compensation
•
Examples…
¬ Cash Rewards (U.S. "Most Preferred")
¬ Drawings and Prizes
¬ Frequent Flyer
Points
¬ Gift Certificates
¬ Amazon.com
¬ McDonalds
¬ Borders, etc.
•
Europe
¬ Highest Response for
Shared Information
Socratic Technologies, Inc. © 2003
44
ND-AMA School of Marketing Research 2003
Satisfaction by Incentive
Top two box satisfaction ratings increase significantly when an incentive is
offered.
(n=7475)
Yes, 74% s
Incentive Offered
No, 63%
0%
20%
40%
60%
80%
100%
IMRO Survey Satisfaction Research:
Respondent Satisfaction Modeling, February 2003
Socratic Technologies, Inc. © 2003
45
ND-AMA School of Marketing Research 2003
Satisfaction by Incentive Type
Satisfaction ratings are significantly higher among those who were paid for
their participation.
(n=7475)
Drawing
PPC
79% s
67%
0%
20%
40%
60%
80%
100%
IMRO Survey Satisfaction Research:
Respondent Satisfaction Modeling, February 2003
Socratic Technologies, Inc. © 2003
46
ND-AMA School of Marketing Research 2003
Satisfaction by “Compensation”
As mirrored in other studies, “adequate compensation” has a broad “mid
level” at which satisfaction is not radically increased by increased awards.
At the high and low ends of compensation, satisfaction is more
significantly impacted.
3
2
y = 1.1308x - 12.544x + 46.259x - 52.653
R2 = 0.5382
5
4.8
Overall Satisfaction
4.6
4.4
4.2
4
3.8
3.6
3.4
3.2
3
3
3.2
3.4
3.6
3.8
4
4.2
4.4
4.6
4.8
5
Felt Adequately Compensated
IMRO Survey Satisfaction Research:
Respondent Satisfaction Modeling, February 2003
Socratic Technologies, Inc. © 2003
47
ND-AMA School of Marketing Research 2003
Best Practices
in
Online Survey Design
Socratic Technologies, Inc. © 2003
48
ND-AMA School of Marketing Research 2003
Variables Influencing Dropout Rates
in Web-based Surveys
•
Design of surveys must take into account the interaction
between burden and personal return variables…to enhance
salience
¬ length of survey (both in terms of time to complete and
number of questions)
¬ incentive (either total incentive offered as a prize package
or the approximate value of the incentive on an individual
basis)
¬ engagement level
•
A combination of these factors influences the number and
proportion of mid-survey abandoners (mid-terms)
Socratic Technologies, Inc. © 2003
49
ND-AMA School of Marketing Research 2003
Background
• Findings from 19 Web-based studies
¬ All of the studies were with business-to-business,
technology-related decision makers
¬ Surveys included U.S., European and Asian respondents
(all surveys conducted in English)
¬ The total number of respondents included in these surveys
= 21,867
¬ Median sample size = 473
Socratic Technologies, Inc. © 2003
50
ND-AMA School of Marketing Research 2003
Criteria for Assessment
•
Mid-terminates -- an indication of the point at which boredom,
respondent fatigue or lack of perceived value becomes
"critical"
¬ Note: Critical

"pleasurable"
•
As a rule of thumb, when surveys have a mid-terminate rate of
more than 30%, a post-hoc evaluation of factors leading to the
problem is probably a good idea
•
Findings from these studies indicated that no significant
differences can be found across different geographies
Socratic Technologies, Inc. © 2003
51
ND-AMA School of Marketing Research 2003
Measurements
¬ Three variables are considered "suspect" in creating
critical dropout rates
¬ Number of Screens
– Other studies (Microsoft) have shown that the total
number of clicks is also a critical variable
¬ Average Time to Complete
¬ Incentives
– Total Value of the Incentive, or
– Known Value of the Incentive
¬ Effect of Animation
Socratic Technologies, Inc. © 2003
52
ND-AMA School of Marketing Research 2003
Total Incentives
(most predictive)
•
A drawing-type prize package of just over $1,000 was needed
for more than 70% of this audience to complete.
50%
45%
R2 = 0.8149
Dropout Rate
40%
35%
30%
25%
20%
15%
10%
5%
0%
$0
$2,000 $4,000 $6,000 $8,000 $10,000 $12,000 $14,000 $16,000 $18,000
Total Incentive Fund
Socratic Technologies, Inc. © 2003
53
ND-AMA School of Marketing Research 2003
Total Average Time of Survey
•
Surveys that took more than 17.5 minutes led to predicted
completion rates of less than 70%.
50%
Dropout Rate
40%
R2 = 0.7968
30%
20%
10%
0%
0
5
10
15
20
25
30
Mean Time of Survey
Socratic Technologies, Inc. © 2003
54
ND-AMA School of Marketing Research 2003
Known Value of the Incentive
•
A cash-equivalent value of only $5.00 would still leave a
predicted 78% completion rate.
¬ Once the value hit $22.00, the curve flattened noticeably
¬ Increasing the individual incentive rate above a certain
level does little to influence mid-terms
30%
R2 = 0.6087
Drop Out Rate
25%
20%
15%
10%
5%
0%
$0.00
$20.00
$40.00
$60.00
$80.00
$100.00
$120.00
Known Incentive Amount
Socratic Technologies, Inc. © 2003
55
ND-AMA School of Marketing Research 2003
Number of Screens
•
The more screens/questions, the greater the number of the midterminates.
¬ Surveys that exceed 30 screens/questions are predicted to
exceed the maximum acceptable level of dropouts
¬ Microsoft reports that 55 clicks is also a threshold level
80%
2
70%
R = 0.6953
Dropout Rate
60%
50%
40%
30%
20%
10%
0%
0
5
10
15
20
25
30
35
40
45
Number of Screens
Socratic Technologies, Inc. © 2003
56
ND-AMA School of Marketing Research 2003
Effects of Animation on Engagement
% Completes with and without Animation
Duration of Survey (in minutes)
60
With Animation
No Animation
50
40
30
•
< 10
minutes, no
sig. effect
•
17 to 42
minutes,
improved
completion
rates
•
> 45
minutes, no
sig. effect
Zone of Effect
20
10
14%
20%
0
100
90
80
70
60
50
40
30
20
10
% Completing (Once Started)
Model Only: Based on observations from multiple studies and sources.
Socratic Technologies, Inc. © 2003
57
ND-AMA School of Marketing Research 2003
Best Practices Conclusions
•
Minimally, surveys should:
¬ Consist of fewer than 30 questions/screens and/or no
more than 55 clicks to complete the entire survey
¬ Last no longer than 17 to 18 minutes
¬ Ideally last 10-12 minutes (as data from other reports
suggest)
¬ Have a total drawing package worth at least $1,200, or
¬ Have an known per-person incentive of $5.00 to $20.00
¬ Use animations to improve completion rates on longer
surveys
Socratic Technologies, Inc. © 2003
58
ND-AMA School of Marketing Research 2003
10 Tips for Successful
Online Survey Projects
Socratic Technologies, Inc. © 2003
59
ND-AMA School of Marketing Research 2003
1. All The Fundamentals Apply
•
Well-written questions are the fundamental building blocks of
a quality survey
¬ Build response lists that are logical
¬ Clearly define unique response items that don’t overlap
¬ Give the respondent a D/K option if the question is forced
•
Web survey respondents are extremely contemptuous of bad
survey form
¬ Leading questions
¬ Biased response choices
¬ Unbalanced scales
¬ Poor grammar/spelling/clarity
Socratic Technologies, Inc. © 2003
60
ND-AMA School of Marketing Research 2003
2. First Impressions Are Crucial
•
How are you contacting them?
¬ Interstitial Pop-up technology is an efficient way to recruit
customers or site visitors.
¬ While this method affords you highly targeted sample
populations, response
rates and satisfaction with the
survey experience are lower than
more direct methods
¬ Direct communication
¬ Personalized email invitations
have the best results when the
email comes from a recognized
source.
¬ Sample sources must use
double-opt in techniques of
recruitment
Socratic Technologies, Inc. © 2003
61
ND-AMA School of Marketing Research 2003
3. Don’t Flush Your Respondents
•
When a potential respondent “screens out” due either to a
“not qualified” or “over quota” condition, don’t just
“terminate” them
¬ Offer a “dummy questionnaire” to gather other information
¬ Offer a “consolation prize drawing”
¬ Thank for time and effort
•
Recognize contribution to encourage future participation
Socratic Technologies, Inc. © 2003
62
ND-AMA School of Marketing Research 2003
4. Value Their Time
•
Once you have a respondent’s participation, don’t waste
his/her time. Keep the survey short, focused, concise and on
task
Respondents who completed the
shortest surveys (1-6 minutes in
length) rated their surveys
significantly higher on overall
satisfaction that those greater
than 6 minutes in length.
Total
(n=7475)
TOP TWO BOX
%
70
1-6
7-12
minutes
19+
minutes
13-18
minutes
minutes
(n=1199)
(n=2809)
(n=1135)
(n=1806)
%
%
%
%
(a)
(b)
(c)
(d)
76 bcd
67
66
68
Correlation w/
Attributes most highly correlated with overall satisfaction with the survey
process.
Overall Survey Satisfaction
(n=7475)
Attributes Tested
%
The time and effort required to complete the survey was reasonable
.727
This survey was a good use of my time
.716
** Source: Results from 2003 IMRO Report
Socratic Technologies, Inc. © 2003
63
ND-AMA School of Marketing Research 2003
5. Don’t Cram Content
•
Too much content per screen
drives mid terminations.
¬ If respondents have to
scroll down or across in
order to review the
question content they are
less likely to spend the
time.
•
If, as in this example, the
entire survey is on a single
screen, there is little chance
to collect diagnostic
information where and why
people stop or give up.
Socratic Technologies, Inc. © 2003
64
ND-AMA School of Marketing Research 2003
6. Sensitive/Personal Information
•
Beginning the survey with
sensitive personal information is a
deterrent to respondent
cooperation.
¬
Limit personal information
in up-front questions to
those that are required for
screening.
¬
Home Address, Home
Phone, Income and the like
should be asked once a
relationship has been
established through the
survey.
Socratic Technologies, Inc. © 2003
65
ND-AMA School of Marketing Research 2003
7. Engage Them
•
Research indicates that game-like or other engaging
environments, where the subject matter is appropriate, leads
to higher satisfaction with surveys and higher likelihood of
future participation.
Voice opinion
Consider surveys dull
Expect to be sold to
Have time constraints
Get paid
Decrease participation Increase participation
** Source: ESOMAR Fall 2001, “Barriers to Respondent Cooperation And Attitudes Toward Survey Participation”
Socratic Technologies, Inc. © 2003
66
ND-AMA School of Marketing Research 2003
8. Avoid The Technotrap
•
“Using technology for the sake of technology”
¬ Technology affords great opportunity in survey logic and
presenting stimuli. The Technotrap occurs when
technology is applied to a research problem when it is
unnecessary
¬ Increases survey costs
¬ Increases the chances of survey failure
¬ Tends to be overly complex or confusing to the
respondent if the technical solution is unnecessary
Socratic Technologies, Inc. © 2003
67
ND-AMA School of Marketing Research 2003
9. Consistency and Standards
•
Users should not have to wonder whether different words,
situations or actions mean the same thing
¬ Provide simple and clear introduction up front
¬ Keep question instructions short and consistent
¬ Maintain consistency in functionality and navigational
components throughout the survey
Socratic Technologies, Inc. © 2003
68
ND-AMA School of Marketing Research 2003
10. Don’t Box Respondents In
•
Help users recognize, diagnose and recover from errors
¬ Error messages should be expressed in plain language (no
codes),
¬ Precisely indicate the problem, and
¬ Constructively suggest possible solutions
You entered “45 hours” when there are only 24 hours in each day.
If this was the hours you spend per week, would you please enter
your answer again for your hours spent on this task daily?
Socratic Technologies, Inc. © 2003
69
ND-AMA School of Marketing Research 2003
To Sum It Up
•
Most rules of good “self administered” surveys apply to the
online environment
•
Reducing burden and effort are key to a successful survey
•
Engaging tasks and entertainment value hold attention longer
•
However, using extensive logic and techno-gimmicks can
backfire…keep it simple
•
Logic that detects and offers help for “errors” can help
improve overall completion rates at high levels of data quality
•
Always thank, recognize and value the contribution of the
respondent
Socratic Technologies, Inc. © 2003
70
ND-AMA School of Marketing Research 2003
Staying Ahead of the Curve
New Techniques and Technologies
Socratic Technologies, Inc. © 2003
71
ND-AMA School of Marketing Research 2003
Historical Evolution of Web Technologies
Socratic Technologies, Inc. © 2003
72
ND-AMA School of Marketing Research 2003
Control/Human Influence Tradeoffs
Socratic Technologies, Inc. © 2003
73
ND-AMA School of Marketing Research 2003
Online data collection via Handheld Device
Outside the box:
•
Relationship with opt-in respondent panel
¬ Call data tracking
¬ Purchase and purchase intent
information
¬ Survey alerts
¬ Instant polls
Socratic Technologies, Inc. © 2003
74
ND-AMA School of Marketing Research 2003
Evolving Offline Interactive Methods:
PDA-based market research
•
A market research company with 150
mobile workers is piloting the use of
PDA’s for in-person intercepts. Users
were able to compare their vote against
others in various demographics
•
Eliminates use of paper, mailing,
transcription etc.
Socratic Technologies, Inc. © 2003
75
ND-AMA School of Marketing Research 2003
Online Research Tools:
Moving Beyond the Online Questionnaire
• New techniques: extending/evolving
¬ Online techniques moving beyond the traditional offline survey
¬ Qual and Quant Techniques evolving/merging
• Examples of new online research technologies:
¬ Reactive – Real-Time Segmentation
¬ Tracking – Remote Usability Testing
¬ Adaptive – Virtual Product Configuration
¬ Visualization – Concept Demonstration and Illumination
¬ Creative – Interactive Perceptual Testing
• Evolutional considerations: Emerging Techniques versus Technology
Socratic Technologies, Inc. © 2003
76
ND-AMA School of Marketing Research 2003
Reactive – Real-Time Segmentation
•
New techniques allow rapid customization of survey questions and
exhibits based on likely segment membership calculations
¬ Multiple layering of segmentation
and profiling data can yield highly
relevant and personalized survey
instruments
¬ Relevance is highly related to
respondent satisfaction and
interest in future participation
Socratic Technologies, Inc. © 2003
77
ND-AMA School of Marketing Research 2003
Tracking – Remote Usability Testing
•
New technologies used for “following along” with online
exploration
¬ Combines both behavioral and perceptual data
¬ Quantitative samples for online usability tasks
¬ Competitive assessments on live sites
¬ Segmentation of navigational strategies yields insights into
user motivations and differing requirements for
“satisfaction”
Socratic Technologies, Inc. © 2003
78
ND-AMA School of Marketing Research 2003
Adaptive – Virtual Product Configuration
•
Technique based on eCommerce site technology
¬ Building “ideal products” trading off feature benefits with rulebased pricing
•
Analysis based
on data from actual
customer decisionmaking process
•
Identifies features/services
with greatest contribution to
perceived value
Demo
Socratic Technologies, Inc. © 2003
79
ND-AMA School of Marketing Research 2003
Comparing Conjoint and
Configurator Results
•
Creating a "custom
ideal product" yields
similar results to
more complex
conjoint tests
•
Virtual Configuration
is faster…
¬ Conjoint: 25 min
¬ Config: 12 min
•
And more enjoyable…
¬ Conjoint: 30%
(T2B Interesting)
¬ Config: 72%
(T2B Interesting)
Socratic Technologies, Inc. © 2003
80
ND-AMA School of Marketing Research 2003
Visualization – Concept Demonstration
and Illumination
•
Rich media technology gives the researcher the opportunity to
demonstrate and explain product features and benefits in
greater detail
•
Time and areas of exploration can be treated as surrogate
“interest” variables
•
Results show that deeper, animated product demos lead to
higher quality data (approaching real world 3D model
reactions)
•
Very complex
stimuli may be
delivered using a
hybrid CD ROM/Web
methodology
Socratic Technologies, Inc. © 2003
Demo
81
ND-AMA School of Marketing Research 2003
Creative – Interactive Perceptual Testing
•
Techniques borrowed from old focus group/in-depth interview
methodologies used to “expand perceptual vocabulary” of
respondents
•
Actions mimic
¬ sorting,
¬ organizing,
¬ recognition and
¬ recall tasks
Socratic Technologies, Inc. © 2003
82
ND-AMA School of Marketing Research 2003
Audience Meter
•
Streaming video feeds can be
analyzed in real time using a
direct response tool that
gauges interest, appeal,
favorablity,
likelihood-to-purchase, etc.
•
Caching allows control over
smooth transmission and
maintains internal
synchronization of response
to stimulus on a second-tosecond basis
PerceptometerSM
Socratic Technologies, Inc. © 2003
83
ND-AMA School of Marketing Research 2003
Collage Assembly Techniques
•
Collage building is an old focus
group technique used to create a
visual representation of brands,
products and concepts
•
Batteries of images calibrated
for general associative
properties can be used to
measure normative assessments
CollageBuilderSM
Socratic Technologies, Inc. © 2003
84
ND-AMA School of Marketing Research 2003
Tachystiscope Recognition and Recall Tests
•
T-Scopes have been
traditionally been used to
test brief exposures of
design for recognition,
appeal and shelf impact
•
Streaming video allows for
control of time and
protects concepts from
download
Socratic Te-ScopeSM
Socratic Technologies, Inc. © 2003
85
ND-AMA School of Marketing Research 2003
Sorting and Descriptive Techniques
•
“What things have in common” is a
helpful way of understanding
“cognitive segmentation”
•
This technique, frequently used in
focus groups, involves putting like
brands, products, etc. into
groups…then describing the
groupings
•
It becomes even more powerful when
quantitative samples can be analyzed
VisualDifferentiatorSM
Socratic Technologies, Inc. © 2003
86
ND-AMA School of Marketing Research 2003
Color and Color Combination Tests
•
How color and design features
combine to produce pleasing
combinations has traditionally been
a difficult process, often involving
hundreds of potential combination
exhibits
•
Online techniques offer the design
researcher thousands of real-time
options that can be manipulated by
the respondent
•
Field examples have shown that
these types of tests can improve the
color and design option forecasting
process
Socratic Technologies, Inc. © 2003
ColorModelerSM
87
ND-AMA School of Marketing Research 2003
Staying Ahead of the Curve
Best Practices in Recruitment and
Sampling
Socratic Technologies, Inc. © 2003
88
ND-AMA School of Marketing Research 2003
Adopt a Code of Ethics (e.g. IMRO’s)
Two Key Elements
•
Rights of Confidentiality
¬ By default and design, confidentiality shall be granted for all
information collected from customers and individuals, and will be
used for the clearly stated and intended purposes only. All personal
data will be secured against access by third parties and/or
unauthorized individuals or organizations.
•
Rights of Privacy
¬ The right of the individual to be free from unsolicited contact is duly
recognized. By default and design, customers and individuals will
have the right to opt out of the research process. Specifically, IMRO
members will abstain from the following types of surreptitious
sampling and unsolicited or unethical recruitment techniques,
known collectively as "spamming."
Socratic Technologies, Inc. © 2003
89
ND-AMA School of Marketing Research 2003
Recruiting Children
•
The Children’s Online Privacy and Protection Act (COPPA), an
FTC law, was passed in 1998 and became effective as of April
21, 2000
•
Prohibits the collection of personal information from anyone
under the age of 13 years without written parental consent
•
Requires researchers to screen out children under the age of
13 at the start of any online survey effort
Socratic Technologies, Inc. © 2003
90
ND-AMA School of Marketing Research 2003
Ten “Unethical” Techniques
•
Sampling
•
Recruitment
¬ Sifting
¬ Spamming (in general)
¬ Agents, Autobots and
Spiders
¬ Baiting
¬ Purchase of Bulk E-mail
Lists
¬ Illegitimate Use of Client
Data
Socratic Technologies, Inc. © 2003
¬ Auto-mail Generators
¬ Scamming
¬ Spoofing
¬ Buddy/Chain Letters
91
ND-AMA School of Marketing Research 2003
Four Main (Ethical) Recruitment Techniques
•
Permission-Based Contact
¬ Offline Recruiting
¬ Pre-Recruited Panels
¬ Site Intercept
¬ Customer Database Sample
Socratic Technologies, Inc. © 2003
92
ND-AMA School of Marketing Research 2003
Offline Recruitment
•
“Real World” techniques to direct people to a survey
¬ Ads in the newspaper
¬ Phone
¬ Mail
•
50% to 60% online response
from those who have agreed
to participate
¬ Greater proportions of people
participate in the U.S. than
in Europe or Asia
Socratic Technologies, Inc. © 2003
93
ND-AMA School of Marketing Research 2003
“Digital/Analog Divide”
•
More difficult to get people to do things online when using
offline techniques to “drive” them to the Web
•
Converse is also true;
it’s hard to get people to
carry out offline tasks
(like keeping a
paper-based diary) when
using online means of request
Socratic Technologies, Inc. © 2003
94
ND-AMA School of Marketing Research 2003
Site Intercept
•
Passive methods
¬ Banners
¬ Buttons
¬ Badges
¬ Hypertext Links
•
Difficult to associate with a specific
response rate, because the contact
rate is hard to track
•
Actual response rate is low (e.g., 1.5%)
¬ Sites with a lot of traffic (over 5,000 hits per day) 
200 to 300 in less than a week
Socratic Technologies, Inc. © 2003
95
ND-AMA School of Marketing Research 2003
Site Intercept
•
Active Methods
¬ Interstitial Window
¬ Intercept Rate Can Be Paced To Display The Invitation
To Every nth Visitor
¬ Reduces
Self-Selection Bias
•
15% To 30% Response Rates
Source: www.dell.com
Socratic Technologies, Inc. © 2003
96
ND-AMA School of Marketing Research 2003
Customer Registration Database
•
Pre-collected email addresses of people who have opted-in
for future contact
•
Emails invite people to take part in a survey and link them
directly to it
•
Beware of purported
“Opt-In” lists that may
be sold by some
unscrupulous sources
Socratic Technologies, Inc. © 2003
97
ND-AMA School of Marketing Research 2003
Pre-Recruited Panels
•
Commercial Panels
•
Ready-To-Use Database
From Which A Random
Sample Can Be Selected
And Invited
•
40% To 70% Participation
Levels Are Not Uncommon
CMOR data for 404 studies show
Internet cooperation rate averages
77% -- April 2002
Socratic Technologies, Inc. © 2003
98
ND-AMA School of Marketing Research 2003
Advantages of a Panel
Compared to One-Time Research
• Involvement
–
Members are actively involved in survey topics
¬ High quality responses
¬ High survey participation rates
• Accelerated Speed
–
Research projects are completed in days rather than months
• Reduced Research Cost
–
Per-study recruitment costs are dramatically reduced
(typically the most expensive part of a project)
• Sense of Affinity
–
Panel members gain a sense of affinity with the panel (and the organization
supporting the panel) without undue halo effects
Socratic Technologies, Inc. © 2003
99
ND-AMA School of Marketing Research 2003
Panel Core Areas
Member Communication
Panel Web Site
Management
Member Services
and Support
Research
Web Survey Tool
Reporting
Real-time Online
Reporting Tool
Socratic Technologies, Inc. © 2003
100
ND-AMA School of Marketing Research 2003
Panel Web Site
•
The Web site: core element for communication with members
•
Typical panel sites contain the following:
–
–
–
–
–
–
–
–
Presentation of panel concept
Web-based registration surveys
Targeted content
Description of projects
Description of incentives
FAQs and “help” functionality
Privacy/Confidentiality rules
Interface to update
personal information and
redeem incentives
Socratic Technologies, Inc. © 2003
101
ND-AMA School of Marketing Research 2003
Member Recruitment
•
IMRO members utilize double “opt-in”
processes to confirm panelist interest and
willingness to participate
¬ All potential panelists complete a
detailed profiling survey
Recruitment through an “optin” customer database or via
a Web site intercept is often
the quickest (and cheapest)
option
¬ Respondents are then directly offered an
opportunity to join the panel, if they
qualify
•
Participation in surveys is voluntary;
respondents are able to “opt-out” of the
panel at any time, if desired
Socratic Technologies, Inc. © 2003
Required panel size is
typically based on anticipated
number of surveys, sample
size and structure, and
survey timing
102
ND-AMA School of Marketing Research 2003
Member Retention
•
Once panelists have been recruited, member retention is key
¬ Regular panel usage is crucial to keep members “engaged”
¬ Less than once per quarter leads to large drop-outs
•
Replenishment recruitment is necessary to to avoid “lapsed” or
“stale” members
¬ Recommend using a
“Member Quality Index” to gauge
the amount of replenishment necessary
¬ Frequency of participation
¬ # of refused invitations
¬ Type of studies completed
Socratic Technologies, Inc. © 2003
Our experience has shown
that the drop-out rate ranges
between 10% and 35%
annually, depending on
population and usage
103
ND-AMA School of Marketing Research 2003
Panel Timing
•
Panels typically take one month to develop
¬ Set-up and infrastructure takes 2 weeks
¬ Recruitment typically takes 2 to 8 weeks, depending on
recruitment methodology and member target specifications
•
Web survey can begin immediately after
community begins to reach “critical mass”
¬ Generally about 500 – 600
members
Socratic Technologies, Inc. © 2003
104
ND-AMA School of Marketing Research 2003
Summary
•
The Web is continuing to advance as a cost-effective, fast and
high quality medium for collecting survey data
•
Over time, Web-based populations are beginning to match
demographics of the offline population
¬ However, psychographic characteristics and Web-related
environmental differences do continue to produce
different, and arguably better results
•
Communicating with respondents over the Web is becoming a
“preferred mode” of contact for many sub-populations
•
The Web environment gives the research designer greater
flexibility in designing surveys with greater salience for
respondents
Socratic Technologies, Inc. © 2003
105
ND-AMA School of Marketing Research 2003
Summary
•
New methods and non-Web-based interactive technologies
continue to provide engaging and effective ways of collecting
data
•
The ethical imperatives of the Web-based culture are different
and distinct from traditional research norms of sampling and
recruitment, and must be respected
Socratic Technologies, Inc. © 2003
106