Transcript Session6 - Duke University`s Fuqua School of Business

Questionnaire Design, Survey Methods,
Sampling and Causal Research
Market Intelligence
Julie Edell Britton
Session 5
September 4, 2009
1
Today’s Agenda
Announcements
Comparative Advertising, Measurement
Scales & Data Analysis
Introduction to Survey Research
Sampling Procedures
Causal Research - maybe
2
Announcements
For Sat prepare Milan Food case – download
data (Milan.sav) from the platform, please
post your responses by 8 pm tonight – no
slides needed.
For Sat prepare WSJ/ Harris Survey – no
slides
3
Comparative Advertising Measurement
Scales & Data Analysis
Page 52 packet
What do you conclude?
Remember – Percentage change or
difference can only be calculated with Ratio
scales
This is an interval scale (at best).
4
Descriptive Survey Research
 Surveys usually used for descriptive research
 Provide a snapshot at a point in time
 Most analyses univariate or bivariate (but can do
elaboration model with control variables)
 Would you recommend National to a friend interested in
insurance services? Yes 1 No 2
 Bivariate allows for hypothesis testing
 Hypothesis: Less educated people more likely to recommend
 Descriptive, not causal
 Recommendation could be driven by some 3rd factor
correlated with education such as income
5
Sources of Survey Errors




Population definition
Representativeness of the sample frame
Sampling Procedure Used
Respondent Participation:




Willing to participate (Do Not Call)
Comprehend questions
Have knowledge, opinions
Willing & able to respond (language or memory)
 Interviewer understands & records accurately
6
Raising Willingness to
Participate
 A good response rate requires persuasion
 Survey Introduction
 Phone or send letter in advance
 Introduce self, give affiliation unless this would
bias
 Describe purpose briefly, w/o making survey
sound threatening or demanding
 Make respondent feel that s/he is getting chance
to provide opinions that will influence market
offerings & that her/his cooperation is extremely
important
7
Comprehends Questions?
Advice on Question Wording
 Be simple and precise
 Give clear instructions
 Check for question applicability
 respondent screening
 question branching based on prior
answers
 Avoid leading & double barrel questions
8
What’s the Problem?
 “Laws should be passed to eliminate the
possibilities of special interests giving huge sums
of money to candidates”
 “Laws should be passed to prohibit interest groups
from contributing to campaigns, as groups do not
have the right to contribute to candidates they
support?”
9
Comprehends Questions?
 Literacy, translation considerations
 Conversational Norms
 How demanding was Term 3? How demanding was
Core Finance?
 How demanding was Core Finance? How
demanding was Term 3?
 How demanding was Managerial Accounting? How
demanding was Core Finance? How demanding was
Global Economic Environment of the Firm? How
demanding was Term 3?
“Question order effects in measuring service quality,” by DeMoranville and Bienstock,
in International Journal of Research in Marketing, September, 2003
10
Do Respondents Have Knowledge?
 Retrieve answer from memory vs. construct it on spot
 Constructed answers are more likely to be influenced by
question wording & prior questions.
 When answering later questions or engaging in later behavior,
likelihood of using earlier answer input A:
 positively related to accessibility of A
 positively related to diagnosticity (relevance) of A
 negatively related to accessibility, diagnosticity of alternative inputs B, C,
etc. (Feldman & Lynch)
 e.g., when political poll respondents asked:

issue opinion A, presidential voting intention, issue opinion B,
 answers to A predict intention, but only for those who did not vote for either
candidate in primary
11
Survey Best Practices:
Survey Content, Question Order
 Survey Questions
 First figure out what questions are needed!
 Then order
 Lead with interesting, nonthreatening, easy questions
 Do you like to play golf?
 Have you ever travelled with your clubs?
 Can you remember the last time you traveled with your clubs?
 Put difficult or sensitive questions well into the interview
 How many times did you have to see your doctor for your
reconstructive surgery?
 What is the size of your company (revenue)?
 Usually use funnel order (general to specific)




Use product category?
Brand X?
Do you like Brand X?
Why?
12
12
Question Order (Cont.)
 Survey Questions (cont.)
 Inverted funnel (specific to general) for complex topics.
 Is your company considering offering training courses on word
processing over the Internet?
 Database? Spreadsheets?
 In general, how big is the untapped market for your software
training courses if offered over the Internet?
 Group questions in logical order
 All questions about one subject together, with transitional
phrases in between, “Now I’m going to ask you about agricultural
applications of GPS systems...”
13
Survey Best Practices:
Question Order (cont.)
 Demographics Questions
 Put last—these are less sensitive to prior questions
 Seem nosy if put first
 Rely on standard approaches for assessing
 http://www.norc.org/GSS+Website/
 The Process of Survey Design
 Use Backwards Marketing Research to decide what is
“need to know”
 Draft the survey
 Pretest for time, clarity, variability in responses
 Revise and retest
 Field the survey and keep an eye open for problems
14
15
Survey Best Practices:
Choosing a Survey Method






Mail, phone, web, in person?
Cost
Complexity of inquiries (branching)
Need for aids
Issue sensitivity
Control over sample
16
Web and Telephone
 Web surveys now dominate. To compare web,
in person, phone, mail, see
 http://knowledge-base.supersurvey.com/
17
17
Free to Fuqua students: Qualtrics
 http://www.qualtrics.com/duke#submit
 Set up an account
 Build surveys
 Allows for complex designs
 Available to you during this course
18
Multi-Attribute Attitude Model
(MAAM)
 Liking for a product as a whole = sum of liking for
component parts
 Attitude toward brand j = (sum from i = 1 to n for
salient attributes)

 Importance of Attributei * Evaluationij
 Importance
 0 – 100 (allocate 100 points across attributes)
 Rating on 1 (unimportant) to 7 (very important) where 0
undefined but implicitly entirely unimportant )
 Evaluation of brand j on attribute I
 -4 = poor to +4 = excellent
19
MAAM and SUVs
Attribute
Sporty Styling
Handling
Cost
Ruggedness
Off-Road Ability
Total Attitude
Importance
1=unimp,
7= important
6
5
2
4
2
Land
Rover
RAV
Land Rover
RAV
Brand Evaluation
-4 = poor, +4 =
excellent
Imp*Eval Imp*Eval
1
2
6
12
0
1
0
5
-2
3
-4
6
4
2
16
8
2
4
4
8
22
39
20
Diagnostics of Advantage
21
Measure Types Revisited
 Nominal (Unordered Categories)
 Just need unique number for each category
 Ordinal: ranking scale, intervals not assumed equal
 Interval: Intervals assumed equal, zero is arbitrary
 Ratio: Intervals assumed equal, zero means zero
 To multiply X * Y, (e.g., importance * evaluation), both X and Y must
be on ratio scales.
 If X1*Y1 > X2*Y2 (XYbrand 1 >XYbrand 2), it does NOT follow that
(X1+a)*Y1 > (X1+a)*Y2….
 e.g., 2*2 > 2*(-2), but (2-4)*2 < (2-4)*(-2)
 To say % change in Y, Y must be on ratio scales
22
More on Scaling
 To multiply importance x evaluation for each
attribute, both must be on ratio scales
 0 on scale must be 0 of underlying quantity
 Importance unipolar (all positive). Completely
unimportant = 0 weight
 Evaluation bipolar (negative to positive). To
multiply, must code “neutral” as zero.
23
Improper Rescaling
Attribute
Sporty Styling
Handling
Cost
Ruggedness
Off-Road Ability
Total Attitude
Importance
-3 =unimp,
+3 = imp
2
1
-2
0
-2
Land
Rover
RAV
Evaluation -4 =
poor, +4 =
excellent
1
0
-2
4
2
Land Rover
Imp*Eval
2
1
3
2
4
RAV
Imp*Eval
2
0
4
0
-4
2
Diff
4
1
-6
0
-8
-9
2
1
-10
0
-4
-11
I got these by subtracting 4 from the values three slides back
24
Consumer Attitudes
 We want to be able to predict consumer behavior
 However, instead of examining behavior directly (e.g.,
choice modeling), we often measure attitudes
because…
 Measuring attitudes is sometimes easier than observing
choice
 Attitudes are more diagnostic
 Attitudes are sometimes easier to interpret
 Attitudes can be reasonable predictors of behavior
 Attitudes toward products or brands typically derive
from beliefs, actions, and perceptions
25
Types of Attitude Scales
 Semantic differential
 Colgate Combo is:

low quality

unappealing __:__:__:__:__:__:__ appealing
__:__:__:__:__:__:__ high quality
 Constant sum (e.g., Importance)
 Purchase intent
 Likert scale (Agree-Disagree)
26
Recap
 Survey Design: responses constructed on the spot
 Moving parts of a good survey Population definition,
choosing a survey method, determining what information
needed
 Order of questions
 Attitude Measurement & multi-attribute attitude model
 To multiply or examine percentage differences, data
must be on ratio scales
27
Sampling Terminology




Elements -- Units about which information is sought
Population -- Elements we want to generalize to
Census -- Collect data on all elements in population
Sampling Frame -- At each stage, list of all
elements
 Sample – Collect data on subset of elements in pop.
28
Sampling Process
 Define population
 Elements, extent, time
 Identify a good sampling frame
 costly to create for yourself
 Determine sample size
 budget, accuracy needs
 Select sampling procedure
 way to select elements from the frame
 Physically select sample
29
Probability Samples
 Each element in population has known, nonzero chance of
being sampled
 Simple random sample: all elements have 1/n chance of
being sampled (e.g., cold caller)
 Systematic sample: start with randomly selected element
and take every nth element (e.g., teams in this class)
 Cluster sampling: pick groups of elements (city blocks,
census tracts, schools) then randomly select n elements from
each cluster
 Stratified sampling: divide frame into strata according to a
characteristic (e.g., gender), then sample randomly from
each strata
30
Complex Sampling Procedures
 Simple random sampling almost never
used in practice
 Stratified Sampling -- Lowers error
 Cluster Sampling -- Lowers cost of
getting frames and of data collection
31
Stratified Random Sample
 Have frames sorted on some stratification
variable believed to influence the variable
you are estimating.
 Lower variance within each subgroup than
across population in general
 By ensuring that each subgroup is
represented in right mix, extreme overall
means less likely -- i.e., smaller std. error.
32
Steps for Stratified Random Sample
 Divide Population into mutually exclusive and
exhaustive categories.
 Decide what sampling fraction f = n / N to
use.
 Draw an independent simple random sample
of size f * N(stratum) from each stratum.
 Compute stratum mean for each
 Estimate overall pop mean as weighted
average of stratum means
 Estimate SE as weighted combo of SEs in
each
33
Cluster Sampling
 Typically “clusters” are geographic territories.
 Start with list of clusters, randomly select
subset, and survey only subset.
 Cheaper travel cost, cost per interview
 Loss of effective sample size if people in
cluster more alike than if in different cluster
34
Non-Probability Samples
 Convenience
 Judgment
 Pick especially informative elements
 Quota
 Sample matches population on key control
characteristics correlated with behavior under
study.
 Match only really matters for control variables
related to thing you are trying to estimate.
35
Sampling quiz:
 Proportion of WEMBA students interested in
changing jobs? Sample of 50







How would you do:
Convenience Sample
Judgment
Quota sample (by male / female)
Simple random
Systematic
Stratified (by male / female)
36
Key Sampling Takeaways
 Probability v Non-Probability Samples
 For Probability Samples, Standard Error is
the measure of precision
 Precision increases with square root of N
 More precision with Stratified if and only if
stratifier is correlated with thing estimated
 Same principal for Quota samples. Quotas
only help if correlated with variable
37
Experiments
 Best way to test causal hypotheses
 Independent Variable = hypothesized cause
 Manipulated by the researcher/manager
 Example: Send a color or black and white
brochure
 Dependent Variable = effect
 Measured (observed) by researcher/manager
 Example: New accounts secured
 Random assignment of subjects to conditions
 Example: receive color or receive b&w brochure
38
Pre-experimental Designs




One group, after-only design
One group, before & after design
Unmatched control group design
Matched control group design
 All have threats to validity not present in a
true experiment with random assignment
to treatments.
39
Validity
The strength of our conclusions
 i.e., Is what we conclude from our experiment correct?
Threats to Validity
 History: an event occurring around same time as treatment
that has nothing to do with treatment
 Maturation: people change pre to post
 Testing: pretest causes change in response
 Instrumentation: measures changed meaning
 Statistical Regression: Original measure was due to a
random peak (SI Cover Curse) or valley
40
One Group After Only
We propose a change in MBA Core, to move Finance
and Marketing up to Term 2 from their position in Term
3. One major motive for this is that students interview
for internships in Term 3, and if they want jobs in
marketing or finance, they have no background at the
time of the interview. Thus, we perceive that we are at
a competitive disadvantage because those courses
are in Term 3.
EG
X
O (Mean = 50%)
X = Marketing Term 3, O = Did/Did Not Get Desired Internship
Key: Lacks a baseline, so worthless.
41
One Group Pre-Post Design
 Breckenridge Brewery wants to assess the efficacy of
TV ad spots for its new amber ale.
 Time 1 (O1): Duke undergrads are brought to the lab
and asked to rate their frequency of buying a series of
brands in various categories over the past week. The
list includes Breckenridge Amber Ale. Mean = 0.2
packs per week.
 Time 2 (X): Two weeks of ads for Breckenridge Ale.
 Time 3 (O2): Same Duke undergrads brought back to
lab to rate frequency of buying same set of brands over
past week. Mean = 1.3 packs per week.
 1.3 - 0.2 = 1.1. We attribute an increase of 1.1 packs
per week to the ad.
42
Online Investor Performance
 X = brick and mortar brokerage
customer moves online to trade in 1999
 O = Annualized turnover
 1998 – 40% annualized turnover
 2000 – 100% annualized turnover
 Did going online cause people to trade
more actively?
 Threats with one-group pre-post?
43
Quasi-Experimental Designs:
Interrupted Time Series
 Same as one-group pretest posttest, but
observations at many points in time before
and after key treatment for same people:
 EG O1 O2 O3 X O4 O5 O6
 Extra time periods help control for history,
maturation, testing. “Quasi-experiment”
44
Online Investor Performance
Gross Returns
10
Net Returns
5
36
33
30
27
24
21
18
15
12
9
6
3
0
-3
-6
-9
-12
-15
-18
-21
-24
-27
-30
-33
0
-36
Cumulative Market Adjusted Return (%)
15
-5
-10
Event Month (0 = month of first online trade)
45
Portfolio Turnover
120%
Size-Matched
100%
Online
80%
60%
40%
20%
24
21
18
15
12
9
6
3
0
-3
-6
-9
-1
2
-1
5
-1
8
-2
1
0%
-2
4
Annualized Turnover (%)
140%
Event Month (0 is month of first online trade)
46
2 Groups: Unmatched Control Group
(Effect of Prior Knowledge on Search)
Hypothesis: People with little knowledge about cars search
less online because they are overwhelmed
100 Durham residents who are in the market for a car
EG
CG
X1 (Auto Shop Course) O1 (6 hrs online)
--------------------------------------------------------X2 (Electronics Course) O2 (3 hrs online)
47
2 Groups: Matched Control Group
(True Experiment)
EG
R X1 (Auto Shop Course) O1 (6 hrs)
---------------------------------------------------------
CG
R X2 (Electronics Course) O2 (3 hrs)
Control for Selection Threat.
Key Point:
For causal research, chance (not respondent) must
determine respondent assignment to condition.
48
Breckenridge Brewery Ads
 Breckenridge Brewery wants to assess the efficacy of TV
ad spots for its new amber ale.
 Time 1 (O1): Duke undergrads are brought to the lab
and asked to rate their frequency of buying a series of
brands in various categories over the past week. The list
includes Breckenridge Amber Ale. Mean = 0.2 packs per
week.
 Time 2 (X): Two weeks of ads for Breckenridge Ale.
 Time 3 (O2): Same Duke undergrads brought back to lab
to rate frequency of buying same set of brands over past
week. Mean = 1.3 packs per week.
 1.3 - 0.2 = 1.1 increase in number of packs per week.
49
2-group Before-After Design
• Now add a randomly assigned “Control”
group with mean scores O1 = 0.3, O2 = 0.5.
Experimental
O1 X O2
Control
O1
O2
O1
O2
O2 - O1
Difference
0.2
1.3
1.1
0.3
0.5
0.2
50
Factorial Designs
 Independent Variable:
 Factor manipulated by the researcher
 Dependent Variable:
 Effect or response measured by researcher
 Factorial Design:
 2 or more independent variables, each with two or
more levels.
 All possible combinations of levels of A & levels of
B.
51
Oreo Promotion Experiment
 Kroger: Supporting a discount on Oreo
cookies







Factor A: Ads in local paper
a1 = no ads
a2 = ad in Thursday local paper
Factor B: Display location
b1 = regular shelf
b2 = end aisle
52
Oreo Promotion Experiment
(Expenditures/customer/2 wks)
a1 = no ads
a2 = ads
Row Ave
b1 =
regular shelf
.60
.90
.75
b2 =
end aisle
.85
.95
.90
Col. Ave
.725
.925
53
Sales of Oreos on Promotion as function of
Local Advertising, Display Location
54
Oreo Example, No Interaction
 Main Effect of A (Ads)?
 Main Effect of B (Display Location)?
 No AxB (say A by B) interaction. Effect of
changing A (Ads) is independent of level of
B (Display Location). Sales go up by $0.30
when you advertise, regardless of location.
 Implies that Ad & Display decisions can be
decoupled…they influence sales additively.
55
Managerial Implications of
Interactions
 If two controllable marketing decision
variables interact (e.g., advertising x display),
implication is that you can’t decouple
decisions; must coordinate.
 If A is a controllable decision variable and B is
a potential segmentation variable (e.g., ads x
urban/suburban), interaction means that
segments respond differently to this lever.
56
Interactions and segmentation
c
Coupons have a bigger effect in the suburbs
Psychology of Consumers
Exposure, Attention, & Perception
57
Takeaways for the Day





Survey Design: responses constructed on the spot?
Order of questions
Attitude Measurement & multiattribute attitude model
Probability v Non-Probability Samples
More precision with Stratified if and only if stratifier is
correlated with thing estimated
 Threats to validity in pre-experimental and quasiexperimental designs
 Factorial Designs – Main effects and interactions
 2 marketing tactics interact  coordinate
 Marketing tactic interacts with customer classification
implies classification a potential basis for
segmentation…different sensitivities to some marketing
mix variable
58