Transcript slides

True and Error Models of
Response Variation in
Judgment and Decision Tasks
Michael H. Birnbaum
Overview
I review three papers that are available at my Website
that involve application and evaluation of models of
variability in response to choice problems.
•Two papers are co-authored with Jeff Bahra on with
tests of transitivity, stochastic dominance, and
restricted branch independence.
•Our findings consistently rule out assumptions of iid
that are required in certain models, such as the
approach of Regenwetter, Dana, and Davis-Stober
(2011) Psych Review.
•Transitivity is often satisfied, but a few show evidence
of intransitive preferences.
•No individual satisfied CPT or the priority heuristic.
Testing Algebraic Models with
Error-Filled Data
• Algebraic models assume or imply
formal properties such as stochastic
dominance, coalescing, transitivity,
gain-loss separability, etc.
• But these properties will not hold if data
contain “error.”
Some Proposed Solutions
• Neo-Bayesian approach (Myung, Karabatsos,
& Iverson.
• Cognitive process approach (Busemeyer)
• “Error” Theory (“Error Story”) approach
(Thurstone, Luce) combined with algebraic
models.
• Random preference model: choices
independent; no errors: variability due to iid
sampling from mixture. Loomes & Sugden.
Variations of Error Models
• Thurstone, Luce: errors related to separation
between subjective values. Case V: SST
(scalability).
• Harless & Camerer: errors assumed to be
equal for certain choices.
• Sopher & Gigliotti: Allow each choice to have
a different rate of error, assumed transitivity.
• Birnbaum proposed using repetitions within
block as estimates of error rates. Birnbaum &
Gutierrez, 2007; Birnbaum & Schmidt, 2008.
Basic Assumptions of TE
model (2 errors model)
• Each choice in an experiment has a true
choice probability, p, and an error rate,
e.
• The error rate is estimated from (and is
the “reason” given for) inconsistency of
response to the same choice by same
person over repetitions
One Choice, Two Repetitions
A
A
B

B
pe 2  (1 p)(1 e)2
p(1e)e  (1 p)(1 e)e
p(1 e)e  (1 p)(1 e)e
p(1 e)2  (1 p)e 2

Choices are not Independent
• In this model, choices are not
independent, in general.
• If there is a mixture of true preferences,
there will be violations of independence.
• This contrasts with the assumption of iid
used by Regenwetter and colleagues.
Solution for e
• The proportion of preference reversals
between repetitions allows an estimate
of e.
• Both off-diagonal entries should be
equal, and are equal to:
(1  e)e
Estimating e
Estimating p
Testing if p = 0
True and Error Model to
Individuals
• When applied to individuals, it is assumed
that each person has a “true” set of
preferences within a trial block.
• True preferences might differ between blocks,
if the person has a mixture. If so, violates
independence.
• A mixture could arise if a person’s
parameters change in response to
experience.
By Testing Individuals…
• we can tailor the experiment (or devise
a fish net) to “catch” violations that
otherwise slip through a static study of a
property.
• we can see if people “learn” from
internal feedback and change their
behavior.
• we can test if a model that holds for one
property can predict the results of other
tests.
Recent Studies with Jeffrey
Bahra
• Tested participants with many
replications.
• Tests of transitivity
• Basic tests of critical properties,
including stochastic dominance,
coalescing, LCI, UCI, and others.
• Today: RBI and SD (+ transitivity)
Stochastic Dominance
A: 10 tickets to win $10
5 tickets to win $90
85 tickets to win $98
B: 5 tickets to win $10
5 tickets to win $12
90 tickets to win $99
Each person received two such problems in each
repetition block of 107 choice problems. Blocks were
separated by at least 50 unrelated choice problems.
SD is a critical property
• This test of stochastic dominance lies outside
the probability simplex on three branch
gambles.
• CPT with ANY strictly monotonic utility
function and decumulative weighting function
must satisfy stochastic dominance in this
choice.
• We don’t need to estimate any parameters or
assume any particular functions to refute
CPT.
Non-nested Models
Testing CPT
TAX:Violations of:
• Coalescing
• Stochastic
Dominance
• Lower Cum.
Independence
• Upper
Cumulative
Independence
• Upper Tail
Independence
• Gain-Loss
Separability
Testing TAX Model
CPT: Violations of:
• 4-Distribution Independence
(RS’)
• 3-Lower Distribution
Independence
• 3-2 Lower Distribution
Independence
• 3-Upper Distribution
Independence (RS’)
• Res. Branch Indep (RS’)
Critical Tests: LS Models
• CPT and TAX satisfy transitivity
• LS Models violate transitivity (includes PH)
• LS Models satisfy priority dominance,
integrative independence, and interactive
independence. This properties systematically
violated. See my JMP 2010 article.
• PH satisfies SD in these tests and it also
violates RBI in the same way as CPT.
Restricted Branch
Independence
• Weaker version of Savage’s “sure thing” ax.
• 3 equally likely events: slips in urn.
• (x, y, z) := prizes x, y, or z, x < y < z
• RBI:
(x, y, z) f (x', y', z)

(x, y, z') f (x', y', z')
TAX, CPT, PH Violate RBI
• 0 < z < x' < x < y < y' < z'
• (x, y) is “Safe”, S
• (x', y') is “Risky”, R
• (z, x, y) f (z, x', y') 
wLu(z) + wMu(x) + wHu(y) >
wLu(z) + wMu(x') + wHu(y')
TAX: SR' Violations
SR':
S = (x, y, z)
S' = (x, y, z')
R = (x', y', z) AND
R' = (x', y', z')
w L < u(y') - u(y) < wM
w M u(x) - u(x')
wH
S = (z, x, y) vs R = (z, 5, 95)
RBI distinguishes models of
RDM
•EU and SWU (Edwards, 54) imply RBI
•Original prospect theory implies RBI
•Cumulative Prospect theory violates RBI
•CPT with inverse-S weighting function
implies RS’ pattern of violations
• TAX model violates RBI with SR’ pattern
Testing iid Assumptions
Each person’s data: rows represent trial blocks
and the columns represent choice problems.
Smith & Batchelder (2008) technique: random
permutations within columns for each person. We
then calculate two statistics on original data and
on 10,000 simulations of the data. Variance of
preference reversals and correlation between
mean preference reversals and difference
between repetitions.
A Test of Independence &
Stationarity
• Within each subject, calculate the number of
reversals between each pair of repetitions.
107 choices.
• 20 reps, so 20*19/2 = 190 pairs of reps.
• Correlate the average no. reversals with the
difference in reps. For 59 participants, mean
r = 0.69. Only 6 were negative. Similar
results for Regenwetter et al data.
• Second test: Variance of no. reversals.
Tests of Independence
Transitivity Findings
•
•
•
•
Results were surprisingly transitive.
True and Error Model Fit data fairly well.
Data violate independence of choices.
People differ in true preferences and
people differ in “noise” levels.
• No one satisfied the priority heuristic
• One person satisfied linked viols of
transitivity consistent with LS models.
Very Few Intransitive Cases
• No one showed pattern predicted by PH
in all three designs.
• Of the 59*3 = 177 Matrices, perhaps 4
show credible evidence of intransitivity.
• This change of procedure did not
produce the higher rates of intransitivity
conjectured.
Allais CC Paradox (JMP’04)
• Choose Between:
A = ($40, 0.2; $2, 0.8)
B = ($98, 0.1; $2, 0.9)
• Choose Between:
C = ($98, 0.8; $40, 0.2)
D = ($98, 0.9; $2, 0.1)
• Many Choose B > A and C > D
Analysis of the Paradox:
($40, .2; $2, .8)
f ($98, .1; $2, .9)
 (Coalescing)
($40, .1;$40, .1;$2, .8) f ($98, .1;$2,.1;$2, .8)
 (RBI)
($40, .1;$40, .1;$98, .8) f ($98, .1;$2,.1;$98, .8)
 (Coalescing)
($98, .8; $40, 0.2) f ($98, .9; $2, .1)
Four Theories Compared
RBI holds
RBI fails
(**cancellation)
Coalescing
EU, CPT**,
OPT*
holds
(*combination)
Coalescing
OPT
fails
CPT
Inverse-S =>
RS’
TAX, RAM
SR’
Error Models & EU
• One error model: P(SR’) = P(RS’)
• Two error model allows P(RS’)>P(SR’)
• Two error model: P(R’) > 1/2 implies
P(R) > 1/2
• Four error model allows above
• All models imply P(SR) = P(RS);
P(SS,SR’)=P(SR’,SS), etc.
Results (refute 4 error EU)
SS’
SR’
RS’
RR’
SS’
36
7
21
3
SR’
5
4
1
3
RS’
18
4
69
11
RR’
1
2
12
14
Results (split form RBI)
SS’
SR’
RS’
RR’
SS’
42
24
4
1
SR’
17
21
2
16
RS’
7
4
14
7
RR’
1
6
6
39
Summary
• Most people conform to transitivity
• A very small number do not, but not
consistently in all three designs as
predicted by LS models.
• Data appear to violate the assumptions
of random utility model; in particular,
they show evidence against the
assumptions of i.i.d.
Summary (cont’d)
• No one satisfied CPT, except those
consistent with EU
• No one satisfied PH
• People appear to change between
blocks. We suspect that parameters
change systematically during the study.
• Suspect TE model oversimplified and
incomplete.
Available at my Website
• Tests of LS models (JMP 2010)
• Psych Review Comment RP Model (2011)
• Reanalysis of Regenwetter et al data testing
iid, in JDM.
• Birnbaum & Bahra: Testing transitivity in
Linked Designs (submitted)
• Birnbaum & Bahra: Testing SD and RBI
(submitted)
• Soon: Birnbaum & Schmidt Allais paradoxes
with 4 errors model