Transcript CAGE Conference A quick tour of the HH fallacy

A Cold Shower for the Hot Hand
Fallacy
Joshua B. Miller (Bocconi University and IGIER)
with Adam Sanjurjo (University of Alicante)
CAGE Conference
A QUICK TOUR OF THE HH FALLACY
(12 SLIDES)
Hot Hand: the belief and the fallacy
• An individual believes in the hot hand if s/he:
– considers a good outcome as more likely than
usual after a streak of good outcomes, or
– regards "hot" streaks of good outcomes to be too
long or too frequent
• And therefore attributable to some unobserved or
transient factor (e.g. human agency).
• The belief becomes a fallacy in an
environment in which a good outcome always
has the same chance of occurring.
Some history
• Research from the 1960s & 1970s: e.g. Wagenaar 1972
– People are not good at discriminating random (iid)
sequences of outcomes from non-random ones.
– People expect negative recency from random
sequences (modal belief: P(H|T)=P(T|H)=0.6 )
– When the generating process is unknown, people
mistakenly judge random sequences to be nonrandom in the absence of negative recency:
• streaks are "too long" to be due to chance
• the distribution does not balance in the near term. (i.e.
law of small numbers, Kahneman & Tversky, 1971)
Early "hot hand fallacy" results
1. Evidence is typically from artificial environments
(lab, casinos)
– Does it matter in more naturalistic decision making
environments?
2. Decision makers are amateurs
3. The decisions may have real consequences for
whomever the decision effects, but the marginal
cost of indulging incorrect beliefs isn't high.
– the opportunity cost of betting on black isn’t high
assuming you are already gambling; if there is any
“irrationality” it is gambling to begin with.
1985
• Gilovich, Vallone & Tversky (1985) had the novel insight
that the sport of basketball was an excellent potential
exhibit:
– Clean and relatively controlled decision environment.
– Experts (coaches and players) believe that streaks occur because
sometimes an individual player exceeds their average ability
level ("hot hand", "in the zone“, “in rhythm”)
– The lab results suggest people may over-infer from streaks; are
streaks exceptional when controlling for a player's ability?
– There is a real economic cost to the experts for believing in the
hot hand (ball allocation, substitution, defensive strategy, etc.).
GVT 1985
•
•
•
Validated that observers and players believe in the
hot hand.
Collected data from in game shooting situations,
free throw shooting, and a controlled shooting
experiment.
Found that shooting performance was consistent
with individual players always shooting with a fixed
probability of success. (i.e. iid Bernoulli trials)
Robustness of GVT results
• Subsequent failure to detect the hot hand:
– Larkey, et al. (1989), Wardrop (1999), Koehler & Conley
(2003) , Avugos, Bar-Eli, Ritov, Sher (2013)
• Persistence of beliefs & behavior:
– Coaches and players believe and act on it (Rao 2009 , Cao
2011, Attali 2013)
The Consensus
• "Many researchers have been so sure that the
original Gilovich results were wrong that they
set out to find the hot hand. To date, no one
has found it"
Thaler & Sunstein, Nudge (2008)
• "The hot hand [in basketball] is a massive and
widespread cognitive illusion."
Daniel Kahneman, Thinking , Fast and Slow (2011)
Influence of the hot hand fallacy
• With diverse applications:
– Camerer (1989); Brown and Sauer (1993);
Barberis, Shleifer, and Vishny (1998); Rabin (2002);
Croson and Sundali (2005); DellaVigna (2009);
Guryan and Kearney (2008);
• More recently:
– Rabin & Vayanos, (2010) have a model that
generates the HH fallacy and can explain financial
puzzles such as fund flows and active fund
puzzles.
Importance of the hot hand fallacy
• Counter-intuitive psychologically.
– Confidence and personal efficacy is influenced by
performance, and there is evidence they can
feedback into performance (Bandura 1982,
Solomon 2001)
– Noise in preparatory neurons vary and can explain
variation in movement (Churchland 2006)
– Natural fluctuations in attention/concentration
But there are simpler reasons to doubt this....
HH fallacy is a severe challenge for rationality
• HH fallacy is found in experts in their domain of
expertise, with high stakes.
– While systematic mistakes among experts is not unheard
of, e.g. the “Moneyball” anomaly (Lewis 2004)
– They tend to go away, e.g. Hakes & Sauer 2006
• HH fallacy goes further, because the HH belief is
robust to learning and advice:
– "Amos Tversky used to describe the robustness of the belief in the hot
hand by saying that he has been in a thousand arguments over this
topic, won them all, but convinced no one" (Gilovich, 2002)
Our study
• One should be skeptical of this degree of
irrationality
• We revisit HH fallacy in domain where the
evidence is considered the strongest and most
readily generalizable.
• The evidence is not as strong as it seems, and
this necessitates a discussion of the
particularities of basketball shooting data (and
an experiment)...
Validity concerns for game data
• "Live ball" shooting:
– confounds: shot distance, angle, type; defense,
incentives, fatigue, etc...
– Shot frequency (fundamental)
– Aggregation across games & shot selection bias
• Free throw data:
– Shot selection bias
– Hot hand or cold hand?
– Recent Evidence: Arkes 2010, Yaari & Eisenmann
2011
GVT's controlled Experiment
• GVT recognized some of the aforementioned
validity concerns.
• Their results rely on a controlled shooting
experiment.
• Their design and analysis has severe
limitations that persist into the subsequent
literature.
Design
• Design:
– A distance is determined where player is believed
to hit 50% of shots.
– Players take 100 shots, moving in an arc
– Payment
• paid fixed $2
• For each shot, players bet on own shot choosing either:
– "high" [variance] bet ($0.05 Hit, $0.04 Miss), or
– "low" [variance] bet ($0.02 Hit, $0.01 Miss)
GVT Empirical Strategy: 4 statistics
1. Serial correlation (lag 1)
2. Runs:
–
–
Streaky: 10 shots, 3 runs: 1100001111
Anti-Streaky: 10 shots, 8 runs : 0110101101
Problems:
• Neither measure hotness
• Both measure first order serial dependence :
GVT Empirical Strategy: 4 statistics
3. Conditional probability
– E.g. Prop(hit| 3+ hits) vs. Prop(hit| 3+ misses)
– Could be driven by the cold hand.
4. Distribution of performance in four-shot
window (test of statitionarity)
OUR STUDY:
Design Overview
• Phase 1:
– 10 players shooting for money.
– Players complete multi-item questionnaire.
• Inter-Phase
– Between Phase 1 & Phase 2
– Subjects in Spain and Italy watch videos and predict
player performance.
• Phase 2:
– 6 months after Phase 1.
– 8 players return and shoot for money
Our Shooters
Phase 1:
• 10 semi-professional male basketball players
– Santo Domingo de Betanzos (former/current)
• In the 5th of 7 categories (can move to 4th next year)
– Recruited at practice
• All players were interested in participating but some
had scheduling conflicts so could not
Phase 2:
• 8 returned (+3 ad-hoc additions)
THE SHOOTING TASK (VIDEO)
Design: Details
1. Shooter warms up
2. Shooter always shoots from same spot
– calibrated so they are expected 50% (3-pt)
3. 300 shots
– Calibrated to avoid fatigue concerns.
4. No betting considerations for shooter
5. Paid 5 € show up fee + 6 € for each make
from 10 randomly selected shots
3 hot hand statisitics
• Hot Streak Length
– Length of the longest run of hits (in 300 shots) :
• Hot Streak Frequency
– % of shots taken during a hit streak of 3+ :
• Hot Streak Momentum
– % hits if hit 3+ :
The null reference model
The null reference model
Distribution of statistics under H0
• In principle, there is an exact distribution of each test
statistic under the null hypothesis of exchangeability:
– Calculate the test statistic for every permutation of
the data, each permutation is equally likely.
• We can approximate this distribution to arbitrary
precision with Monte Carlo permutations of shots.
• Exchangeability is consistent with more than Bernoulli
– We find that our significance levels are the same or more
conservative for Bernoulli shooters.
• When aggregating we use subject and session strata (to
avoid the influence of good day/bad day effects)
STRATEGY TO DETECT THE
EXISTENCE OF THE HOT HAND
Selecting the hot player
Phase 1:
1. Multi-item questionnaire of teammates:
–
–
If hot hand belief has any validity, teammate opinion
should predict the hottest player.
If the task has any external relevance, in-game hot hand
should predict the hot hand in the shooting task.
2. Hot Hand Statistics in the 300-shot task.
3. Student performance in a laboratory prediction task
(watching videos of the player's 300 shots)
–
If observers have hot hand beliefs and a player is truly
hot, observers should outperform chance in a prediction
task.
Identifying the hot player
Phase 2:
1. Run additional sessions for the "hot player" in
Phase 2.
–
If the hot hand is a player specific trait, it should persist
across time .
Phase one statistics...
Bin(7,8)=.035
.7
.2
Phase 1
1
Momentum (H M)
1
.15
8
5
5
.5
3
2
8
3
.4
2
6
8
.45
.55
.5
7
2
Runs (R)
8
53
140
8
3
.4
150
10
7
.35
160
12
1
Length (H L)
.3
.2
.15
.1
.05
0
4
6
.3
.05
4
2
4
5
6
130
1
4
6
120
6
4
Observed
7
7
.1
Observed
.6
Frequency (H F)
5
6
7
Median
8
9
130
135
140
Median
145
150
RC is hot
• (Stats) RC was the hottest player in phase 1
• (Teammates) RC was selected by teammates
as hottest in games.
• (Human Detectors) Observers predicted well
with him, significantly better than chance.
Will this predict what he does 6 months later?
Bin(6,6)=.016
Player RC (subject 1)
Not
Cold
hand!
RC not the cold hand
• Linear probability model of fg%
– Base category: Hit 3+
PANEL OF 8 PLAYERS: PERSISTENCE
.25
.7
Panel (averaged over sessions)
Momentum (H M)
.4
.5
.15
.3
.05
.1
Observed
.2
.6
Frequency (H F)
.15
.2
.25
.4
.45
150
.1
14
.05
.55
.6
.65
Runs (R)
145
135
6
8
140
10
Observed
12
Length (H L)
.5
6
8
10
Median
12
140
142
144
146
Median
148
150
Aggregate analysis
More data: Jagacinski, Newell, Issac (1979)
More data: GVT
Overall
Discussion
• Hot hand exists
• Hot hand is substantial in some players.
What about the hot hand fallacy?
• The basketball result is the central exhibit of the HH fallacy
being general and powerful.
– It can no longer be used.
• More generally, evidence appears to be thin that mistaken
belief in the hot hand is powerful and costly.
• Our view: Rabin has it right, HH belief is a product of a
(conditionally) rational inference if you are uncertain about
the possibility of a regime change (when in fact it cannot
happen)
Conclusion