Transcript judgmental adjustments

Behavioral Issues in
Forecasting
Paul Goodwin
University of Bath, UK
1
Why forecasting is important
• Short term: production planning, staff work scheduling
inventory control, materials purchasing, stock market
decisions.
• Long term – new production facilities, new infrastructure
projects, training new staff.
• Very long term –probably not worth forecasting: use
methods like scenario planning.
2
All forecasts involve judgment…
• Choice of forecasting method
• How much past data to use
• Which variables to include in your model etc.
However, we are concerned here with the direct use of
judgment in forecasting
i.e. forecasts based wholly on judgment or judgmental
adjustments to statistical forecasts
3
Role of direct judgment in forecasting
• Judgmental extrapolation of time series
60
50
40
30
20
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
• Judgment based on non-time series information
4
Role of direct judgment in forecasting
• Judgmental adjustment of statistical forecasts
• Probability judgments.
5
Topics
• Ten cognitive biases in judgmental forecasting and
methods that might overcome them.
• Future research challenges.
6
Management judgment can bring benefits to
forecasts
1. It can take into account special events:
e.g.
- a disease outbreak,
- a government initiative
- a sales promotion
2. It can compensate for a lack of relevant
historical data.
7
Management judgment can bring benefits to forecasts
3. It can allow for situations where managers have some
control over the variable to be forecast
4. It may produce more acceptable forecasts because:
-managers have a sense of
ownership of the forecasts
- complex statistical methods may lack
transparency and hence lack
credibility…..
8
British Gas. Daily forecast of gas demand
(Taylor & Thomas, 1982)
9
But judgment can also be
problematical……
10
Test your judgment quiz
Questions 1 to 3
11
Biases in judgmental forecasts and possible
solutions
12
1. Anchoring and adjustment
• When making a forecast people often start with an initial
value (the anchor) and then adjust from this.
Danger
Estimates tend to be too close to the anchor -even if it's
an implausible value...
13
How old was Gandhi when he died?
Source: Strack & Mussweiler
One group asked was he older or younger than 9?
Then asked to estimate his age at death
Mean answer 50
Another group asked was he older or younger than 140?
Then asked to estimate his age at death
Mean answer 67
Correct answer: 78
14
Test your judgment quiz
• Answer to question 1
Population of Egypt in 2006
= 72.6 million
15
Anchoring can
cause people to
underestimate
upward trends
because they
stay too close to
the most recent
value…..
No. of cases
Effect on forecasts of anchoring
250
245
240
235
230
225
220
215
210
205
200
Actual
Judgmental
forecast
Mar
Apr
May
Jun
16
Test your judgment quiz
2001 Q3:
Actual =£4.323
billion
Q2 Answer:
4.500
4.000
3.500
3.000
2.500
2.000
1.500
1.000
0.500
0.000
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
£ billions at 2002 prices
UK Net borrowing by consumers in real terms
1993
1994
1995
1996
1997
1998
1999
2000
2001
Linear regression line would forecast £4.800 billion
17
Underestimation can be particularly
severe for exponential trends…
9000
6000
3000
0
1
2
3
4
5
6
7
8
9
18
A piece of paper is 0.193 mm (0.0076
inches) thick.
If the paper could be folded 40 times, use
your judgment to forecast the thickness of
the folded sheet.
19
Answer
Mathematics shows that after 40 folds the paper’s
thickness will be:
0.193mm x 240
= 212,206 kilometres or 131,862 miles.
That’s more than half the distance to the Moon
It’s highly likely that you
underestimated this!
20
Test your judgment quiz
• Answer to question 3
Sales = 0.6 et where t = year
So in year 12 sales = 97,653 units
120000
100000
Sales (units)
•
80000
60000
40000
20000
0
0
2
4
6
8
10
12
14
Years
21
Solutions
1. Use statistical methods to extrapolate trends rather
than judgment.
2. Increasing the accessibility of anchor-inconsistent
knowledge mitigates the effect.
-Considering the opposite (i.e., generating reasons why
an anchor is inappropriate).
Mussweiler et al. (2000)
22
Test your judgment quiz
Question 4
23
2. Overconfidence
Overconfidence that an outcome will fall within an
estimated range….
E.g. 90% judgmental prediction interval
Lower limit
Upper limit
Most Likely
Actual
outcome
• Not clear what causes overconfidence. Anchoring
used to be the accepted explanation.
• Some evidence of cultural differences in its extent
(Meng et al. 2016).
• Ben-David et al. (2013) asked a large sample of U.S.
financial executives to produce 80% prediction intervals
of one-year-ahead stock market values
• They were so narrow that they only included the true
values on 36.3% of occasions
25
Answers to “Test your judgment”
Question 4
a) 6300 miles or 10,139 kilometres
b) 31, 241, 030
c) 1938 by Biro
d) 1876
e) 14000 feet or 4000 metres
26
Solution 1
• Augmentation: once you’ve estimated a range –double it!
E. g. Initial 90% prediction interval for sales:
40 to 60 thousand units
After Augmentation:
30 to 70 thousand units
27
Solution 2
• We are better at estimating a coverage probability for a given
interval than we are at producing an interval to have a given
probability
E.g. What is the probability that the interval 45 to 55 captures
Obama’s age when he became President?
rather than..
Estimate a 90% interval for Obama’s age when he
became President.
28
Test your judgment quiz
Question 5
29
3. Availability bias
• Forecast is over influenced by recent, easily recalled or
vivid events or events reported recently in the media…..
30
• Study by Johnson et al. (1993) found people willing to
pay more for airline travel
insurance covering terrorist attacks
than for deaths from all possible
causes
• Sales of earthquake insurance
are highest after an earthquake
when the risk is lowest.
They then decline as time passes
(while the risk gradually increases).
Test your judgment quiz
Answer to question 5
In the USA in 2007 the proportion of people (aged
12 or over) who were victims of robbery was
approximately
2 in every 1000
or 0.2%
(Source US Dept of Justice)
32
Test your judgment quiz
Question 6
33
4. Seeing patterns in randomness
34
100
6 monthly
cycle in
demand?
90
Demand (units)
80
70
60
50
40
30
20
10
0
1
3
5
7
9
11
13
15
17
19
21
23
25
Months
35
Rats handled randomness better than Yale
students
60% of time
40% of time
•Rat went to side where food appeared most frequently
– correct 60% of time
•Students tried to spot patterns
– correct only 52% of time
(see Tetlock, 2005)
36
5. The narrative fallacy
-People are brilliant at inventing explanations for
these random movements
“Our customers have been
stocking up in anticipation
of a price increase”
Demand (no of units)
60
50
40
Actual
Stats Forecast
30
20
10
0
1
11
Month
21
“OK last time they didn’t
stock up when they
expected a price increase
–but then interest rates
were high so holding
inventory was costly”
37
Even Nobel Laureates are not immune…
Output
per
worker
It turned out the
anomaly was a
result of
mistakes in
arithmetic
Robert Solow
developed a
hypothesis to explain
this ‘structural shift’ in
the US economy for
Capital per worker
1943-1949 38
• The dull statistical forecast can offer“Our
no new Harvard
-educated Sales
competition to the colorful, but often groundless
Manager is
tales of why the graph has swung inworking
a particular
miracles”
direction
45
40
35
Sales
30
25
20
15
10
5
0
0
5
10
Months
The exponential
smoothing forecast based
on α = 0.15
15
20 predicts little
change
-and so it gets adjusted.
39
Test your judgment quiz
Answer to question 6
It’s best not to adjust the
statistical forecasts as the
deviations from it are purely
random
40
Solution
• When identifying patterns in data, don’t
believe that you can do a better job than a
statistical forecasting system.
• Easier said than done..
41
Key research challenge
• How do we discourage managers from
over interfering with statistical forecasts?
42
Many companies judgmentally adjust a large % of their
statistical forecasts (Fildes et al., 2009)
Evidence from four companies:
100%
80%
Over 26000
SKUs
60%
40%
20%
0%
Pharmaceutical
company
Food
Domestic
manufacturer cleaning prods.
Retailer
43
How large are the adjustments people make?
Three Manufacturers
G1
Retailer
G2
12%12%
8%
Percent
8%
4%
4%
0%
-100%
0.00
0%
1.00
100%
2.00
Many small adjustments
200%
0.00
0%
1.00
100%
2.00
200%
A few very large + adjustments
44
Size of adjustments
Mean Improvement in
Absolute % error
20
Results
from one
company
15
10
5
0
-5
Smallest
25%
Q1 to
Median
Median to
Q3
Largest 25%
Size of adjustment
Small adjustments to software’s forecasts waste
time and often reduce accuracy
45
Test your judgment quiz
Question 7
46
6. Optimism bias
• So prevalent that
UK’s Department for
Transport has an
optimism bias
adjustment system
..though some of this
attributable to ‘strategic
misrepresentation’ or
lying
E.g. In an analysis of 258 transport
infrastructure projects completed
between 1927-1998 costs were
underestimated in 9 out of 10 cases
47
One reason for optimism bias:
Ignoring the base rate
• We focus on the specific characteristics of a future event
and ignore, or discount, the underlying statistical rate at
which the event occurs…
48
“We are 90% certain this product will be profitable.
• It’s been designed by an enthusiastic team.
• It’s the biggest investment we’ve made in R & D
• Competing products are nowhere near to being
launched.
• It’ll be backed by a huge advertising campaign”
The base rate: only 35 % of products launched in this
market make a profit.
49
Over optimism in the success of marriage?
Base rates:
• UK: 33% marriages between 1995 and 2010 ended in
divorce
(Source: Office of National Statistics)
• USA: Divorce rate in America for third marriages is
between 70% to 73%
(Source: Forest Institute of Professional Psychology, Springfield)
50
Test your judgment quiz
Answer to question 7
There is no exact answer but the base rate of 23% should
be given quite a lot of weight and an answer close to this
might be reasonable.
51
Solutions
• Get devil’s advocates to challenge your perspective.
• Make a list of reasons why the event you hope will occur,
might not occur.
• Obtain data on the base rate
E.g. Use reference class forecasting…
52
Drawing attention to base-rates:
Reference class forecasting
Identify a set of similar products (the reference class):
% of
Market Share
Products
0 to under 20%
45
20 to under 40%
25
40 to under 60%
20
60 to under 80%
10
Suggests that chances of market share of 40% or more are only 30%
53
7. False consensus
• False belief that because you like a product others will like it.
• In experiment in Italy forecasting sales of new handbags
based on 10 weeks sales data from a leading manufacturer of
leather goods.
• Management students invited to
judgmentally adjust statistical extrapolations.
-very strong liking for products led to optimism bias in sales
forecasts.
-disliking a product led to greater accuracy…
54
55
Overcoming false consensus
• Avoid asking people to specify whether they like a
product –this intensifies the bias (Marks and Miller,
1987).
• Ask people to think of reasons why the product might
not sell as well as they think.
• Provide people with market research data.
• Use forecasters who are neutral or who do not have an
extreme liking for a product.
56
8. Inconsistency
Danger:
Inconsistent use of information e.g. on
product promotions
57
Solution
• Judgmental bootstrap models…
We fit a regression model to the past judgments made by a
forecaster.
This captures how they use information to make their
forecasts.
The model should ‘average out’ their inconsistencies leading
to more accurate forecasts.
We then replace the person with the model and use it for all
future forecasts.
58
Judgmental bootstrap exercise
59
9. Information overload
E.g., combining effects of advertising, competition & price
• Reliability of judgment declines as people have to process
more information (Karelaia & Hogarth, 2008).
-though confidence in judgment increases (Zacharakis &
Shepherd (2001).
• Poor at handling negative linear relationships between cue
and the variable-to-be-forecast, and U and inverted-U
relationships, (Sniezek & Naylor, 1978).
• Poor at making accurate predictions when there is inter-cue
redundancy - people possibly add-in the effect of the
redundant cues.
60
Solution
• Decomposition -break judgmental task down
into smaller and easier tasks
61
• Forecast of market potential for product G for 2014 for
sales through allergists
= Forecast of prevalent cases in US [71,733,000 Data Monitor]
x % aged 20 to 64
x % diagnosed
x % with specific grass allergy
x % visiting allergist
x [% mild cases x % oral medication x % prescribe G
+% mod. cases x % oral medication x % prescribe G
+% sevr. cases x % oral medication x % prescribe G]
+ ditto for forecast of sales through ENT specialists
+ ditto for forecasts of sales through Primary Care
Physicians (PCPs)
62
Exercise using Monte-Carlo decomposition
• Forecast adoption in 2017 by people aged under 31 in
the USA of new Apple iPhone 7.
(see exercise sheet)
63
10. Group biases…..
64
Conforming to the group: Asch’s experiments
65
Solution
The Delphi method
Designed to avoid biases of face-toface discussion
(Also see: prediction markets)
66
Phases of Delphi
1. Panellists provide forecasts individually and privately
2. Results of polling are tallied and statistics fed back
3. Re-polling takes place
4. Process is repeated until consensus emerges.
5. Median estimate is then used as forecast
67
Delphi exercise
68
Answers to Delphi questions
Q1. The IBM PC was introduced in 1981
Q2. Facebook was launched in 2004
69
Some points on Delphi
• Can be used for large, geographically dispersed
groups
• No pressures from dominant individuals
• Anonymity allows change of mind without loss of face
• Information sharing is small –therefore little chance to
be persuaded by other people’s arguments
•
Choosing panel members –no evidence that experts
are better forecasters…
70
Trusting an expert’s intuition…
• In many fields as diverse as:
-Economic forecasting, Tetlock collected 82,361 political
-Stock market forecasting, and economic forecasts from
experts asking them to guess
-Conflict forecasting
probabilities for various events.
-Political forecasting
They performed worse than
chance
-Psychiatry
-Personnel selection
…no evidence that expert judgment is any
better than that of people with minimal
knowledge.
71
London Dustmen Trash
Finance Ministers in
Economic Forecasting!
72
Future research challenges
• Risk (or uncertainty) –estimating it & persuading
managers to understand and accept it.
• Need to encourage better division of labour between
judgment and statistics.
• Most research is lab-based. We need more research in
companies –including qualitative research.
73