Transcript Applications of the Golden Rule

Applications of the Golden Rule
J. Scott Armstrong*, University of Pennsylvania
Kesten C. Green*, University of South Australia
*Ehrenberg-Bass Institute at University of South Australia
International Symposium on Forecasting
Riverside, California
24 June 2015
Slides available at ForPrin.com
A&G ISF 2015 – Golden-R11
Golden Rule of Forecasting:
“Be Conservative”
or
“Forecast unto others as you would have them
forecast unto you.”
Be conservative by adhering to
cumulative knowledge about:
1.the situation, and
2.evidence-based forecasting methods
The “Golden Rule of Forecasting” was published
in June 2015.
2
Golden Rule of Forecasting (GR) Guidelines
Procedure: By logic, we developed 28 guidelines.
Validity testing by analyzing prior experimental
comparisons relevant to the guidelines, almost none of
which were done in awareness of the Golden Rule.
Directional effects were consistent with comparative
tests of accuracy.
70 papers tested effect sizes: On average, the use of a
single guideline reduced forecast error by 31%.
3
Why experimental findings?
We believe that experimentation is the basis for
scientific advances.
Not feasible to identify causality from analyses of
nonexperimental data in uncertain complex
situations. Illusions in regression analysis
Directions of effects from nonexperimental
studies often differ with those from experimental
studies. Armstrong & Patnaik (2009).
4
Golden Rule based on cumulative knowledge
about forecasting methods
Proved to be a large undertaking to develop the
hypothesis and to accurately summarize the
evidence.
1. Required three years to complete the paper.
2. Eighteen people provided peer review.
3. Eleven researchers contributed on various
aspects.
4. Four editors worked on the writing.
5
Support for the Golden Rule
1. We contacted all authors of key studies
for whom we found email addresses.
2. Of those, 84% responded.
3. All but one agreed with our summary of
their work (an issue not to be taken for
granted. See “Fawlty Towers” paper).
6
Example: Predicting Election outcomes
The Golden Rule Checklist was used to evaluate PollyVote.
1. Independent judgments were made as to whether the
situation involved:
a) complexity
Modest
b) uncertainty
Modest
c) likelihood of bias
Low
2. Independent judgments were made as to which
guidelines were relevant, and
3. Ratings were made as to whether the the Golden Rule
was used properly or not.
4. Comparisons were made of the accuracy of alternative
methods.
7
Forecast Accuracy of the PollyVote vs.
typical econometric model
We rated the PollyVote against the Golden Rule
checklist.
13 of the 28 guidelines were relevant to forecasting
elections.
The PollyVote adheres to all 13 guidelines.
8
80%
3.5
70%
3.0
Error reduction
60%
2.5
50%
2.0
40%
1.5
30%
1.0
20%
10%
0.5
0%
0.0
91
81
MAE PollyVote
71
61
51
41
31
Days to Election Day
MAE typical model
21
11
Mean absolute error (MAE)
Forecast Accuracy of the PollyVote vs.
typical econometric model
(across remaining days to election, 1992-2012)
1
Error reduction due to PollyVote
9
Climate Change Forecasts
We rated two methods used to forecast
global mean temperatures.
10
The Chart Behind Global Warming:1981-2013
(by Anthony Watts)
This is the surface temperature record, on the scale of human experience.
Warming alarmists do not forecast,
they create “scenarios” via computer simulations
1. Scenarios are:
a. Stories… about “what happened in the future”
b. Biased… so do not provide valid forecasts
(Gregory & Duran, 2001).
2. The stories are based on expert judgments.
3. According to prior research, expert judgments
about what will happen in complex, uncertain
situations are no more accurate than forecasts
from people with little expertise:
a. Seer-sucker Theory
b. Tetlock’s 20-year experiment
12
No-change model is conservative given
cumulative knowledge about the situation
Disagreement about the effects of
the many variables that affect
temperature.
See e.g. NIPCC’s Climate Change
Reconsidered II: Physical Science
&
Climate Change: The Facts 2014
13
Golden Rule applied to IPCC scenario
Golden Rule of Forecasting Checklist was used to evaluate IPCC
“business as usual” global warming scenario and no-change model
forecasts.
Consensus ratings by Armstrong and Green indicated that of the 20
relevant Golden Rule Checklist guidelines:
• the IPCC scenarios followed none
• the no-change model followed 95%
Don’t believe us? Rate them yourself and send us your ratings and
reasons!
Tests of forecast accuracy over the 1851-1975 forecasting period
yielded 58 forecasts for horizons of 91 to 100 years.
Average error (MAE) of no-change forecast for 50-year horizon was
0.24°C.
Errors from the IPCC scenario of .03°C warming-per-year were 12.6
14
Example of a conservative guideline
1.1.2 Decompose to best use prior information.
Look for data where there are different causal
causal factors (e.g., to forecast traffic deaths
forecast miles driven and deaths per mile driven,
then recombine as shown in this paper.)
Is the weather getting warmer in Las Vegas?
Each day’s high and low temperatures are
averaged. Than an average is taken across all days.
15
Does everyone agree that Las Vegas has
gotten warmer since 1937?
Any complainers?
17
Why is it warmer at night in Las Vegas, but
cooler in the day?
During the day, the pavements and buildings store
the heat.
The heat is emitted during the night.
Our thanks to Anthony Watts for this example.
20
Earlier evidence on accuracy of IPCC
projections vs. no-change forecasts
Tests of forecasts over the 1851-1975 forecasting period
yielded 58 forecasts for horizons of 91 to 100 years.
The errors of these IPCC forecasts were 12.6 times larger
than those from the simpler no-change model.
21
Conclusions for Climate Change
1. Alarming IPCC temperature projections are
based on procedures that are insufficiently
conservative to be trusted
2. Cumulative knowledge about the situation
was ignored. In the belief that “this time it
is different.”
22
Possible Applications
Golden Rule checklist allows commentators and decision
makers to assess whether forecasters were forecasting
as they would expect others to forecast unto them.
Especially useful for areas subject to bias, such as
a. corporate mergers,
b. mass transportation systems,
c. law suits (advertising: Lance Armstrong case)
d. public policies (e.g., gun control, minimum wages).
23
Conclusions on the use of the
Golden Rule Checklist
1. Rapid application of evidence developed over the past
century.
2. Inexpensive (raters don’t need high expertise in
forecasting)
3. Use of the guidelines produces more accurate ex ante
forecasts
4. Avoids the biggest forecast errors (e.g. Winter Storm)
5. Leads to actions that can improve accuracy.
6. Allows commentators and decision makers to assess
whether forecasters were forecasting as they would
expect others to forecast unto them.
24