Transcript Are forecasting methods too complex?

Are forecasting methods too complex?
Kesten C. Green*, University of South Australia
J. Scott Armstrong*, University of Pennsylvania
*Ehrenberg-Bass Institute at University of South Australia
International Symposium on Forecasting
Riverside, California
24 June 2015
Slides available at ForPrin.com
G&A ISF 2015 – Complex-V13
On the value of complex forecasting methods
Analysts have long assumed that complex methods are
needed to deal with complex problems.
As long ago as 1985 (Armstrong, pp.225-232), a review of
the literature found that complexity tends to:
• reduce accuracy and understanding, and
• increase costs and mistakes
Despite the evidence, forecasters continue to use ever
more complex methods.
We reviewed this issue by organizing a JBR Special Issue on
Simple versus complex forecasting. We report on our paper
in that issue reviewing the evidence.
2
See
HANDOUT
3
Review of experimental research
Reviewed published research from all areas of
forecasting, including the Special Issue
papers…
1.defined simplicity in forecasting
2.identified studies with evidence on
comparative accuracy
3.assessed directional and effect size
evidence
4
Simple forecasting defined
Simplicity in forecasting requires that are all of the
following are understood by clients.
1. forecasting method,
2. representation of cumulative knowledge,
3. relationships in models,
4. relationships among models, forecasts, and
decisions
5
Findings
Found 32 papers with 97 comparisons:
a) None of the papers found that complexity
helped accuracy
b) Complexity increased error by 27% on average
across papers
“Simple versus complex forecasting: The evidence”
was published in JBR in 2015.
Due out in print in August.
6
Summary of evidence on accuracy of forecasts
from complex vs. simple methods
Method type
--------- Number of Comparisons --------Total
Simple
Total
compar- better or Effect
papers
isons
similar
size
Error
increase vs
simple (%)
Judgmental
4
4
4
4
28.2
Extrapolative
17
62
51
12
27.5
Causal
8
23
19
5
25.3
Combined
3
8
7
4
23.9
32
97
81
25
All method types
Weighted average*
26.7
*Weighted by total papers
7
Simplicity: A checklist
Score the following on a 0-to-10 scale
Use of prior knowledge in forecasting models
1.Do you know what prior knowledge about the situation was used?
2.Do you know how prior knowledge about the situation was used?
3.How simply is prior knowledge represented?
Nature of the relationships among the model elements
(Non-linear… Multiplicative… Additive… Single?)
Nature of the relationships among models, forecasts, and decisions
(Weak… Strong?)
Explaining the forecasting process
I am confident that I could explain… to the decision maker
1.the forecasting methods
2.how prior knowledge about the situation is represented in the
forecasting models
3.the nature of the relationships among the model elements
4.how the models, forecasts, and decisions are related to each other
8
Demonstration: Effect of simplicity vs.
complexity in climate forecasting
IPCC warming alarmists do not forecast,
they create “scenarios” via computer simulations
1.Scenarios are:
a. Stories about “what happened in the future”
b. Biased, so they do not provide valid forecasts
(Gregory & Duran, 2001).
2.The stories are based on expert judgments.
According to prior research, expert judgments
about what will happen in complex, uncertain
situations are useless:
a. Seer-sucker Theory
b. Tetlock’s 20-year experiment
9
IPCC process of creating climate scenarios via
computer simulations is enormously complex
1. Judgments are made on what variables to
include (e.g. CO2), and exclude (e.g. Sun);
2. Judgments are made on the values of many
parameters, and their (nonlinear) relationships
3. Around 50,000 grid squares are modeled
4. Grid square models interact
5. Models are adjusted to produce expected
outputs
6. Budget for computer simulations is enormous
10
Green, Armstrong, & Soon (2009) no-change
model is sophisticatedly simple
1. Based on an examination of diverse long
temperature histories…
2. No long-term trend
3. Reversals on all time scales
4. Correlated with solar cycles and variations in
activity
5. Weakly correlated with CO2…
but temperature changes precede CO2 changes,
and high CO2 levels have been associated with
ice ages.
11
Simple Forecasting Checklist ratings:
IPCC projections vs. no-change forecasts
Our Average Compliance Ratings (% of perfect score)
IPCC
No Change
19
96
Ratings can be done by novices (and experts) in forecasting.
The ratings take only minutes to do. (15 minute for us to rate
the IPCC method. Once we read their report).
Don’t take our word. Rate the method yourself.
12
Evidence on accuracy of IPCC projections vs.
no-change forecasts using Hadley data
Tests of forecasts over the 1851-1975 forecasting period yielded 58
forecasts for horizons of 91 to 100 years. The errors of these IPCC
forecasts were 12.6 times larger than those from the easily understood
no-change model (Green, Armstrong, & Soon 2009).
Chart from Forecasting Global Climate Change (2014)
13
Evidence on accuracy of IPCC projections vs. no-change
forecasts using Loehle & McCulloch (2008) data
From Forecasting Global Climate Change (2014)
14
Validation over different time-periods; data:
Similar results
From Forecasting Global Climate Change (2014)
15
The Complexity Penalty
In the late 1970, a research review found that
complex forecasting methods harmed accuracy.
In the late 1990s, the Forecasting Principles Project
developed 139 principles for forecasting. All were
simple to understand.
Our recent systematic review failed to find a single
study showing that a complex method was more
accurate than a simpler method.
16
Two Questions for you
What percentage of papers at this
conference propose complex methods?
Why does that happen?
17
Seduced by complexity
Some evidence suggests that the popularity of
complexity may be due to incentives:
(1)researchers are rewarded for publishing in highly
ranked journals, which favor complexity;
(2)forecasters can use complex methods to provide
forecasts that support decision-makers’ plans; and
(3)forecasters’ clients may be impressed and
reassured by incomprehensibility.
18
Conclusions
1. Those who prefer their forecasts to be
accurate should accept forecasts only from
simple evidence-based procedures.
2. Alarming IPCC temperature projections are
based on procedures that are too complex to be
trusted.
3. Rate the simplicity of forecasting procedures
used for problems you are interested in using the
questionnaire at…
simple-forecasting.com.
19