Transcript Document

The Impact of Probabilistic Information on Deterministic
& Threshold Forecasts
Susan Joslyn, Earl Hunt & Karla Schweitzer
University of Washington
EXPERIMENT
OUR QUESTION: Can weather forecasters incorporate uncertainty
information expressed in probabilities into a non-probabilistic
forecasting decision in a way that improves the forecast? Without
bias?
Background
•How do people understand
and use probabilistic
information?
PROCEDURE
–People, in general, do not treat
probability linearly
10 Participants: Atmospheric Science Students
Task: Forecast wind speed and direction for a 48 hour period.
(Gonzalez & Wu, 1999)
–Even experts have trouble
incorporating prior probabilities
whether to
issue high wind advisory (20 + knots). Forecasts were made for 4 days for four different
locations on each day.
(Eddy 1982 )
–However weather forecasters are good
at estimating probabilities of e.g
precipitation
.Decide
• Information Provided
Historical data for all products used in
previous cognitive task analysis (Joslyn et
(Baars & Mass, 2004)
al, submitted)
• Radar Imagery
• Probabilistic information is
• Satellite Imagery
• TAFs and current METARs
• Model output (AVN, MM5 & NGM)
increasingly important
product of ensemble
forecasts.
Design: Within subjects
• Each subject made 2 forecasts with the probability product,
2 forecasts without probability product
• Manipulation
On 1/2 of the forecasts, participants had the
MM5 ACME Ensemble Probability of winds
greater than 20 knots
• Yet few operational weather
• All subjects saw weather data from all of the same
dates
• Probability product rotated through the dates,
making sure that no subject saw the same date twice
• Instructions:
forecasters make use of
available probability
products
Subject
Free choice of products
Except: forecasts with the probability product
Required to read & record the range provided by the
probability product
(Joslyn, Jones & Tewson, 2005)
2/14
2/20
3/11
3/21
1
Without
With
probability
With
probability
Without
2
With
probability
Without
Without
With
probability
RESULTS
Wind speed forecast: Deterministic forecast
R2 Predicting observed from forecasted wind speeds
We conducted regression analyses predicting observed wind speeds from
forecasted wind speeds. See R2 for each condition in the table to the right.
Date
• For 3 dates people did better with the probability product
2/14
.151
.175
.124
• Note especially 3/21 which was the most difficult and for which the probability product made a significant
difference.
2/20
.172
.180
.180
3/11
.222
.248
.197
3/21
.046
.107*
.012*
Posting wind advisories: Threshold forecast
D prime
Beta
%Advisories
With
Probability
Product
.71
1.2
38%
• d with and without the probability product were
similar
• However, WITH the probability product:
– the criterion was higher (less willing to post advisory)
– Fewer advisories overall.
Without
Probability
Product
.72
1
45%
• Same forecasters/same weather
• Difference must be due to probability product
Percent of times forecasters posted an advisory
in each condition & the expected response
Percent of Advisories, Expected response and the observered
percentage of wind speed exceeding 20 knots
100%
With
Probability
Product
Percent Advisories
90%
80%
70%
Without
Probability
Product
60%
50%
40%
Expected
Response
30%
20%
Percent Advisories
% Observed Winds > 20 k
100%
90%
With Probability
Product
80%
70%
Without
Probability
Product
60%
50%
% times
observed
winds> 20kts
40%
30%
20%
Expected
Response
10%
10%
0%
0%
0-10
10-30
30-50
50-70
70-90
90-100
Probability of Winds Exceeding 20 knots
Subjects tended to
– Over forecast in low-mid probability
situations
– Underforecast in high probability ranges
– Attenuated with probability product
0-10
10-30
30-50
50-70
70-90
90-100
Probability of Winds Exceeding 20 knots
Probability product
– Good in very low probability situation
– Overforecast in the midranges
– In general, improved subjects performance
Total
With
Without
Probability
Probability
CONCLUSIONS
Probability product had a slight impact on the
deterministic wind speed forecast.
Wind Advisory Forecasts:
– Forecasters, in general, had a liberal bias in the low to
mid probability ranges: Biased to post wind advisories
• Makes sense in this task: being cautious, keeping people off the water if
there is any chance of danger
• However, boaters may come to disregard the advisory if it often proves to
be a false alarm (Roulston & Smith, 2004)
– Probability product attenuated this tendency without
causing them to lose sensitivity--they weren’t more likely
to miss high wind situations
– Forecasters, in general, had a conservative bias in the
highest probability ranges
• They did not post advisory as often as they should have
– Probability product attenuated this tendency
• more advisories in high range with probability product
References
Gonzalez, R., & Wu, G. (1999). On the form of the probability weighting function. Cognitive Psychology, 38, 129-166.
Eddy, D.M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic, &
A. Tversky (Eds), Judgment under uncertainty: Heuristics and biases pp. 249-267)
Baars, J. A. & Mass, C. F. (2004) Performance of National Weather Service Forecasts Compared to Operational, Consensus
and Weighted Model Output Statistics.
Joslyn, S. Jones, D.W. & Tewson, P.(2005) Designing Tools for Uncertainty Estimation in Naval Weather Forecasting. 7 th
International Conference on Naturalistic Decision Making
Roulston, M.S. & Smith, L. A. (2004) The boy who cried wolf revisited: The impact of false alarm intolerance on cost-loss
scenarios. Weather & Forecasting (19) 391-397.
This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745