Transcript Annex 2

Communication of Uncertainty in Weather Forecasts
Pricilla Marimo, Todd R. Kaplan, Ken Mylne and Martin Sharpe
3. Results
1. Introduction
In our study, we test a bar graph format and a table format transition using experimental economics lab techniques. The Met Office
Public Weather Service (PWS) is constantly developing new products for disseminating weather information to users. After public
consultation, they have a new format for presenting probabilistic forecast information for use on the Met office website. Different
presentation formats/designs can be used to illustrate the same data in various fields. However, the way that information is
presented and consequently how we interpret or process it has the potential to influence decision making.
Average
earnings
£6.33
% correct
(Format C)
75.6
74.3
72.5
78.2
59.7
81.7
75.0
78.0
75.5
70.7
75.1
£7.30
35
30
25
20
15
10
5
order 1
order 2
swing
£7.45
A “correct” response was one in which the participant chose the most probable outcome.
On average, participants who were provided with format B or C outperformed those with
Format A.
General decrease in time as experiment
progressed
hard
order 3
format C
format B
format A
format C
format B
format A
format C
format B
0
format A
% correct
(Format B)
80.0
76.1
77.5
75.0
80.2
57.9
86.0
79.2
79.6
76.7
75.3
77.6
format C
% correct
(Format A)
64.9
66.4
67.6
64.9
67.6
18.0
86.9
66.6
67.4
62.0
70.0
66.2
format B
Business
Humanities
Sciences
Female
Male
Swing questions
Non swing questions
Order 1
Order 2
Order 3
Order 4
Overall
b)Time Analysis
format A
Studies that have been done to assess decision making when provided with probabilistic weather information, have concluded that
on average, participants who were given uncertainty information made significantly better decisions than those without (Roulston et
al. 2006; Roulston and Kaplan 2009). Still questions arise on whether or not the presentation format makes a difference in
interpretation and understanding. This study follows the same approach that was used by (Roulston and Kaplan 2009). Their study
tested the ability of subjects to understand the information in a fan chart format for expressing uncertainty in 5-day temperature
forecasts.
a) Summary statistics
median response time (seconds)
Providing probabilistic weather information to users has the potential to improve decision making. Weather is uncertain due to the
chaotic and complex nature of the atmosphere, hence the need to communicate this uncertainty information. According to (NRC
2006), a forecast is therefore incomplete if uncertainty information is not included. The Met Office, through the use of ensemble
forecasting and other techniques is capable of providing probabilistic estimates of weather forecasts. Including such information,
although possibly valuable to users, can increase the complexity and amount of information being conveyed.
order 4
easy
participants took more time on the swing
questions compared to the other question
types despite the format shown or order
except for Group B participants with order
1, where the easy took more time.
For the easy and hard questions,
participants took almost the same median
time for Order 2, 3 and 4.
c) Probit Model
2. Method
A total of 289 undergraduate students from various disciplines at the University of Exeter were recruited to participate in the experimental
sessions. The sessions were computer based and took place in the Finance and Economics Experimental Laboratory (FEELE) at the
University of Exeter. Participants were presented with a set of 20 “lotteries” based on the maximum temperature up to five days ahead and
asked to chose the ‘most likely’ outcome. If true statement was chosen, participants were rewarded with £0.50. Participants were divided
into three treatment groups: A, B and C. The 5-day temperature forecast information was presented as follows: Group A: Table with a point
forecast , Group B: Table with point forecast and uncertainty information , Group C: Bar graph with point forecast and uncertainty
information. The same graphs were shown for all the participants in a particular group but in randomised order Four question orders were
used :
1st order: 1, 2,..., 20
2nd order: 20, 19,..., 1
3rd order: 11, 12,..., 20, 1, 2,..., 10
4th order: 10, 9,..., 1, 20, 19,..., 11
This was done to test speed of learning differences between the different presentation formats. An example of a lottery is shown below.
After each lottery students were informed of the actual temperatures and whether either of the criteria had been satisfied. At the end of the
experiment students were paid their lottery winnings in addition to a £3.00 payment for participating.
Format B
Format A
Format C
Question 4: Would you prefer to receive £0.50 if...
A: The maximum temperature on Saturday is above 5 ºC OR if
B: The maximum temperature on Wednesday is above 8 ºC
Of the 20 lotteries, eight were easy questions and the remaining were equally divided between hard and swing. Question 4 above was
a “swing” question and these were:
Questions in which a hypothetical participant with just a point forecast assuming mode=median and same uncertainty for all
forecasts would make a different decision to someone with uncertainty information (i.e. Format B or C)
Discipline
Humanities
Business/Econ
Science/ Eng
Gender
Format A
Format B
Format C
Male
Female
Male
Female
Male
Female
9
12
16
19
20
11
31
14
12
15
7
18
27
14
29
35
-
A summary of the numbers of participants in the
different treatment groups is shown in the table on the
left
Pricilla Marimo
School of Business and Economics
University of Exeter Streatham Court, Rennes Drive, Exeter, EX4 4PU, UK
Email: [email protected]
Ken Mylne
Met Office FitzRoy Road Exeter, EX1 3PB, UK
Email:[email protected]
Statistical analysis to estimate the determinants of choosing the most probable outcome was done using
a probit regression model. The model
£6.33
predicts the probability that a participant will answer a question correctly as a function of a collection of predictor variables.


Pr(correct|predictors)     coefficient i  predictori 
 i

Where Φ is the cumulative distribution function of the standard normal distribution. The list of predictors with the corresponding coefficients found by
fitting the model are listed in the table below.
Predictor
Round number
Swing question
Swing question & Format A
Swing question & Format C
Male
English is first language
Checks internet for weather forecast
Length
Area
Checks weather at least every 2-3 days
Sample question mistake
Die question mistake
Format B
Format C
Order 1
Order 1 & Format C
Order 1 & Format B
Order 2
Order 2 & Format B
Order 2 & Format C
Order 4
Order 4 & Format B
Order 4 & Format C
Early day correct
Early date correct & Format C
Early date correct & Format A
Early date correct & Order 1
Early date correct & Order3
Early date correct & Order4
Above & certain
Above & uncertainty
Test question dummy
Business
Humanities
New format B
Hard question
Constant
*** sig at 1% level
** sig at 5% level
Coeff
0.0045
-0.9611***
-1.1290***
0.1506
0.0720*
0.0524
0.0042
-0.4162***
-0.5460***
-0.0528
-0.3607***
-0.1971***
0.2498*
0.3633***
0.2229**
-0.2841**
-0.0561
0.1822
-0.0575
-0.1773
0.3017**
-0.3968**
-0.5807***
-0.3565***
-0.2973***
-0.1262
0.1048
-0.0403
0.2734**
-0.1595**
-0.3156***
0.0321
0.0079
-0.0121
-0.0450
0.0862
1.1657
P-Value
0.195
0.000
0.000
0.144
0.084
0.316
0.963
0.000
0.000
0.211
0.000
0.000
0.067
0.007
0.043
0.036
0.689
0.116
0.693
0.207
0.026
0.013
0.000
0.000
0.002
0.238
0.333
0.718
0.015
0.039
0.000
0.643
0.909
0.852
0.661
0.149
0.000
% change in probability
0.14
-32.28
-41.40
4.40
2.20
1.61
0.13
-14.25
-19.16
-1.62
-12.21
-6.20
7.39
10.67
6.57
-9.33
-1.74
5.39
-1.79
-5.68
8.63
-13.38
-20.40
-11.14
-9.73
-3.99
3.10
-1.25
7.64
-5.06
-9.87
0.97
0.24
-0.37
-1.39
2.60
0.00
Marginal effects were also computed and are shown as the percentage change in
probability. For instance, male participants were 2% more likely to get a question
correct.
Participants were 32% less likely to get “swing” questions correct; 74% less likely if
they had no uncertainty information
Getting probability or test question wrong would decrease chances
Uncertainty information increased chances of choosing most probable
Biased towards picking the later option (early day correct) ; if format C increased
further
Area and Length variables
Area: 1= questions where the greatest area between the high/low range & the asked
temperature does not get the correct answer for Group C participants for questions 1,12
and 13
Length: 1= questions where the greatest length between the high/low range & the asked
temperature does not get the correct answer for Group B participants for questions 1, 12
and 13.
e.g. Round 12 of 20 ( shown below)
Statement A - The maximum temperature on Saturday is above 13 deg. C
Statement B - The maximum temperature on Wednesday is above 12 deg. C
It is possible that some did not understand or could not use the uncertainty information correctly- chose option with biggest
area
More likely to choose statement B
because of the big area above 12 deg. C
. Most likely outcome however is A.
Percent correct:
Format A – 80.5
Format B – 66.0
Format C – 49.5
* sig at 10% level
4. Conclusion
On average participants who were given uncertainty information made significantly better decisions than those without. Both table and the graph
with uncertainty information were significant determinants of choosing the most probable outcome, though the graph is a stronger predictor
compared to the table. There was a learning effect as experiment progressed. Graph with uncertainty information took on average less response
time compared to those who were shown a table with uncertainty information. Analysis with the general public (underway at the Met Office)
5. References
NRC. 2006. Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts Edited by Committee on Estimating and Communicating Uncertainty in Weather and Climate Forecasts.
Washington D.C: National Academies Press.
Roulston, M.S, G.E. Bolton, A.N. Kleit, and A.L. Sears-Collins. 2006. A Laboratory Study of the Benefits of Including Uncertainty Information in Weather Forecasts. Weather and Forecasting 21: 116-122.
Roulston, Mark S., and Todd R. Kaplan. 2009. A laboratory-based study of understanding of uncertainty in 5-day site-specific temperature forecasts. Meteorological Applications 16 (2): 237-244.