Transcript An Experiment to Evaluate the use of Quantitative

An Experiment to Evaluate the Use of
Quantitative Precipitation Forecasts from
Numerical Guidance by Operational
Forecasters
Joshua M. Boustead and Daniel Nietfeld
NOAA/NWS WFO Omaha/Valley, NE
Ray Wolf
NOAA/NWS WFO Davenport, IA
Presentation Overview
• Study purpose and methodology
• Data results
– Survey results
– Snowfall forecast
– Watch/warning statistics
– Gridded forecast results
• Forecasting implications, conclusions, and
future work
Study Motivation
• Strong interest in the role of the future
forecaster
– Can we still add value to the everyday
forecast?
– How can we better concentrate on highimpact weather?
– How can we better utilize increasingly hightech tools into the everyday forecast?
• How does this increasingly high-tech information
affect the forecaster?
Example
NSSL 4km WRF 00Z 8/14/07
Results
Study Purpose
• To evaluate if and how operational
forecasters are biased by numerically
generated quantitative precipitation
forecasts (QPF)
• Use these results to develop an updated
methodology for operational forecasters
on how to approach a daily forecast and
utilize the latest technology, including high
resolution model output
Study Methodology
• Utilizing the National Weather Service’s (NWS)
Warning Event Simulator (WES) operational
forecasters from two NWS offices made two
forecasts for two winter weather case
– The forecasters first completed the forecast,
including making a warning decision, without the use
of model QPF
– The forecasters then went through the same case
again with model QPF, again making a snowfall
forecast as well as a warning decision
• Once each scenario was completed, the
forecasters completed a survey about the
specific case
Study Methodology
• Two winter weather
cases were chosen
from the Central and
Northern Plains
– December 7-8, 2005
from the Pleasant Hill,
MO (EAX) forecast
area
– February 28 – March
1, 2004 from the
Bismarck, ND (BIS)
forecast area
Survey Results
Distribution of Forecaster Experience
Percent of Forecasters
60%
50%
40%
30%
20%
10%
0%
0 to 3 yrs
3 to 10 yrs
10 to 20 yrs
20 + yrs
Years of Operational Forecasting
• Forecaster Demographics:
– Forecasters were from the NWS offices in
Omaha/Valley, NE and Davenport, IA
– Operational forecasters involved were of a high
experience level
Survey Results
Forecaster Confidence Level without Using QPF
60%
50%
40%
30%
20%
10%
0%
Very Confident
Confident
Neutral
Unsure
Very Unsure
• Forecaster confidence without using model QPF:
– Majority of operational forecasters felt confident in making a
forecast without model QPF
– Potentially due to the high experience level of the forecasters
Survey Results
Change in Forecaster Confidence with QPF
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Sig increased
Increase
Unchanged
Decreased
Sig decreased
• Forecaster confidence after seeing QPF:
– Most forecasters indicated that seeing QPF either
increased their forecast confidence or it was
unchanged
Snowfall Forecast Results
Inches
Pre and Post QPF Mean Absolute Error for EAX and BIS
5
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
Pre-QPF
Post-QPF
Pre and Post QPF EAX
Pre and Post QPF BIS
• MAE was computed for each location and then
averaged for before and after the use of QPF
• MAE decreased 0.5 inches for both the EAX and
BIS case post QPF
Snowfall Forecast Results
• Majority of the
forecasts were
unchanged
post QPF
• Majority of the
forecasts that
did change
their forecast,
increased
accuracy
Distribution of Change in Forecast Accuracy with QPF among
Forecasters for EAX
34%
37%
Improved
Degraded
Unchanged
29%
Distribution of Change in Forecast Accuracy with QPF among
Forecasters for BIS
36%
38%
Improved
Unchanged
26%
Degraded
Warning Results
Warning Statistics
80%
70%
60%
50%
POD
40%
FAR
30%
20%
10%
0%
Pre-QPF POD and FAR
Post-QPF POD and FAR
• The probability of detection (POD) and false
alarm ratio (FAR) were computed by county for
each of the forecast areas
• Forecasters showed improvement in both the
POD and in FAR ratio once QPF was used
Warning Results
Distribution of Change in Probability of Detection Accuracy with QPF
for BIS
30%
Improved
Distribution of Change in False Alarm Ratio Accuracy with QPF for
BIS
29%
Degraded
Improved
Unchanged
Degraded
46%
Unchanged
57%
13%
25%
Distrubution of Change in Probability of Detection Accuracy with
QPF for EAX
Distrubution of Change in False Alarm Ratio Accuracy with QPF for
EAX
29%
33%
Improved
Degraded
Improved
45%
50%
Unchanged
Unchanged
21%
22%
Degraded
Gridded Forecast Results
EAX Case
• EAX Pre and Post
QPF MAE
– Forecasters had
the most
confidence in the
northern CWA
– Much better
agreement over
the southern CWA
post QPF
– Also a 2 to 3 inch
decrease in MAE
over the south
Gridded Forecast Results
EAX Case
• EAX Pre and
Post QPF
Standard
Deviation
– Forecast
differences
decreased over
the north and
south
– Slight increase in
differences over
the center
Gridded Model Forecasts
EAX Case
• Greatest
agreement of
snow band
across central
CWA
• Viewing QPF
increased the
forecast
confidence in
the southern
CWA
Actual Snowfall
EAX Case
Gridded Forecast Results
Bismarck Case
• BIS Pre and
Post QPF MAE
– Good agreement
and low error
over the
northwest
forecast area
– Mean errors of 5
to 6 inches over
the southern and
eastern forecast
area
Gridded Forecast Results
Bismarck Case
• Pre and Post QPF
Standard
Deviation
– Significant
increase in
forecaster
clustering across
the central forecast
area
– Greater than 4 inch
differences
continue over the
southern forecast
area
Gridded Model Forecasts
BIS Case
• Models agree
northwest CWA
to get least QPF
• Larger
uncertainty in the
south
• Forecasters
tended to pick a
model
– Led to continued
large MAE in the
southern CWA
Actual Snowfall
BIS Case
Discussion
• Only a slight improvement in snowfall forecasts was
noted once forecasters viewed QPF
– When snowfall forecasts were modified, a higher percentage
were improved than degraded
• Model QPF seemed best utilized to resolve snow-no
snow areas
– This led to improvements in both FAR and POD
• High MAE did not always mean high standard deviation,
which can indicate a systematic forecasting error
• Doesn’t clearly answer the question does model QPF
bias forecasters
– Some evidence in the BIS case where model agreement was
poor
• Possible forecast methodology
– Make entire forecast without QPF
– Utilize QPF for placement for defining snow-no snow areas
Future Work
• Conduct the study using two warm season
convective cases
• Investigate forecaster philosophy from the
surveys where standard deviation is low and
mean absolute error is higher
• Investigate what, if any, synoptic patterns
increase forecaster uncertainty and MAE
• Continue to increase the number of forecasters
in the study, and from different areas of the
CONUS