Transcript Local-BMA Mean - Atmospheric Sciences

Local Bayesian Model Averaging
for the UW ProbCast
Eric P. Grimit, Jeffrey Baars, Clifford F. Mass
University of Washington, Atmospheric Sciences
Patrick Tewson
University of Washington, Applied Physics Laboratory
Research supported by:
Office of Naval Research
Multi-Disciplinary University Research Initiative (MURI)
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Motivation
“As high as 81! Hey, Eric!
Those intervals are too wide!
And 15% chance of precip?
Hmm…”
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Summary from Last Fall
Mean error climatology (MEC):
Ensemble-mean + its error variance over some history.
Good benchmark to evaluate competing calibration methods.
Generally beats the raw ensemble, even though it is not a statedependent forecast of uncertainty.
Local Bayesian model averaging (Local-BMA):
Model forecast performance varies locally:
BMA parameters should depend on grid point location.
Train BMA using elevation, land-use, and proximity constraints.
Can consistently beat MEC in tests with grid-based verification.
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Global-BMA Calibration and Sharpness
calibration
Probability integral transform (PIT) histograms
 an analog of verification rank histograms for
continuous forecasts
FIT
MEC
BMA
[00 UTC Cycle; October 2002 – March 2004; 361 cases]
sharpness
MEC
GlobalBMA
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Local-BMA Calibration and Sharpness
calibration
FIT
MEC
BMA
[00 UTC Cycle; October 2002 – March 2004; 361 cases]
sharpness
MEC
LocalBMA
Probability integral
transform (PIT)
histograms
 an analog of
verification rank
histograms for
continuous forecasts
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
BMA Forecast Skill Comparison
Global-BMA CRPS % improvement over MEC
Local-BMA CRPS % improvement over MEC
An Observation-Based Approach to Local-BMA
Development and testing: Winter-Spring 2006
Expect it to drive the MURI “killer application”  UW ProbCast.
Several “tuning” parameters available, which can hopefully be optimized.
Deploy it initially for MAXT2 and MINT2 forecasts.
Application to mixed discrete-continuous quantities (e.g., QPF) and 2-D
quantities (wind) will require further exploration.
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
An Observation-Based Approach to Local-BMA
Allow BMA parameters to vary by
grid point.
Use observations, remote if
necessary, as training data.
Follow the Baars et al. procedure
for bias correction (optimized from
the Mass-Wedam-Steed method)
to also select the training data for
Local-BMA.
For each grid point, search for n
(e.g. 8) nearby stations (e.g.
within 864-km) at similar elevation
(e.g. within 250-m) and having
similar land-use.
Land-use categories were
concatenated into 9 categories (down
from 24 in MM5).
Figure shows methodology for Mass-Wedam-Steed settings.
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Maximum 2-m Temperature – Case Study
----Station: KHMS (Hanford, WA)
Latitude, Longitude: 46.56, -119.60
South-north grid point: 52.289021
West-east grid point : 73.611740
obs:
model#:1, forecast:
model#:2, forecast:
model#:3, forecast:
model#:4, forecast:
model#:5, forecast:
model#:6, forecast:
model#:7, forecast:
model#:8, forecast:
ENS-MEAN:
-----
71.00 F
64.24 F
67.12 F
62.01 F
60.36 F
62.30 F
61.59 F
64.80 F
66.88 F
63.67 F
Spring MURI Meeting; Seattle, WA
Global-BMA Mean
----Station: KHMS (Hanford, WA)
Latitude, Longitude: 46.56, -119.60
South-north grid point: 52.289021
West-east grid point : 73.611740
obs:
71.00 F
model#:1, forecast: 67.56 F
model#:2, forecast: 69.57 F
model#:3, forecast: 65.71 F
model#:4, forecast: 64.62 F
model#:5, forecast: 67.23 F
model#:6, forecast: 66.28 F
model#:7, forecast: 68.80 F
model#:8, forecast: 69.23 F
Global-BMA-MEAN: 65.11 F
-----
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
Local-BMA Mean
----Station: KHMS (Hanford, WA)
Latitude, Longitude: 46.56, -119.60
South-north grid point: 52.289021
West-east grid point : 73.611740
obs:
model#:1, forecast:
model#:2, forecast:
model#:3, forecast:
model#:4, forecast:
model#:5, forecast:
model#:6, forecast:
model#:7, forecast:
model#:8, forecast:
Local-BMA-MEAN:
-----
71.00 F
67.56 F
69.57 F
65.71 F
64.62 F
67.23 F
66.28 F
68.80 F
69.23 F
68.84 F
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Bias-Corrected Ensemble Mean
----Station: KHMS (Hanford, WA)
Latitude, Longitude: 46.56, -119.60
South-north grid point: 52.289021
West-east grid point : 73.611740
obs:
model#:1, forecast:
model#:2, forecast:
model#:3, forecast:
model#:4, forecast:
model#:5, forecast:
model#:6, forecast:
model#:7, forecast:
model#:8, forecast:
BC-ENS-MEAN:
-----
71.00 F
67.56 F
69.57 F
65.71 F
64.62 F
67.23 F
66.28 F
68.80 F
69.23 F
67.38 F
Spring MURI Meeting; Seattle, WA
Global-BMA Sharpness
----Station: KHMS (Hanford, WA)
Latitude, Longitude: 46.56, -119.60
South-north grid point: 52.289021
West-east grid point : 73.611740
obs:
71.00 F
Global-BMA-95%: 72.55 F
Global-BMA-MEAN: 65.11 F
Global-BMA- 5%: 57.68 F
-----
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
Local-BMA Sharpness
----Station: KHMS (Hanford, WA)
Latitude, Longitude: 46.56, -119.60
South-north grid point: 52.289021
West-east grid point : 73.611740
obs:
Local-BMA-95%:
Local-BMA-MEAN:
Local-BMA- 5%:
-----
71.00 F
73.64 F
68.84 F
63.36 F
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
Local-MEC Sharpness
----Station: KHMS (Hanford, WA)
Latitude, Longitude: 46.56, -119.60
South-north grid point: 52.289021
West-east grid point : 73.611740
obs:
Local-MEC-95%:
Local-MEC-MEAN:
Local-MEC- 5%:
-----
71.00 F
72.15 F
67.38 F
62.61 F
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
Calibration (all stations)
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
Calibration (water only)
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
Sharpness (all stations)
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
Sharpness (water only)
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Minimum 2-m Temperature – Same Story
(water only)
(all stations)
(calibration)
(sharpness)
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Continuous Ranked Probability Scores
(water only)
(all stations)
(MAXT2)
(MINT2)
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Next Steps
Go operational with Local-BMA for MAXT2 and MINT2.
Code almost ready.
Some issues remaining with “blank” grid points.
Parameter optimization?
Work on precip next (PoP & PQPF).
Issues with small training samples and precip.
What if all zeroes?
Probably need to modify the search parameters.
Distance to crest?
Up-slope / down-slope? Depends on terrain gradient and wind!
Wind (2-D vector).
Established methods for wind speed and direction, separately.
Use gamma and Von Mises mixture distributions, respectively.
Need to build an EM-like algorithm or employ CRPS (energy score)
minimization for 2-D wind forecasts.
Work is being done on the CRPS (energy score) for 2-D variables. [statistics]
QUESTIONS and DISCUSSION
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
MEC Performance with Grid-Based Verification
Comparison of *UWME 48-h 2-m temperature forecasts:
Member-specific mean bias correction applied to both [14-day running mean]
FIT = Gaussian fit to the raw forecast ensemble
MEC = Gaussian fit to the ensemble-mean + the mean error climatology
[00 UTC Cycle; October 2002 – March 2004; 361 cases]
FIT
MEC
CRPS = continuous ranked probability score
[Probabilistic analog of the mean absolute error (MAE) for scoring deterministic forecasts]
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Local-BMA Forecast Performance
After several attempts to
implement BMA with local or
regional training data,
EXCELLENT results were
achieved:
when the training data is
selected from a
neighborhood* of grid
points with similar land-use
type and elevation
Example application to 48-h
2-m temperature forecasts
uses only 14 training days.
Dramatic improvements in
CRPS nearly everywhere.
*neighbors have same land use type
and elevation difference < 200 m within
a search radius of 3 grid points (60 km)
MEC
BMA
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
An Advanced Calibration Method
Bayesian Model Averaging (BMA) Summary
BMA has several advantages over MEC:
Member-specific
mean-bias
correction
Member-specific
BMA
parameters
weights
BMA variance
(not-member specific
here, but can be)
A time-varying uncertainty forecast.
A way to keep multi-modality, if it is warranted.
Maximizes information from short (2-4 week) training periods.
Allows for different relative skill between members through the BMA
weights (multi-model, multi-scheme physics).
[c.f. Raftery et al. 2005, Mon. Wea. Rev.]
Spring MURI Meeting; Seattle, WA
4 May 2006 10:50 AM
Extending BMA to Non-Gaussian Variables
For quantities such as wind speed and precipitation, distributions are
not only non-Gaussian, but not purely continuous – there are point
masses at zero.
For probabilistic quantitative precipitation forecasts (PQPF):
Model P(Y=0) with a logistic regression.
Model P(Y>0) with a finite Gamma mixture distribution.
Fit Gamma means as a linear regression of the cubed-root of observation on
forecast and an indicator function for no precipitation.
Fit Gamma variance parameters and BMA weights by the EM algorithm, with
some modifications.
[c.f. Sloughter et al. 200x, manuscript in preparation]