Transcript UWME + - University of Washington

The University of Washington
Mesoscale Short-Range Ensemble
System
Eric P. Grimit, F. Anthony Eckel,
Richard Steed, Clifford F. Mass
University of Washington
The UW Mesoscale Ensemble System:
The Big Picture
• The UW Mesoscale Ensemble System was born out of
our experience with high-resolution prediction: MM5
run at 36, 12 and 4 km twice a day for many years.
• High-resolution forecasts can produce highly realistic
mesoscale structures, but there is considerable
uncertainty in initial conditions and physics.
• High resolution can amplify such uncertainty and thus
it is dangerous to provide users with high resolution
output for direct and literal use.
• Mesoscale ensembles are probably the best way to
provide the probabilistic information required by
users…information they are currently denied…but there
are significant roadblocks that need to be overcome.
Mesoscale Ensembles: In its Infancy
• At a national level, mesoscale ensembles are at a very
primitive stage:
• NCEP’s system at 48 km grid spacing is not really on
the mesoscale and uses a method (breeding) that is
probably not ideal for short range ensembles.
• Operational SREF have not had bias removal or proper
post-processing
• There have been a few short-term ensemble
experiments (e.g. SAMEX)--generally for convection
• The value of mesoscale SRRF have not been proven,
useful intuitive products are lacking, and there is little
experience in the user community.
• But most of us are convinced that this is the way to go.
The UW Mesoscale Ensemble System Essential Features
• A true mesoscale system: 36 - 12 km. Out to 48 h
• Testing the value of mesoscale ensembles over a
different environment: eastern Pacific, coastal zone,
area of terrain. Moist to desert locations.
• The diversity generation is based on using the varying
initial conditions and boundary conditions from a
broad range of operational synoptic models… all with
differing data assimilation, model structure and
numerics, and physics. Finesses BC problem.
• This approach is politically unacceptable to many
operational centers (who don’t like to be dependent on
others), but probably represents a high bar for others
to attempt to better.
• Additional diversity from varying model physics and
surface boundary conditions.
UW Mesoscale Ensemble System
• Single limited-area mesoscale modeling system
(MM5)
• 2-day (48-hr) forecasts at 0000 UTC in real-time
since January 2000. Now twice a day
• 36 and 12-km domains.
a)
b)
36-km
12-km
Configurations of the MM5 short-range ensemble grid domains. (a) Outer 151127 domain with 36-km
horizontal grid spacing. (b) Inner 103100 domain with 12-km horizontal grid spacing.
“Native” Models/Analyses Available
Resolution (~ @ 45 N )
Abbreviation/Model/Source
Type
avn, Global Forecast System (GFS),
Spectral T254 / L64
~55 km
National Centers for Environmental Prediction
cmcg, Global Environmental Multi-scale (GEM),
Computational
Distributed
1.0 / L14
~80 km
Objective
Analysis
SSI
3D Var
Finite
Diff
0.90.9/L28 1.25 / L11
~70 km
~100 km
3D Var
Finite
Diff.
32 km / L45
90 km / L37
SSI
3D Var
Spectral T239 / L29
~60 km
1.0 / L11
~80 km
3D Var
Spectral T106 / L21
~135 km
1.25 / L13
~100 km
OI
Spectral T239 / L30
Fleet Numerical Meteorological & Oceanographic Cntr.
~60 km
1.0 / L14
~80 km
OI
tcwb, Global Forecast System,
1.0 / L11
~80 km
OI
Canadian Meteorological Centre
eta, limited-area mesoscale model,
National Centers for Environmental Prediction
gasp, Global AnalysiS and Prediction model,
Australian Bureau of Meteorology
jma, Global Spectral Model (GSM),
Japan Meteorological Agency
ngps, Navy Operational Global Atmos. Pred. System,
Taiwan Central Weather Bureau
ukmo, Unified Model,
United Kingdom Meteorological Office
Spectral T79 / L18
~180 km
Finite
Diff.
5/65/9/L30 same / L12
~60 km
3D Var
UW Ensemble System
• Made use of the infrastructure already in place (grids,
data feeds, systems and application programmers), plus
took advantage the natural parallelization using large
clusters..which are ideal for ensemble work.
• The system was built and maintained by two
exceptional graduate students (Eric Grimit and Tony
Eckel), plus key staff members (Rick Steed, David
Ovens)
• Was designed as a real-time system from the beginning,
with verification as a core component
• Had two operational groups as prime subjects: the
Seattle NWS office and the Navy Whidbey Island
forecasting detachment.
• Had strong partners with UW Statistics and APL (under
MURI support)
Computer
Infrastructure:
Linux DualProcessor
Clusters
“Ensemblers”
Eric Grimit (l ) and
Tony Eckel (r) are
besides themselves
over the acquisition
of the new 20
processor Athlon
cluster
Key Goals: End-to-End Evaluation
• To build a viable, operational mesoscale SREF
• To verify it using both deterministic (ensemble mean)
and probabilistic approaches.
• To determine whether a system with members of varying
skill can be combined to produce reliable and usefully
sharp ensemble pdfs.
• To determine the best approaches for post-processing
(e.g., bias removal, calibration, optimal pdf generation)
• To determine whether the ensemble system can be used
to predict deterministic and probabilistic skill
• To create ensemble-based products that are valuable to
users.
• To learn how to optimally combine high resolution
deterministic forecasts and lower-res ensembles
UW Ensemble Web Page
48 h
Probabilistic
Forecast
Of 1 inch
In 12h:
UW’s Ensemble of Ensembles
Imported
Homegrown
Name
# of
EF
Members Type
Initial
Conditions
Forecast
Model(s)
Forecast
Cycle
Domain
ACME
17
SMMA
8 Ind. Analyses, “Standard”
1 Centroid,
MM5
8 Mirrors
00Z
36km, 12km
UWME
8
SMMA
8 Independent
Analyses
“Standard”
MM5
00Z
36km, 12km
UWME+
8
PMMA
8 Independent
Analyses
PME
8
MMMA 8 Independent
Analyses
8 MM5
00Z
variations
8
operational, 00Z, 12Z
large-scale
36km, 12km
36km
SMMA: Single Model Multi-Analysis
PMMA: Perturbed-model Multi-Analysis
MMMA: Multi-model Multi-Analysis
ACME: Analysis-Centroid Mirroring Ensemble
PME: Poor Man’s Ensemble
MM5: 5th Generation PSU/NCAR Mesoscale Modeling System
Multi-Analysis, Mixed Physics: UWME+
PBL
vertical
IC
ID#
ACME
Soil
MRF
diffusion
Cloud Microphysics
36km
Domain
Cumulus
12km
Domain
SST
cumulus Radiation Perturbation
shallow
Land Use Table
5-Layer
Y
Simple Ice
Kain-Fritsch
Kain-Fritsch
N
cloud
standard
standard
ACMEcore+
avn
plus01 MRF
LSM
Y
Simple Ice
Kain-Fritsch
Kain-Fritsch
Y
RRTM
SST_pert01
LANDUSE.TBL.plus1
cmcg
plus02 MRF
5-Layer
Y
Reisner II (grpl), Skip4
Grell
Grell
N
cloud
SST_pert02
LANDUSE.TBL.plus2
eta
plus03 Eta
5-Layer
N
Goddard
Betts-Miller
Grell
Y
RRTM
SST_pert03
LANDUSE.TBL.plus3
gasp
plus04 MRF
LSM
Y
Shultz
Betts-Miller
Kain-Fritsch
N
RRTM
SST_pert04
LANDUSE.TBL.plus4
jma
plus05 Eta
LSM
N
Reisner II (grpl), Skip4
Kain-Fritsch
Kain-Fritsch
Y
cloud
SST_pert05
LANDUSE.TBL.plus5
ngps
plus06 Blackadar
5-Layer
Y
Shultz
Grell
Grell
N
RRTM
SST_pert06
LANDUSE.TBL.plus6
tcwb
plus07 Blackadar
5-Layer
Y
Goddard
Betts-Miller
Grell
Y
cloud
SST_pert07
LANDUSE.TBL.plus7
ukmo
plus08 Eta
LSM
N
Reisner I (mx-phs)
Kain-Fritsch
Kain-Fritsch
N
cloud
SST_pert08
LANDUSE.TBL.plus8
Perturbations to:
1) Moisture Availability
2) Albedo
3) Roughness Length
see Eckel (2003) for further details
Research Dataset
November
December
January
3
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
1 1 1 1 1
1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4
February
March
1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2
5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
of 129, 48-h forecasts (Oct 31, 2002 – Mar 28, 2003) all initialized at 00z
- Missing forecast case days are shaded
 Total
 Analyzed
Parameters :
- 36 km Domain: Mean Sea Level Pressure (MSLP), 500mb Geopotential Height (Z500
- 12 km Domain: 10-m Wind Speed (WS10 ), 2-m Temperature (T2 )
 Verification:
- 36 km Domain: centroid analysis
(mean of 8 independent analyses,
available at 12-h increments)
12 km Domain
(101103)
36 km Domain (151127)
- 12 km Domain: RUC20 analysis
(NCEP 20 km mesoscale analysis,
available at 3-h increments)
Subjective Evaluation
• Often large differences in initializations and
forecasts
• Very useful forecasting tool
Thanksgiving Day 2001 Wind Forecast Bust
Eta-MM5 model 12-km runs on Tue &
Wed forecast severe wind storm for the
Puget Sound on Thu AM. Expected
widespread damage and power outage was
all over the news.
The storm came ashore weaker and further
south giving light and variable winds in the
Puget Sound.
eta-MM5 Initialized 00z, 21 Nov 01 (Tue. evening)
42h Forecast, valid 10AM Thursday
Verification, 10AM Thursday
42h forecast (valid Thu 10AM)
SLP and winds
Verification
- Reveals high uncertainty in storm track and intensity
- Indicates low probability of Puget Sound wind event
1: cent
2: eta
5: ngps
8: eta*
11: ngps*
3: ukmo
6: cmcg
9: ukmo*
12: cmcg*
4: tcwb
7: avn
10: tcwb*
13: avn*
The Importance of Grid-Based Bias Removal
•Particularly important for mesoscale SREF in which model
biases are often large
• Significantly improves SREF utility by correctly adjusting
the forecast PDF
Gridded Bias Removal
Bias-corrected
Forecast Period
November
December
Bia
For
January
3
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
1 1 1 1 1
1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4
Training
Period
February
Train
Bias-corrected Perio
Forecast Period
March
1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2
5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
Training
Period
For the current forecast cycle:
1) Calculate bias at every grid point and lead time using previous 2 weeks’ forecasts
1
bi, j ,t 
N



n 1 
N

f i, j ,t 

oi, j 
n
N number of forecast cases (14)
fi,j,t forecast at grid point (i, j ) and lead time (t )
oi,j observation (centroid-analysis or ruc20 verification)
2) Postprocess current forecast to correct for bias:
f i*, j ,t 
f i , j ,t
bi , j ,t
* bias-corrected forecast at grid point (i, j ) and lead time (t)
fi,j,t
Uncorrected ACMEcore+ T2
4.0
3.5
2.0
1.5
1.0
0.5
48 h
2.5
12 h
24 h
36 h
Average RMSE (C)
and
Bias
Average
(shaded)
Average
Average
RMSE
RMSE
and
and Bias
Bias
(mb)
(C)
3.0
0.0
-0.5
-1.0
-1.5
-2.0
-2.5
plus01
plus02
plus03
plus04
plus05
plus06
plus07
plus08
mean
Bias-Corrected ACMEcore+ T2
4.0
3.5
2.0
1.5
1.0
0.5
48 h
2.5
12 h
24 h
36 h
Average RMSE (C)
and
Bias
Average
(shaded)
Average
RMSE
RMSE
and
andBias
Bias
Bias
(mb)
(C)
Average
RMSE
and
(mb)
3.0
0.0
-0.5
-1.0
-1.5
-2.0
-2.5
plus01
plus01
plus02
plus02
plus03
plus03
plus04
plus04
plus05
plus05
plus06
plus06
plus07
plus07
plus08
plus08
mean
mean
Physics and Surface Diversity
Substantially Enhance a Mesoscale
SREF, Particularly for Surface
Quantities
Probabi
Probabil
Comparison of 36-h VRHs
0.2
0.2
(a) Z500
VOP
0.4
0.4
0.1
0.1
Probability
Probability
0.3
0.3
0.0
0.0
11
22
3
4
4
5
5
6
6
77
7
88
36h
ACMEcore
*UWME
36h
ACMEcore
36h
ACMEcore+
36h
ACMEcore+
36h
ACMEcore
*UWME+
36h
ACMEcore
36h
*ACMEcore
5.0 %
4.2 %
36h
ACMEcore
*UWME
36h
*ACMEcore
36h
ACMEcore
36h
ACMEcore+
36h
*ACMEcore+
36h
ACMEcore+
36h
ACMEcore
*UWME+
36h
*ACMEcore
36h
ACMEcore
9.0 %
6.7 %
*UWME
36h
*ACMEcore
25.6 %
13.3 %
99
Verification Rank
Verification
Rank
0.2
0.2
(b) MSLP
0.1
0.1
Synoptic
Variable
0.0
0.0
0.4
11
22
33
44
55
66
777
88
99
Verification Rank
Verification
Rank
Surface/Mesoscale
Variable
0.3
Probability
( Errors Depend on
Analysis Uncertainty )
(c) WS10
0.2
0.1
( Errors Depend on
Model Uncertainty )
*UWME+
36h
*ACMEcore+
36h
*ACMEcore
0.0
0.4
1
2
3
4
5
6
7
8
9
Verification Rank
(d) T2
Probability
0.3
0.2
0.1
*UWME
36h
*ACMEcore
*UWME+
36h
*ACMEcore+
36h
*ACMEcore
0.0
1
2
3
4
5
6
Verification Rank
7
8
9
43.7 %
21.0 %
0.5
0.6
0.5
0.4
0.4
0.3
BSS
BSS
Skill for
P(T2 < 0°C)
0.3
0.1
0.0
0.0
-0.1
00
03
06
09
12
15
18
21
24
27
30
33
36
39
42
45
-0.1
48
Reliability
0.00
0.02
00
03
06
09
12
*Indicates
bias
removal
0.04
0.06
0.14
0.12
Resolution
core
*UWME
*ACMEcore
UWMEcore
ACMEcore
core+
*ACMEcore+
*UWME
ACMEcore+
UWMEcore+
Uncertainty
Uncertainty
0.2
0.1
Importance
Of Bias
Removal
And
Physics
Diversity
0.2
0.10
0.08
0.06
0.04
00
03
06
09
12
15
18
21
24
27
30
Lead Time (h)
33
36
39
42
45
48
15
Smaller Scales Generate Ensemble Dispersion
3h Cumulative Precipitation
4.5
UWMEcore WS10
2
2
(mMean
/s )
EF Ensemble
Spread orVariance
MSE of EF
4.0
36km Grid Spacing
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
0
12
24
36
Lead Time (h)
12km Grid Spacing
48
0.96
Using members with varying skill is OK,
but there is a limit to how bad a member
can be and still add value to the ensemble.
0.94
Removing Very
Unskillful Members
Can Help
0.92
BSS
0.90
0.88
*PME
0.86
*PME (x-tcwb)
*PME (x-ukmo)
0.84
*PME (x-ngps)
0.82
0
12
24
Lead Time (h)
36
48
Relating Forecast Skill and Model Spread
Mean Absolute Error of Wind Direction is Far Less When
Spread is Low
UW MM5 SREF 10-m Wind Direction
[c.f. Grimit and Mass 2002]
Spatial Distribution of Local Spread-Error Correlation
(no bias correction)
UWME
Domain-Averaged
STD-AEM correlation
~ 0.62
Maximum Local
STD-AEM correlation
~ 0.54
12-km
T2
A Simple Stochastic Model of Spread-Skill
An extension of the Houtekamer (1993) model of spread-skill
PURPOSES:
1)
To establish practical limits of forecast error predictability that could
be expected given perfect ensemble forecasts of finite size.
2)
To address the user-dependent nature of forecast error estimation
by employing a variety of predictors and error metrics.
3)
To extend spread-skill analysis to a probabilistic framework of
forecast error prediction.
A Simple Stochastic Model of Spread-Skill
1.
Draw today’s “forecast uncertainty” from a log-normal
distribution (Houtekamer 1993 model).
ln( s ) ~ N( ln(sf) , b 2 )
2.
Create synthetic ensemble forecasts by drawing M
values from the “true” distribution.
Fi ~ N( Z , s 2 ) ; i = 1,2,…,M
3.
Draw the verifying observation from the same “true”
distribution (statistical consistency).
V ~ N( Z , s 2 )
• Statistical ensemble forecasts at a single, arbitrary location
• 104 realizations (cases)
• Assumed:
– Gaussian statistics
– statistically consistent (perfectly reliable) ensemble forecasts
• Varied:
– temporal spread variability (b)
– finite ensemble size (M)
– spread and skill metrics (continuous and categorical)
Idealized Spread-Error Correlations
STD-AEM correlation
spread
STD = Standard Deviation
error
AEM = Absolute Error of the ensemble Mean
N = 10000
b = 0.5
STD-error correlation
error
AES = Absolute Error of a Single ensemble member
AAE = ensemble-Average Absolute Error
RASE = square Root of ensemble-Average Squared Error
CRPS = Continuous Ranked Probability Score
The Conditional Error Climatology (CEC) Method
• Use historical errors,
conditioned by spread
category, as
probabilistic forecast
error predictions
1 2 3
4
5
Idealized, statistical
ensemble forecasts.
N = 2000
M = 50; b = 0.5
Probabilistic Forecast Error Predictability
• Or might use the
ensemble variance
directly to get a
probabilistic error
forecast
ENS-PDF
– Most skillful approach if
PDF is well-forecast
Idealized, statistical
ensemble forecasts.
N = 10000
M = 50; b = 0.5
Effect of Post-Processing
UWME+
(14-day grid point bias correction)
• Bias correction reduces spread-error correlations and
effectiveness of the VAR-CEC approach
• ENS-PDF closes the gap in performance, but is still
below the baseline
12-km
T2
Future UW Ensemble Work
• Evaluation of value of temporal ensembles for adding
to diversity of on-time ensembles and for prediction of
ensemble skill
• Perfect grid-based bias removal of component
members
• Replace physics ensemble with one based on key
uncertainties in parameterizations. (Stochastic physics
is not ready for prime time yet)
• Creation of a new generation of products to help break
the ice with forecasters.
• Work with statistics on emos and BMA front-end to
our ensemble system
Future Work
• Add model diversity using available WRF
dynamical cores.
• Creation of web interface that combines
ensemble and high-resolution products.
• Evaluation of value of nudging outer
domain towards parent forecasts to improve
diversity.
The END
Ensemble Post-Processing
• In several of the talks today you will be viewing
some results of gridded bias correction of the
individual ensemble forecasts.
• More terminology: * indicates bias removal
• All mesoscale modeling system have significant
systematic biases.
• The biases vary by ensemble member, season,
time of day, etc.
• Removal of such biases has a very beneficial
effect on the value of ensembles.
Results
3.0
GFS-MM5 Mean Absolute Error
Data Info
MSLP MAE (mb)
2.5
Average of 65 forecasts
(25 Nov 02 – 01 Feb 03)
2.0
1.5
raw
bias corrected
1.0
36km domain from Rockies
to central Pacific
2-week bias training for
each forecast
0.5
0.0
12
18
24
30
36
Lead Time (hours)
42
48
Verification: centroid
analysis
Idealized Probabilistic Error Forecast Skill
• May use the ensemble
variance directly to get a
probabilistic error
forecast
ENS-PDF
– Most skillful approach if
PDF is well-forecast
• Predictability highest for
extreme spread cases
– Reinforces earlier results
Continuous case
Idealized, statistical
ensemble forecasts.
N = 10000
M = 50; b = 0.5
Idealized Probabilistic Error
Forecast Skill
(categorical case)
Idealized, statistical
ensemble forecasts.
N = 10000
M = 50; b = 0.5
The Future is Probabilistic
• We never will know exactly what the forecast will
be due to initialization uncertainty, inadequate
model physics, and other reasons.
• Thus, probabilistic forecasting is the only rational
way to forecast.
• We will also gain some ability to forecast
(probabilistically) forecast skill
• We have to retrain ourselves AND our users.
• The UW system is an attempt to develop and
evaluate this approach using ensembles. We
acutely need feedback from forecasters.
Probabilistic Products
• Currently Using Uniform Ranks (UR)
method.
• Democratic voting (DV) method was
good as MOS. UR is even better.
Calibration would provide further
improvements.
Comparison of Brier skill scores for NGM MOS and 12-km ACMEcore
forecasts of 12h probability of precipitation accumulations greater
than 0.01 in (CAT1). The skill scores are relative to the sample
climatology during the period from 1 Nov 2002 – 20 Jan 2003.