Transcript ppt - Cosmo
Deutscher Wetterdienst
COSMO-DE-EPS
Susanne Theis
Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold
Presentation Overview
setup of COSMO-DE-EPS
first results of pre-operational phase
verification
forecasters‘ feedback
COSMO-DE-EPS plans
COSMO GM – September 2011
Deutscher Wetterdienst
Setup of COSMO-DE-EPS
COSMO-DE-EPS status
pre-operational phase has started:
Dec 9th, 2010
pre-operational setup:
20 members
grid size: 2.8 km
convection-permitting
lead time: 0-21 hours,
8 starts per day (00, 03, 06,... UTC)
variations in
physics, initial conditions, lateral boundaries
COSMO GM – September 2011
model domain
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
COSMO GM – September 2011
Boundaries
Model Physics
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
Boundaries
“multi-model”
driven by different
global models
COSMO GM – September 2011
Model Physics
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
“multi-model”
Boundaries
“multi-model”
COSMO-DE initial
driven by different
conditions modified by global models
different global models
COSMO GM – September 2011
Model Physics
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
“multi-model”
Boundaries
“multi-model”
COSMO-DE initial
driven by different
conditions modified by global models
different global models
COSMO GM – September 2011
Model Physics
“multi-configuration”
different configurations
of COSMO-DE model
Generation of Ensemble Members
plus variations of
• initial conditions
• model physics
Ensemble Chain
COSMO-DE-EPS
2.8km
COSMO 7km
GME, IFS, GFS, GSM
COSMO GM – September 2011
BC-EPS
Generation of Ensemble Members
plus variations of
• initial conditions
• model physics
Ensemble Chain
COSMO-DE-EPS
2.8km
COSMO 7km
GME, IFS, GFS, GSM
COSMO GM – September 2011
BC-EPS
BC-EPS is running as a time-critical
application at ECMWF
Generation of Ensemble Members
20 Members
1
IFS
GME
GFS
BC-EPS
COSMO GM – September 2011
GSM
2
3
4
5
Generation of Ensemble Members
Perturbation Methods
Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011:
Uncertainties in COSMO-DE precipitation forecasts introduced by model
perturbations and variation of lateral boundaries. Atmospheric
Research 100, 168-177. (contains status of 2009)
Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the
COSMO-DE-EPS, COSMO Newsletter 11, 115–123.
Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold,
2011: Accounting for initial condition uncertainties in COSMO-DE-EPS.
Submitted to Journal of Geophysical Research.
COSMO GM – September 2011
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
Verification Method
SYNOP
RADAR
Ensemble Members
Probabilities of Precipitation
COSMO GM – September 2011
PREC 1h accumulation, threshold: 0.1 mm
DETERMINISTIC SCORES
for Individual Members
0.5
Equitable Threat Score
0.4
JUNE 2011
Do the ensemble members
have different long-term statistics?
(multi-model / multi-configuration)
Are there many cases with
the same „best member“
or „wettest member“?
0.3
0.2
IFS
GME
GFS
GSM
0.1
0.0
0
5
} 20 members
10
15
Forecast Time [h]
COSMO GM – September 2011
20
- look at Equitable Threat Score
- look at Frequency Bias Index
(results similar, not shown)
PREC 1h accumulation, threshold: 0.1 mm
DETERMINISTIC SCORES
for Individual Members
0.5
Equitable Threat Score
0.4
JUNE 2011
Do the ensemble members
have different long-term statistics?
(multi-model / multi-configuration)
Are there many cases with
the same „best member“
or „wettest member“?
0.3
0.2
IFS
GME
GFS
GSM
0.1
0.0
0
5
} 20 members
10
15
20
- look at Equitable Threat Score
- look at Frequency Bias Index
(results similar, not shown)
Forecast Time [h]
Only small differences in long-term statistics
Members may be treated as equally probable
COSMO GM – September 2011
PREC 1h accumulation
RANK HISTOGRAM
observation...
- …treated as „Ensemble Member“
Frequency
JUNE 2011
- …ranked according to prec amount
at each grid point and forecast hour
How frequent is each rank?
0.05
If ensemble underdispersive
U-shaped rank histogram
0.00
1
6
11
16
Rank
COSMO GM – September 2011
21
Observation
PREC 1h accumulation
observation...
RANK HISTOGRAM
- …treated as „Ensemble Member“
- …ranked according to prec amount
at each grid point and forecast hour
Frequency
JANUARY 2011
How frequent is each rank?
0.05
If ensemble underdispersive
U-shaped rank histogram
0.00
1
6
11
16
Rank
COSMO GM – September 2011
21
Observation
PREC 1h accumulation
observation...
RANK HISTOGRAM
- …treated as „Ensemble Member“
- …ranked according to prec amount
at each grid point and forecast hour
Frequency
JANUARY 2011
How frequent is each rank?
0.05
If ensemble underdispersive
U-shaped rank histogram
0.00
1
6
11
16
Rank
21
Observation
a) Underdispersiveness relatively small
b) Four groups Many cases with large influence by global models
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JANUARY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
Forecast Time [h]
COSMO GM – September 2011
20
PREC 1h accumulation
BRIER SKILL SCORE
JANUARY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive! Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JUNE 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive! Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
MAY - JULY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
- for different precipitation thresholds (x-axis)
(probabilites of exceeding a certain threshold)
0.1 1
2
5
10 20
- for all foreast lead times
Threshold [mm/h]
For larger precipitation amounts (summer):
even more additional value
COSMO GM – September 2011
PREC 1h accumulation
RELIABILITY DIAGRAM
JUNE 2011
log (# fcst)
> 0.1 mm
> 1 mm
> 2 mm
COSMO GM – September 2011
Are the probabilities already
well calibrated?
(without extra calibration)
If we isolate all cases with a
forecast probability of -say- 75-85%
…
did the event occur in 80%
of these cases?
diagonal line: optimal
- for different prec thresholds (colors)
(probs of exceeding a threshold)
PREC 1h accumulation
RELIABILITY DIAGRAM
JUNE 2011
log (# fcst)
> 0.1 mm
> 1 mm
> 2 mm
Are the probabilities already
well calibrated?
(without extra calibration)
If we isolate all cases with a
forecast probability of -say- 75-85%
…
did the event occur in 80%
of these cases?
diagonal line: optimal
- for different prec thresholds (colors)
(probs of exceeding a threshold)
Reliability diagram shows some bias and underdispersiveness
Lines are not flat additional calibration has good potential
COSMO GM – September 2011
Summary of Verification (Precipitation)
Ensemble provides additional value to COSMO-DE
(for all accumulations, lead times, precipitation thresholds,…)
Ensemble underdispersiveness is relatively small
Ensemble members may be treated as equally probable
Additional calibration has good potential
COSMO GM – September 2011
Summary of Verification (Precipitation)
Ensemble provides additional value to COSMO-DE
(for all accumulations, lead times, precipitation thresholds,…)
Ensemble underdispersiveness is relatively small
Ensemble members may be treated as equally probable
Additional calibration has good potential
Pre-operational COSMO-DE ensemble prediction system
already meets fundamental quality requirements for precipitation
COSMO GM – September 2011
Other Variables
T_2M and VMAX have been verified
ensemble spread is far too small
nevertheless, ensemble provides additional value to COSMO-DE
COSMO GM – September 2011
Other Variables
T_2M and VMAX have been verified
ensemble spread is far too small
nevertheless, ensemble provides additional value to COSMO-DE
COSMO-DE ensemble prediction system
has been developed with focus on precipitation
COSMO GM – September 2011
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
Forecasters‘ Feedback
available products:
see figure
precipitation, snow, wind gusts, T_2m
probability thresholds: warning criteria
all products on grid-scale (2.8km)
in addition: precipitation probabilities
for larger areas (10x10 grid boxes)
„probability that the precipitation event
will occur anywhere within the region“
COSMO GM – September 2011
probabilities, quantiles,
ensemble mean,
spread, min, max, …
Forecasters‘ Feedback
evaluate „full package“
- including the visualization tool
- consistency of products
select relevant cases
consider forecasters‘ interpretation
- perception as intended?
- is there any value in the forecast,
additional to forecasters‘ knowledge?
COSMO GM – September 2011
Forecasters‘ Feedback
what they prefer to use:
90%-quantile of precipitation
precipitation probabilities for an area (10x10 grid points)
COSMO GM – September 2011
Forecasters‘ Feedback
what they prefer to use:
90%-quantile of precipitation
precipitation probabilities for an area (10x10 grid points)
what they appreciate:
early signals for heavy precipitation
indication that deterministic run may be wrong
COSMO GM – September 2011
Forecasters‘ Feedback
what they prefer to use:
90%-quantile of precipitation
precipitation probabilities for an area (10x10 grid points)
what they appreciate:
early signals for heavy precipitation
indication that deterministic run may be wrong
what they criticize:
jumpiness between subsequent runs
lack of spread in T_2M and VMAX
COSMO GM – September 2011
Forecasters‘ Feedback
what they prefer to use:
90%-quantile of precipitation
precipitation probabilities for an area (10x10 grid points)
what they appreciate:
early signals for heavy precipitation
indication that deterministic run may be wrong
what they criticize:
jumpiness between subsequent runs
lack of spread in T_2M and VMAX
what they are learning:
dealing with low probabilities (10% probability for extreme weather
issue a warning?)
COSMO GM – September 2011
Deutscher Wetterdienst
COSMO-DE-EPS plans
COSMO-DE-EPS plans (2011-2014)
under consideration:
including past production cycles in product generation
2011
upgrade to 40 members, redesign
2012
reach operational status
statistical postprocessing
initial conditions by LETKF
lateral boundary conditions by ICON EPS
COSMO GM – September 2011
2013
COSMO GM – September 2011