Transcript ppt - Cosmo

Deutscher Wetterdienst
COSMO-DE-EPS
Susanne Theis
Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold
Presentation Overview
 setup of COSMO-DE-EPS
 first results of pre-operational phase
 verification
 forecasters‘ feedback
 COSMO-DE-EPS plans
COSMO GM – September 2011
Deutscher Wetterdienst
Setup of COSMO-DE-EPS
COSMO-DE-EPS status
 pre-operational phase has started:
Dec 9th, 2010
 pre-operational setup:
 20 members
 grid size: 2.8 km
convection-permitting
 lead time: 0-21 hours,
8 starts per day (00, 03, 06,... UTC)
 variations in
physics, initial conditions, lateral boundaries
COSMO GM – September 2011
model domain
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
COSMO GM – September 2011
Boundaries
Model Physics
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
Boundaries
“multi-model”
driven by different
global models
COSMO GM – September 2011
Model Physics
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
“multi-model”
Boundaries
“multi-model”
COSMO-DE initial
driven by different
conditions modified by global models
different global models
COSMO GM – September 2011
Model Physics
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
“multi-model”
Boundaries
“multi-model”
COSMO-DE initial
driven by different
conditions modified by global models
different global models
COSMO GM – September 2011
Model Physics
“multi-configuration”
different configurations
of COSMO-DE model
Generation of Ensemble Members
plus variations of
• initial conditions
• model physics
Ensemble Chain
COSMO-DE-EPS
2.8km
COSMO 7km
GME, IFS, GFS, GSM
COSMO GM – September 2011
BC-EPS
Generation of Ensemble Members
plus variations of
• initial conditions
• model physics
Ensemble Chain
COSMO-DE-EPS
2.8km
COSMO 7km
GME, IFS, GFS, GSM
COSMO GM – September 2011
BC-EPS
BC-EPS is running as a time-critical
application at ECMWF
Generation of Ensemble Members
20 Members
1
IFS
GME
GFS
BC-EPS
COSMO GM – September 2011
GSM
2
3
4
5
Generation of Ensemble Members
Perturbation Methods
Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011:
Uncertainties in COSMO-DE precipitation forecasts introduced by model
perturbations and variation of lateral boundaries. Atmospheric
Research 100, 168-177. (contains status of 2009)
Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the
COSMO-DE-EPS, COSMO Newsletter 11, 115–123.
Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold,
2011: Accounting for initial condition uncertainties in COSMO-DE-EPS.
Submitted to Journal of Geophysical Research.
COSMO GM – September 2011
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
Verification Method
SYNOP
RADAR
 Ensemble Members
 Probabilities of Precipitation
COSMO GM – September 2011
PREC 1h accumulation, threshold: 0.1 mm
DETERMINISTIC SCORES
for Individual Members
0.5
Equitable Threat Score
0.4
JUNE 2011
Do the ensemble members
have different long-term statistics?
(multi-model / multi-configuration)
Are there many cases with
the same „best member“
or „wettest member“?
0.3
0.2
IFS
GME
GFS
GSM
0.1
0.0
0
5
} 20 members
10
15
Forecast Time [h]
COSMO GM – September 2011
20
- look at Equitable Threat Score
- look at Frequency Bias Index
(results similar, not shown)
PREC 1h accumulation, threshold: 0.1 mm
DETERMINISTIC SCORES
for Individual Members
0.5
Equitable Threat Score
0.4
JUNE 2011
Do the ensemble members
have different long-term statistics?
(multi-model / multi-configuration)
Are there many cases with
the same „best member“
or „wettest member“?
0.3
0.2
IFS
GME
GFS
GSM
0.1
0.0
0
5
} 20 members
10
15
20
- look at Equitable Threat Score
- look at Frequency Bias Index
(results similar, not shown)
Forecast Time [h]
Only small differences in long-term statistics
 Members may be treated as equally probable
COSMO GM – September 2011
PREC 1h accumulation
RANK HISTOGRAM
observation...
- …treated as „Ensemble Member“
Frequency
JUNE 2011
- …ranked according to prec amount
at each grid point and forecast hour
How frequent is each rank?
0.05
If ensemble underdispersive
 U-shaped rank histogram
0.00
1
6
11
16
Rank
COSMO GM – September 2011
21
Observation
PREC 1h accumulation
observation...
RANK HISTOGRAM
- …treated as „Ensemble Member“
- …ranked according to prec amount
at each grid point and forecast hour
Frequency
JANUARY 2011
How frequent is each rank?
0.05
If ensemble underdispersive
 U-shaped rank histogram
0.00
1
6
11
16
Rank
COSMO GM – September 2011
21
Observation
PREC 1h accumulation
observation...
RANK HISTOGRAM
- …treated as „Ensemble Member“
- …ranked according to prec amount
at each grid point and forecast hour
Frequency
JANUARY 2011
How frequent is each rank?
0.05
If ensemble underdispersive
 U-shaped rank histogram
0.00
1
6
11
16
Rank
21
Observation
a) Underdispersiveness relatively small
b) Four groups  Many cases with large influence by global models
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JANUARY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
Forecast Time [h]
COSMO GM – September 2011
20
PREC 1h accumulation
BRIER SKILL SCORE
JANUARY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive!  Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JUNE 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive!  Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
MAY - JULY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
- for different precipitation thresholds (x-axis)
(probabilites of exceeding a certain threshold)
0.1 1
2
5
10 20
- for all foreast lead times
Threshold [mm/h]
For larger precipitation amounts (summer):
even more additional value
COSMO GM – September 2011
PREC 1h accumulation
RELIABILITY DIAGRAM
JUNE 2011
log (# fcst)
> 0.1 mm
> 1 mm
> 2 mm
COSMO GM – September 2011
Are the probabilities already
well calibrated?
(without extra calibration)
If we isolate all cases with a
forecast probability of -say- 75-85%
…
did the event occur in 80%
of these cases?
diagonal line: optimal
- for different prec thresholds (colors)
(probs of exceeding a threshold)
PREC 1h accumulation
RELIABILITY DIAGRAM
JUNE 2011
log (# fcst)
> 0.1 mm
> 1 mm
> 2 mm
Are the probabilities already
well calibrated?
(without extra calibration)
If we isolate all cases with a
forecast probability of -say- 75-85%
…
did the event occur in 80%
of these cases?
diagonal line: optimal
- for different prec thresholds (colors)
(probs of exceeding a threshold)
Reliability diagram shows some bias and underdispersiveness
Lines are not flat  additional calibration has good potential
COSMO GM – September 2011
Summary of Verification (Precipitation)
 Ensemble provides additional value to COSMO-DE
(for all accumulations, lead times, precipitation thresholds,…)
 Ensemble underdispersiveness is relatively small
 Ensemble members may be treated as equally probable
 Additional calibration has good potential
COSMO GM – September 2011
Summary of Verification (Precipitation)
 Ensemble provides additional value to COSMO-DE
(for all accumulations, lead times, precipitation thresholds,…)
 Ensemble underdispersiveness is relatively small
 Ensemble members may be treated as equally probable
 Additional calibration has good potential
Pre-operational COSMO-DE ensemble prediction system
already meets fundamental quality requirements for precipitation
COSMO GM – September 2011
Other Variables
 T_2M and VMAX have been verified
 ensemble spread is far too small
 nevertheless, ensemble provides additional value to COSMO-DE
COSMO GM – September 2011
Other Variables
 T_2M and VMAX have been verified
 ensemble spread is far too small
 nevertheless, ensemble provides additional value to COSMO-DE
COSMO-DE ensemble prediction system
has been developed with focus on precipitation
COSMO GM – September 2011
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
Forecasters‘ Feedback
 available products:
see figure
precipitation, snow, wind gusts, T_2m
probability thresholds: warning criteria
 all products on grid-scale (2.8km)
 in addition: precipitation probabilities
for larger areas (10x10 grid boxes)
„probability that the precipitation event
will occur anywhere within the region“
COSMO GM – September 2011
probabilities, quantiles,
ensemble mean,
spread, min, max, …
Forecasters‘ Feedback
 evaluate „full package“
- including the visualization tool
- consistency of products
 select relevant cases
 consider forecasters‘ interpretation
- perception as intended?
- is there any value in the forecast,
additional to forecasters‘ knowledge?
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
 what they appreciate:
 early signals for heavy precipitation
 indication that deterministic run may be wrong
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
 what they appreciate:
 early signals for heavy precipitation
 indication that deterministic run may be wrong
 what they criticize:
 jumpiness between subsequent runs
 lack of spread in T_2M and VMAX
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
 what they appreciate:
 early signals for heavy precipitation
 indication that deterministic run may be wrong
 what they criticize:
 jumpiness between subsequent runs
 lack of spread in T_2M and VMAX
 what they are learning:
 dealing with low probabilities (10% probability for extreme weather
 issue a warning?)
COSMO GM – September 2011
Deutscher Wetterdienst
COSMO-DE-EPS plans
COSMO-DE-EPS plans (2011-2014)
 under consideration:
including past production cycles in product generation
2011
 upgrade to 40 members, redesign
2012
reach operational status
 statistical postprocessing
 initial conditions by LETKF
 lateral boundary conditions by ICON EPS
COSMO GM – September 2011
2013
COSMO GM – September 2011