Transcript Presentation - Copernicus.org

Deutscher Wetterdienst
Preliminary evaluation and verification
of the pre-operational
COSMO-DE Ensemble Prediction System
Susanne Theis
Christoph Gebhardt, Zied Ben Bouallègue, Carlos Peralta, Michael Buchhold
Presentation Overview
 Setup of COSMO-DE-EPS
 First results of pre-operational phase
 Objective verification
 Forecasters‘ feedback; evaluation of case studies
EMS – September 2011
Deutscher Wetterdienst
Setup of COSMO-DE-EPS
COSMO-DE-EPS status
 pre-operational phase has started:
Dec 9th, 2010
 pre-operational setup:
 20 members (upgrade to 40 member in 2012)
 grid size: 2.8 km
convection-permitting
 lead time: 0-21 hours,
8 starts per day (00, 03, 06,... UTC)
 variations in
physics, initial conditions, lateral boundaries
EMS – September 2011
model domain
Generation of Ensemble Members
Variations in Forecast System
for the Representation of Forecast Uncertainty
Initial Conditions
Boundaries
Model Physics
“multi-model”
“multi-model”
“multi-configuration”
COSMO-DE initial
conditions modified by
different global models
driven by different
global models
different configurations
of COSMO-DE model
EMS – September 2011
Generation of Ensemble Members
plus variations of
• initial conditions
• model physics
Ensemble Chain
COSMO-DE-EPS
2.8km
COSMO 7km
GME, IFS, GFS, GSM
EMS – September 2011
BC-EPS
Deutscher Wetterdienst
First Results of Pre-operational Phase
- Objective verification
- Forecasters‘ feedback;
Evaluation of case studies
Verification
Focus on precipitation
SYNOP
RADAR
 Ensemble Members
 Probabilities of Precipitation
EMS – September 2011
PREC 1h accumulation
DETERMINISTIC SCORES
for Individual Members
0.5
Equitable Threat Score
Do the ensemble members have different
long-term statistics? (multi-model / multiconfiguration)
• Are there many cases with the same „best
member“ or „wettest member“?
threshold 0.1 mm
0.4
JUNE 2011
0.3
FBI
0.2
IFS
GME
GFS
GSM
0.1
0.0
0
5
} 20 members
10
15
threshold 1 mm
20
Forecast Time [h]
Only small differences in long-term statistics
 Members may be treated as equally probable
EMS – September 2011
PREC 1h accumulation
How well does the ensemble spread of
the forecast represent the true
variability of the observation?
RANK HISTOGRAM
JUNE 2011
Frequency
Frequency
JANUARY 2011
0.05
0.00
0.05
0.00
1
6
11
16
Rank
21
1
6
11
16
Rank
21
a) Underdispersiveness relatively small
b) Four groups  Many cases with large influence by global models
EMS – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JANUARY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive!  Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
EMS – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JUNE 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive!  Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
EMS – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
MAY - JULY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
- for different precipitation thresholds (x-axis)
(probabilites of exceeding a certain threshold)
0.1 1
2
5
10 20
- for all foreast lead times
Threshold [mm/h]
For larger precipitation amounts (summer):
even more additional value
EMS – September 2011
PREC 1h accumulation
RELIABILITY DIAGRAM
JUNE 2011
log (# fcst)
> 0.1 mm
> 1 mm
> 2 mm
Are the probabilities already
well calibrated?
(without extra calibration)
If we isolate all cases with a
forecast probability of -say- 75-85%
…
did the event occur in 80%
of these cases?
diagonal line: optimal
- for different prec thresholds (colors)
(probs of exceeding a threshold)
Reliability diagram shows some bias and underdispersiveness
Lines are sloped  additional calibration has good potential
EMS – September 2011
Summary of Verification (Precipitation)
 Ensemble provides additional value to COSMO-DE
(for all accumulations, lead times, precipitation thresholds,…)
 Ensemble underdispersiveness is relatively small
 Ensemble members may be treated as equally probable
 Additional calibration has good potential
Pre-operational COSMO-DE ensemble prediction system
already meets fundamental quality requirements for precipitation
EMS – September 2011
Other Variables
V_MAX 10m Jan./Feb./ Mar.
 T_2M and VMAX have been verified
 ensemble spread is far too small
 nevertheless, ensemble provides
additional value to COSMO-DE
COSMO-DE ensemble prediction system
has been developed with focus on precipitation
Upgrade to 40 members will also look at other variables
EMS – September 2011
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback;
evaluation of case studies
22.07.2011 Severe weather Rostock
Observation: 12h-precip, 18 UTC, Synop and Radar
22.07.2011 Rostock (2)
COSMO-DE: 12h-precip
•Synop: 18 UTC, 12h precip
• COSMO-DE 00 UTC + 18 h
EMS – September 2011
• Synop: 18 UTC, 12h precip
• COSMO-DE 06 UTC + 12 h
22.07.2011 Rostock (3)
COSMO-DE-EPS: Prob > 25 mm, 12h-RR 06-18 UTC
21 UTC-run (previous day)
00 UTC-run
03 UTC-run
+21h
+18
+15
Good consistency of successive model runs
EMS – September 2011
22.07.2011 Rostock (4)
COSMO-DE-EPS: Prob > 40 mm, 12h-RR 06-18 UTC
21 UTC-Lauf (Vortag)
00 UTC-Lauf
03 UTC-Lauf
+21h
+18
+15
Probabilities provide good information on location and intensity of the event
EMS – September 2011
22.07.2011 Rostock (2)
Prob > 40 mm, 12h-prec 06-18 UTC
12h-prec, SRNWP PEPS 00+18h
12h-prec, COSMO-DE-EPS 00+18h
COSMO-DE-EPS is able to capture heavy convective precip events
provides optimal guidance for forecasters (in this case)
EMS – September 2011
4.8.2011: Severe storm Bremen
4.8.2011, 06 UTC 1h precip [mm/h]
• Synop
• Radar
4.8.2011, 06 UTC 1h precip [mm/h]
• Synop
• COSMO-DE deterministic 21 UTC + 09 h
Forecaster: Where to expect the main shower
activity and what will be maximum intensity?
EMS – September 2011
4.8.2011: Bremen (2)
• Synop 06 UTC, 1h precip
• C-EPS upscaled prob >15mm 18 UTC + 12 h
Probability defines area of
occurrence
EMS – September 2011
• Synop 06 UTC,
1h precip
• C-EPS Max of all members 18 UTC + 12 h
Maxmem (or 90% quantile) provides
estimation of possible intensities
18.8.2011: cold front NRW
• Radar and Synop: 19 UTC, 1h precip
• Radar and Synop: 19 UTC, 1h precip
• COSMO-DE deterministic 12 UTC + 07 h
Large location error in C-DE, horizontal extent of severe precip. too small
EMS – September 2011
18.8.2011: cold front NRW (2)
• Radar, Synop: 19 UTC, 1h precip
• C-DE-EPS upscaled Prob >15mm/h 12 UTC + 7 h
• Radar, Synop: 19 UTC, 1h precip
• C-DE-EPS Perc90% 12 UTC + 7 h
C-DE-EPS: better localization and intensity estimation
But: BC-EPS does not adequately capture uncertainty on synoptic scale
EMS – September 2011
24.8.2011: severe storm Frankfurt
• Radar 19 UTC, 1h precip
• Synop 19 UTC, max wind (1h)
• COSMO-DE max wind (1h), 09 UTC + 08 h
• Synop 19 UTC, max wind (1h)
C-DE determ. run: no indication of gale-force winds (fx>29 m/s)
EMS – September 2011
25.8.2011: Frankfurt (2)
• C-EPS 90%quant max wind (1h), 06 UTC + 11 h
• Synop 19 UTC, max wind (1h)
• C-EPS maxwind prob > 29m/s 06 UTC + 11 h
• Synop 19 UTC, max wind (1h)
C-DE -EPS: clear signals for gale-force winds above 29 m/s. Useful
guidance for forecasters despite of timing error
EMS – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile or maximum of all members of precipitation
 precipitation probabilities for an area (10x10 grid points)
 what they appreciate:
 early signals for heavy precipitation
 indication that deterministic run may be wrong
 what they criticize:
 jumpiness between subsequent runs
 lack of spread in T_2M and VMAX
 what they are learning:
 dealing with low probabilities (10% probability for extreme weather
 issue a warning?)
EMS – September 2011
EMS – September 2011
Generation of Ensemble Members
Perturbation Methods
Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold,
2011: Accounting for initial condition uncertainties in COSMO-DE-EPS.
Submitted to Journal of Geophysical Research.
Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the
COSMO-DE-EPS, COSMO Newsletter 11, 115–123.
Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011:
Uncertainties in COSMO-DE precipitation forecasts introduced by model
perturbations and variation of lateral boundaries. Atmospheric
Research 100, 168-177. (contains status of 2009)
EMS – September 2011
COSMO-DE-EPS plans (2011-2014)
 upgrade to 40 members, redesign
2011
reach operational status
2012
 statistical postprocessing
 initial conditions by LETKF
 lateral boundary conditions by ICON EPS
EMS – September 2011
2013