Transcript ppt - Cosmo

Deutscher Wetterdienst
COSMO-DE-EPS
Susanne Theis
Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold
Presentation Overview
 setup of COSMO-DE-EPS
 first results of pre-operational phase
 verification
 forecasters‘ feedback
 enlarging the sample at low cost
 looking into past production cycles
COSMO GM – September 2011
Presentation Overview
 setup of COSMO-DE-EPS
 first results of pre-operational phase
 verification
 forecasters‘ feedback
new
 enlarging the sample at low cost
 looking into past production cycles
COSMO GM – September 2011
new
Deutscher Wetterdienst
Setup of COSMO-DE-EPS
COSMO-DE-EPS status
 pre-operational phase has started:
Dec 9th, 2010
 pre-operational setup:
 20 members
 grid size: 2.8 km
convection-permitting
 lead time: 0-21 hours,
8 starts per day (00, 03, 06,... UTC)
 variations in
physics, initial conditions, lateral boundaries
COSMO GM – September 2011
model domain
Generation of Ensemble Members
plus variations of
• initial conditions
• model physics
Ensemble Chain
COSMO-DE-EPS
2.8km
COSMO 7km
GME, IFS, GFS, GSM
COSMO GM – September 2011
BC-EPS
Generation of Ensemble Members
plus variations of
• initial conditions
• model physics
Ensemble Chain
COSMO-DE-EPS
2.8km
COSMO 7km
GME, IFS, GFS, GSM
COSMO GM – September 2011
BC-EPS
BC-EPS is running as a time-critical
application at ECMWF
Generation of Ensemble Members
Perturbation Methods
Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold,
2011: Accounting for initial condition uncertainties in COSMO-DE-EPS.
Submitted to Journal of Geophysical Research.
Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the
COSMO-DE-EPS, COSMO Newsletter 11, 115–123.
Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011:
Uncertainties in COSMO-DE precipitation forecasts introduced by model
perturbations and variation of lateral boundaries. Atmospheric
Research 100, 168-177. (contains status of 2009)
COSMO GM – September 2011
COSMO-DE-EPS plans (2011-2014)
 upgrade to 40 members, redesign
2011
reach operational status
2012
 statistical postprocessing
 initial conditions by LETKF
 lateral boundary conditions by ICON EPS
COSMO GM – September 2011
2013
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
new
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
new
Verification Method
SYNOP
RADAR
 Ensemble Members
 Probabilities of Precipitation
COSMO GM – September 2011
PREC 1h accumulation, threshold: 0.1 mm
DETERMINISTIC SCORES
for Individual Members
0.5
Equitable Threat Score
0.4
JUNE 2011
Do the ensemble members
have different long-term statistics?
(multi-model / multi-configuration)
Are there many cases with
the same „best member“
or „wettest member“?
0.3
0.2
IFS
GME
GFS
GSM
0.1
0.0
0
5
} 20 members
10
15
Forecast Time [h]
COSMO GM – September 2011
20
- look at Equitable Threat Score
- look at Frequency Bias Index
(results similar, not shown)
PREC 1h accumulation, threshold: 0.1 mm
DETERMINISTIC SCORES
for Individual Members
0.5
Equitable Threat Score
0.4
JUNE 2011
Do the ensemble members
have different long-term statistics?
(multi-model / multi-configuration)
Are there many cases with
the same „best member“
or „wettest member“?
0.3
0.2
IFS
GME
GFS
GSM
0.1
0.0
0
5
} 20 members
10
15
20
- look at Equitable Threat Score
- look at Frequency Bias Index
(results similar, not shown)
Forecast Time [h]
Only small differences in long-term statistics
 Members may be treated as equally probable
COSMO GM – September 2011
PREC 1h accumulation
RANK HISTOGRAM
observation...
- …treated as „Ensemble Member“
Frequency
JUNE 2011
- …ranked according to prec amount
at each grid point and forecast hour
How frequent is each rank?
0.05
If ensemble underdispersive
 U-shaped rank histogram
0.00
1
6
11
16
Rank
COSMO GM – September 2011
21
Observation
PREC 1h accumulation
observation...
RANK HISTOGRAM
- …treated as „Ensemble Member“
- …ranked according to prec amount
at each grid point and forecast hour
Frequency
JANUARY 2011
How frequent is each rank?
0.05
If ensemble underdispersive
 U-shaped rank histogram
0.00
1
6
11
16
Rank
COSMO GM – September 2011
21
Observation
PREC 1h accumulation
observation...
RANK HISTOGRAM
- …treated as „Ensemble Member“
- …ranked according to prec amount
at each grid point and forecast hour
Frequency
JANUARY 2011
How frequent is each rank?
0.05
If ensemble underdispersive
 U-shaped rank histogram
0.00
1
6
11
16
Rank
21
Observation
a) Underdispersiveness relatively small
b) Four groups  Many cases with large influence by global models
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JANUARY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
Forecast Time [h]
COSMO GM – September 2011
20
PREC 1h accumulation
BRIER SKILL SCORE
JANUARY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive!  Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
JUNE 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
> 0.1 mm
> 1 mm
> 2 mm
- for different precipitation thresholds (colors)
(probabilites of exceeding a certain threshold)
- for different forecast lead times (x-axis)
0
5
10
15
20
Forecast Time [h]
Always positive!  Ensemble provides additional value to COSMO-DE
Additional value grows with lead time (less deterministic predictability)
COSMO GM – September 2011
PREC 1h accumulation
BRIER SKILL SCORE
MAY - JULY 2011
How good are the probabilities
derived from the ensemble?
compared to the deterministic COSMO-DE
(always forecasting 0% or 100%)
Look at Brier Skill Score (no skill: zero)
- for different precipitation thresholds (x-axis)
(probabilites of exceeding a certain threshold)
0.1 1
2
5
10 20
- for all foreast lead times
Threshold [mm/h]
For larger precipitation amounts (summer):
even more additional value
COSMO GM – September 2011
PREC 1h accumulation
RELIABILITY DIAGRAM
JUNE 2011
log (# fcst)
> 0.1 mm
> 1 mm
> 2 mm
COSMO GM – September 2011
Are the probabilities already
well calibrated?
(without extra calibration)
If we isolate all cases with a
forecast probability of -say- 75-85%
…
did the event occur in 80%
of these cases?
diagonal line: optimal
- for different prec thresholds (colors)
(probs of exceeding a threshold)
PREC 1h accumulation
RELIABILITY DIAGRAM
JUNE 2011
log (# fcst)
> 0.1 mm
> 1 mm
> 2 mm
Are the probabilities already
well calibrated?
(without extra calibration)
If we isolate all cases with a
forecast probability of -say- 75-85%
…
did the event occur in 80%
of these cases?
diagonal line: optimal
- for different prec thresholds (colors)
(probs of exceeding a threshold)
Reliability diagram shows some bias and underdispersiveness
Lines are not flat  additional calibration has good potential
COSMO GM – September 2011
Summary of Verification (Precipitation)
 Ensemble provides additional value to COSMO-DE
(for all accumulations, lead times, precipitation thresholds,…)
 Ensemble underdispersiveness is relatively small
 Ensemble members may be treated as equally probable
 Additional calibration has good potential
COSMO GM – September 2011
Summary of Verification (Precipitation)
 Ensemble provides additional value to COSMO-DE
(for all accumulations, lead times, precipitation thresholds,…)
 Ensemble underdispersiveness is relatively small
 Ensemble members may be treated as equally probable
 Additional calibration has good potential
Pre-operational COSMO-DE ensemble prediction system
already meets fundamental quality requirements for precipitation
COSMO GM – September 2011
Other Variables
 T_2M and VMAX have been verified
 ensemble spread is far too small
 nevertheless, ensemble provides additional value to COSMO-DE
COSMO GM – September 2011
Other Variables
 T_2M and VMAX have been verified
 ensemble spread is far too small
 nevertheless, ensemble provides additional value to COSMO-DE
COSMO-DE ensemble prediction system
has been developed with focus on precipitation
Upgrade to 40 members will also look at other variables
COSMO GM – September 2011
Deutscher Wetterdienst
First Results of Pre-operational Phase
- verification
- forecasters‘ feedback
new
Forecasters‘ Feedback
 available products:
see figure
precipitation, snow, wind gusts, T_2m
probability thresholds: warning criteria
 all products on grid-scale (2.8km)
 in addition: precipitation probabilities
for larger areas (10x10 grid boxes)
„probability that the precipitation event
will occur anywhere within the region“
COSMO GM – September 2011
probabilities, quantiles,
ensemble mean,
spread, min, max, …
Forecasters‘ Feedback
 evaluate „full package“
- including the visualization tool
- consistency of products
 select interesting cases
 consider forecasters‘ interpretation
- perception as intended?
- is there any value in the forecast,
additional to forecasters‘ knowledge?
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
 what they appreciate:
 early signals for heavy precipitation
 indication that deterministic run may be wrong
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
 what they appreciate:
 early signals for heavy precipitation
 indication that deterministic run may be wrong
 what they criticize:
 jumpiness between subsequent runs
 lack of spread in T_2M and VMAX
COSMO GM – September 2011
Forecasters‘ Feedback
 what they prefer to use:
 90%-quantile of precipitation
 precipitation probabilities for an area (10x10 grid points)
 what they appreciate:
 early signals for heavy precipitation
 indication that deterministic run may be wrong
 what they criticize:
 jumpiness between subsequent runs
 lack of spread in T_2M and VMAX
 what they are learning:
 dealing with low probabilities (10% probability for extreme weather
 issue a warning?)
COSMO GM – September 2011
Deutscher Wetterdienst
Enlarging the Sample at Low Cost
new
- looking into past production cycles
Deutscher Wetterdienst
Enlarging the Sample at Low Cost
new
- looking into past production cycles
- Introduction
- Verification Study
- Recommendation
COSMO-DE-EPS Overview
 start of pre-operational phase / evaluation
 20 members  probabilities, quantiles, etc
 runs at 00 UTC, 03 UTC, 06 UTC,…
COSMO GM – September 2011
December 2010
COSMO-DE-EPS Overview
 start of pre-operational phase / evaluation
December 2010
 20 members  probabilities, quantiles, etc
 runs at 00 UTC, 03 UTC, 06 UTC,…
 start of pre-operational phase
Q1 2012
with 40 ensemble members
 reach operational status
End of 2012
 statistical postprocessing
2013
of probabilities of precipitation
COSMO GM – September 2011
COSMO-DE-EPS Overview
 start of pre-operational phase / evaluation
December 2010
 20 members  probabilities, quantiles, etc
 runs at 00 UTC, 03 UTC, 06 UTC,…
 start of pre-operational phase
further
improvements ?
Q1 2012
with 40 ensemble members
 reach operational status
End of 2012
 statistical postprocessing
2013
of probabilities of precipitation
COSMO GM – September 2011
Enlarging the Sample at Low Cost
 looking into past production cycles
0 UTC
15 UTC
18UTC
} 20 members
21UTC
} 20 members
00UTC
} 20 members
COSMO GM – September 2011
Enlarging the Sample at Low Cost
 looking into past production cycles
0 UTC
15 UTC
18UTC
} 20 members
21UTC
} 20 members
00UTC
} 20 members
}
60 members
COSMO GM – September 2011
Important Check
 How good are forecasts from past production cycles?
COSMO GM – September 2011
Important Check
 How good are forecasts from past production cycles?
Forecast
Start:
FBI
June 2011
precipitation
1h accumulations
threshold: 1 mm/h
ETS
similar results for
- different months
- different starting times
- different thresholds
- 6h accumulations
00UTC
21UTC
18UTC
0 2
4 6
8 10 12 14
time of day [UTC]
COSMO GM – September 2011
0 2
4 6
8 10 12 14
time of day [UTC]
Important Check
 How good are forecasts from past production cycles?
Forecast
Start:
FBI
June 2011
precipitation
1h accumulations
threshold: 1 mm/h
ETS
similar results for
- different months
- different starting times
- different thresholds
- 6h accumulations
00UTC
21UTC
18UTC
0 2
4 6
8 10 12 14
time of day [UTC]
0 2
4 6
8 10 12 14
time of day [UTC]
The quality is similar.
Exception: Most recent run with forecast lead time 0-3 hours is the best.
COSMO GM – September 2011
Probability of Precipitation > 10 mm/h
Resulting Probabilities
20 members
20 + 20 + 20 members
2011-05-22 15UTC
Forecast Start: 09 UTC
COSMO GM – September 2011
%
Forecast Start: 09 UTC, 06 UTC, 03 UTC
Summary of Study
 looking into past production cycles (20+20+20)
 quality gain for precipitation
(except first 2 forecast hours)
 does not harm quality of T_2M and VMAX_10M
 also applicable to „area probabilities“
Recommendation:
Look into past production cycles
COSMO GM – September 2011
Technical aspect:
how to derive probabilities from 20+20+20 members
0 UTC
15 UTC
18UTC
21UTC
00UTC
21h forecast in (pre-)operational mode
+ 6h additional lead time required
COSMO GM – September 2011
21UTC
COSMO GM – September 2011
Deutscher Wetterdienst
Extra Slides
Deutscher Wetterdienst
Example
Resulting Probabilities
P(TOT_PREC) > 10mm/h 2011-05-22 : 00UTC 14-15h
20 members
COSMO GM – September 2011
20 + 20 + 20 members
Resulting Probabilities
P(TOT_PREC) > 10mm/h 2011-05-22 : 03UTC 11-12h
20 members
COSMO GM – September 2011
20 + 20 + 20 members
Resulting Probabilities
P(TOT_PREC) > 10mm/h 2011-05-22 : 06UTC 08-09h
20 members
COSMO GM – September 2011
20 + 20 + 20 members
Resulting Probabilities
P(TOT_PREC) > 10mm/h 2011-05-22 : 09UTC 05-06h
20 members
COSMO GM – September 2011
20 + 20 + 20 members
Resulting Probabilities
P(TOT_PREC) > 10mm/h 2011-05-22 : 12UTC 02-03h
20 members
COSMO GM – September 2011
20 + 20 + 20 members
Resulting Probabilities
P(TOT_PREC) > 10mm/h 2011-05-22 : 12UTC 02-03h
20 members
20 + 20 + 20 members
very nice side effect:
less “jumpiness”
COSMO GM – September 2011
Enlarging the Sample at Low Cost (2)
 looking into a spatial neighbourhood
at each grid point
Schwartz et al., 2010
Wea. Forecasting
only applicable to probabilites at a grid point
not applicable to „area probabilities“
COSMO GM – September 2011
Probability of Precipitation > 5 mm/h
2011-06-16 20UTC
Example
20 members
20 members
+ neighbourhood (radius 10 Δx)
%
Forecast Start: 12 UTC
COSMO GM – September 2011
Forecast Start: 12 UTC
Probability of Precipitation > 1 mm/h
Quality Gain in Resulting Probabilities
Resolution Gain
Reliability Gain
20 + 20 + 20
members
Sharpness Loss
most important
20 members
+ neighbourh.
reference:
20 members
0
5
10
time of day [UTC]
15
0
5
10
15 0
time of day [UTC]
5
10
15
June 2011
time of day [UTC]
Definition of scores:
Ben Bouallègue, Z., 2011: Upscaled and fuzzy probabilistic foreasts: verification results.
COSMO Newsletter 11, 124-132.
COSMO GM – September 2011
Probability of Precipitation > 1 mm/h
Quality Gain in Resulting Probabilities
Resolution Gain
Reliability Gain
20 + 20 + 20
members
Sharpness Loss
most important
20 members
+ neighbourh.
reference:
20 members
0
5
10
time of day [UTC]
15
0
5
10
15 0
time of day [UTC]
5
10
15
June 2011
time of day [UTC]
Precipitation:
Both methods achieve clear quality gain (except first 2 hours).
COSMO GM – September 2011
Probability of Precipitation
Quality Gain in Resulting Probabilities
Resolution Gain
Reliability Gain
Sharpness Loss
most important
20 + 20 + 20
members
20 members
+ neighbourh.
reference:
20 members
0.1 1.
2.
5.
threshold [mm/h]
0.1 1.
2.
5.
threshold [mm/h]
0.1 1.
2.
5.
threshold [mm/h]
Same conclusion for different precipitation thresholds
COSMO GM – September 2011
June 2011
Probability of 2m-Temperature
2m-Temperature (T_2M)
Resolution Gain
Reliability Gain
Sharpness Loss
most important
20 + 20 + 20
members
20 members
+ neighbourh.
reference:
20 members
10
20
30
threshold [°C]
10
20
30
threshold [°C]
10
20
threshold [°C]
For 2m-temperature:
20+20+20 does not harm quality
COSMO GM – September 2011
30
June 2011
Probability of Wind Gusts
Wind Gusts (VMAX_10M)
Resolution Gain
Reliability Gain
Sharpness Loss
most important
20 + 20 + 20
members
20 members
+ neighbourh.
reference:
20 members
14
18
threshold [m/s]
14
18
threshold [m/s]
14
18
threshold [m/s]
For wind gusts:
20+20+20 does not harm quality
COSMO GM – September 2011
June 2011
Weights?
Gain compared to L1
F = (1- W) L1 + W L2
COSMO GM – September 2011
0<W<1
Technical Setup of the Ensemble Chain
12
4 global
models
06
00
12
18
IFS
start
BC-EPS
at ECMWF
arrival BC-EPS
at DWD
start
COSMO-DE-EPS
COSMO GM – September 2011
18
12
12
+06
12
18
15
+09
15
18
00
21
00
06
03
06
12
09
12
18
15
18
+06 +09
+06 +09
+06 +09
+06 +09
+06
18
00
06
12
18
21
03
09
15
Looking back how far?
00UTC
09UTC
2runs 3runs 4runs
BSS raw ensemble (1run) as reference
COSMO GM – September 2011
2runs 3runs 4runs