Transcript PowerPoint Part2 - c

MAP IOP2b
LAMBO-LEPS vs LOKAL-LEPS:
- LAMBO at 20 km
- Lokal Modell at 7 km
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
MAP IOP 2B:
P > 20mm/24h
LAMBO-LEPS vs Lokal-LEPS
P > 50mm/24h
P > 20mm/24h
P > 50mm/24h
20 km
7 km
LEPS
probability maps for 24h cumulated precipitation exceeding
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia,threshold
Madrid, 3-4 October
2002 hours
at +66
Some more MAP cases
For each case observations are shown together with
associated LEPS probability maps.
The five different LEPS scenarios, from which
probabilities are computed, are also shown.
From these cases it is evident a good capability of
LEPS to identify high precipitation areas.
Good behaviour with respect to false alarms.
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
19-20 September 1999 hp
obs
> 10 mm
> 50 mm
> 20 mm
> 100 mm
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
RM 1
RM 2
RM 3
RM 4
RM 5
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
02-03 November 1999
> 10 mm
> 20 mm
> 50 mm
> 100 mm
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
RM 1
RM 2
RM 3
RM 4
RM 5
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
12-13 November 1999
> 10 mm
> 20 mm
hp
> 50 mm
> 100 mm
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
RM 1
RM 2
RM 3
RM 4
RM 5
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Need for a more systematic
verification/diagnosis
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
LEPS VERIFICATION on the MAP SOPs
(15 “mixed”cases of strong and weak precip)
•ECMWF EPS system: TL159L40 (hor. res.~120 km)
•Limited Area Models
 LAMBO 20 km h.r.
•Obs. precipitation from the high-res. MAP data set
•Super-ensemble 102 cases
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
LEPS VERIFICATION on the MAP SOPs
(15 cases)
 Use of probabilistic (and deterministic) indices:
 ROC Curve (hit rate vs. false alarms)
 ROC area (integral of ROC-C over false alarms)
 Brier Score and Brier Skill Score
 Cost-Loss Analysis
Hypothesis testing by RESAMPLING (statistical signif.)
(HAMILL, Weather and Forecasting 1999)
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
ROC curve and ROC area
contingency
table
Forecast
Observed
Yes
No
Yes
a
b
No
c
d
a
Hit Rate (H) =
ac
b
False Alarm Rate (F) = b  d
A contingency table can be built
each probability
class
probability
class can
becan
defined
as the For
1.for
A contingency
table
for(aeach
probability
class
be built.
% of ensemble elements which actually forecast a given event). For the k-th probability class:
the k-th probability class:
M
Hk =
H
Fk =
F
i
i k
M
i
i k
1. Hit rates are plotted against the corresponding false alarm rates
to generate the ROC Curve.
2. The area under the ROC Curve (ROC area) is a probabilistic
index
The area under the ROC curve
is used as a statistic measure of forecast usefulness
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Brier Skill Score
Brier Score
1 n
2
BS    f k  ok 
n k 1
•oik
= 1 if the event occurs
= 0 if the event does not occur
•fk is the probability of occurrence
according to the forecast system (e.g.
the fraction of ensemble members
forecasting the event)
•BS can take on values in the range
[0,1], a perfect forecast having BS = 0
Brier Skill Score
BS  BS
BSS  cli
BS cli
BS cli  o 1  o 
The forecast system has predictive skill if
BSS is positive, a perfect system having
BSS = 1.
o = total frequency of the event
(sample climatology)
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Brier Score decomposition
1
BS 
N
M
 N k  f k  ok 
2
k 0
1

N
reliability
M

k 0
N k ok  o   o 1  o 
resolution
2
2
uncertainty
M
k 

 k 
BS  o  H k 1    1  o  Fk  
 M
M 
k 0
k 0
M
Hit Rate term
2
False Alarm Rate term
k = 0, …, M
number of ensemble members
forecasting the event (probability classes)
M = ensemble size
N = total number of point in the verification domain
Nk = number of points where the event is forecast by k members.
Nk
ok   oi frequency of the event in the sub-sample N
k
i 1
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Scores computed against observations on 15 cases
213 214 21
10 mm
107
Most recent EPS
of 51 members
834 838 753 750
-18
50 mm
SuperEnsemble:
102 (51+51) EPS
members
22
105 129
742 740 751 749
LEPS with the 5
runs Weighted with
clusters population
Leps with the 5
runs equally
weighted
Hypothesis testing by RESAMPLING
(Hamill, Weather and Forecasting 1999)
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Comparison of scores computed against observations
grouping cases in two different samples (weighted)
ROC
BSS
Solid line:
the 4 most precipitating cases
Dashed line:
the remaining 11 cases
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
VERIFICATION:
COST-LOSS ANALYSIS
•The event E causes a damage which incur a loss L.
The user U can avoid the damage by taking a
preventive action which cost is C.
•U wants to minimize the mean total expense over a
great number of cases.
Decisional
model
U take
action
•U can rely on a forecast system to know in advance
if the event is going to occur or not.
E happens
yes
no
yes
C
C
no
L
0
•If the forecast system is probabilistic, the user has
to fix a probability threshold. When this threshold is
exceeded, it take protective action.
contingency
table
Forecast
Observed
Yes
No
Yes
a
b
No
c
d
MEk=
Fk
Vk =
C
1  o   H k o 1  C   o
L
L

MEcli  ME k f
MEcli  MEp
Mean
expense
Value
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
VERIFICATION:
COST-LOSS ANALYSIS
Value of the forecast
systems (expressed
as a percentage of
the value of a
perfect
forecast
system)
as
a
function of the costloss
ratio.
The
systems are: LEPS
(green lines), EPS
102-member (blue
line) and 3-day 51member (cyan line)
and EPS 5-member
(red
lines).
The
considered event is
precipitation
exceeding
50mm/24h.
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
WEIGHTING PROCEDURE
ECMWF-EPS
Percentage of times the best member of the 5-RM EPS belongs to the most
populated cluster, to the two most populated clusters, to the i-most
populated clusters as a function of i.
Z700 ana: forecast
geopotential height at 700 hPa
against ECMWF analysis
tp ana: forecast precipitation
against +24h forecast
(“analysis”)
tp obs: forecast precipitation
interpolated on station points
against MAP observations
tp obs2ecbox: forecast
precipitation against MAP
observation averaged over
ECMWF TEPS grid boxes
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
WEIGHTING PROCEDURE
LEPS
Percentage of times the best member of the LEPS belongs to the most
populated cluster, to the two most populated clusters, to the i most
populated clusters as a function of i.
Z700 ana: forecast
geopotential height at 700 hPa
against ECMWF analysis
tp ana: forecast precipitation
against MAP analysed
precipitation (Frei and Haller)
tp obs: forecast precipitation
interpolated on station points
against MAP observations
tp obs2ecbox: forecast
precipitation averaged over
ECMWF TEPS grid boxes against
observation averaged over the
same boxes
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Scores computed against observations on 15 cases
COMMENTS
Global Ensemble without member reduction performs
better at low-moderate precipitation thresholds.
LEPS seems to give a valuable improvement at high
precipitation thresholds.
Statistics is still poor: more cases are needed
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
From LEPS to COSMO-LEPS
From LEPS
to
COSMO-LEPS
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
From LEPS to COSMO-LEPS
On the basis of these results, LEPS seems to give a valuable
contribution in forecasting heavy precipitation; good performance
also with respect to false-alarms.
To better assess the quality of the system more statistics is required
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
From LEPS to COSMO-LEPS
COSMO Consortium agreed to ask ECMWF for the necessary
support to test COSMO-LEPS on a regular basis, making use of
the additional EPS 00GMT integration targeted for extreme
events forecasting
The proposal has been accepted by ECMWF and the system is
currently being implemented: the required ECMWF computer
resources will be provided by COSMO members national
allocations
ECMWF will provide mass storage requirements and general
support
ARPA-SMR will manage the COSMO-LEPS suite
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
COSMO-LEPS super-ensemble
oldest eps
153
members
(+144)
middle eps
(+132)
youngest eps
00 12
day
n –1
00 12
n
00 12
n+1
(+120)
00 12
+2
00 12
00 12
+3
+4
00
+5
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
COSMO-LEPS POSSIBLE PRODUCTS
Lokal
Model
scenarios
Deterministic products
for each of the 5 LM
runs:
Precipitation
Transfer to Cosmo Mean Sea level pressure
Partners
and other interested 700 hPa Geopotential
Member States
850 hPa Temperature
Probability Maps

prob of 24h rainfall exceeding

prob of 24h Tmax
exceeding 20,30,35,40 C

prob of 24h Tmin
below -10,-5,0,+5 C

prob of 24h Vmax
exceeding 10,15,20,25 m/s

20,50,100,150 mm
prob of 24h snowfall exceeding 1,5,10,20 cm
of equivalent water (to be checked)
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
An example of a test case in
quasi-operational configuration
IOP-15 6-7 Nov 1999
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
6-7 November 1999 – IOP15
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
6-7 November 1999 – IOP15
Results in COSMO-LEPS
Configuration – (Italian domain)
3 global ensembles (SE): Tl255 L40 EPS (Dx about 80 km)
LEPS: Lokal Modell – 7 km hor. Res. Nested on 5 RMs of SE
+24
+12
2/11/99 00 GMT
2/11/99 12 GMT
3/11/99 00 GMT
Start of
integrations
Verification period
24h 6/11/99 06 GMT
7/11/99 06 GMT
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
6-7 November 1999 – IOP15
Results in COSMO-LEPS
Configuration – (Italian domain)
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
6-7 November 1999 – IOP15
Results in COSMO-LEPS
Configuration – (Italian
domain)
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
COSMO-LEPS future plans
•June 2002: procedure implementation has started
•September 2002: start of COSMO-LEPS regular suite
(hopefully)
Within the COSMO consortium, COSMO-LEPS
verification will be coordinated to get the best possible
evaluation of the system. The verification activity will
address unanswered questions and will drive future
developments.
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Any conclusions?
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
 The LEPS system is theoretically “reasonable” but still
requires a-posteriori practical validation
 The LEPS system is realistic in terms of resources required
 From these cases it emerges a good capability of the LEPS
technique to identify high precipitation areas
 In particular, LEPS seems to give added value contribution
in probabilistic forecasting of heavy precipitation at the
highest thresholds
 Good performance also with respect to false-alarms
 As always, to assess more reliably the quality of the system
more statistics is required but will be provided by the
COSMO-LEPS database
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
References
Molteni F., R. Buizza, C.Marsigli, A.Montani, F.Nerozzi and T.Paccagnella, 2001: A
strategy for high-resolution ensemble prediction. Part I: definition of representative
members and global-model experiments. Q.J.R.Meteorol.Society, 2001, 127, pp.20692094.
Marsigli C., A.Montani, F.Nerozzi, T.Paccagnella, S. Tibaldi, F.Molteni and R.
Buizza, 2001: A strategy for high-resolution ensemble prediction. Part II: limited
area experiments in four alpine flood events. Q.J.R.Meteorol.Society, 2001, 127,
pp.2095-2115
Montani A., C. Marsigli, F. Nerozzi, T. Paccagnella and R. Buizza, 2001:
Performance of the limited area ensemble prediction system for cases of heavy
rainfall. Nonlinear Processes in Geophysics (NPG) , 2001, 25:123-135.
Montani A., C. Marsigli, F. Nerozzi, T. Paccagnella, S. Tibaldi and R. Buizza: The
Soverato flood in Southern Italy: performance of global and limited-area ensemble
forecasts. Submitted to Nonlinear Processes in Geophysics (NPG).
Marsigli C., A.Montani, F.Nerozzi and T.Paccagnella: Objective verification of the
ARPA-SMR Limited Area Ensemble Prediction System. To be submitted.
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Time Trend of TEPS at +3d and at +5d (45 cases)
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Spread: TEPS started at 6 Feb 2000, 12 UTC
The same 700hPa
isoline plotted for the
51 Members at + 72
The same 700hPa
isoline plotted for the
51 Members at + 120
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Small worsening of both (SE)
with respect to FC D+3 only
for Z500 anomalies
PROB SCORES
Z 500 ANOM
DJF 1999-2000
AREA 28°N - 60°N; 18°W - 45°E
(WHOLE ENSEMBLE)
Z 500 ANOM
DJF 1999-2000
AREA 28°N - 60°N; 18°W - 45°E
(5 RM COMPLETE LINKAGE)
800
600
600
400
400
FC D+3
FC D+5
both
200
BSS
TP ACC 24 H
DJF 1999-2000
AREA 28°N - 60°N; 18°W - 45°E
(WHOLE ENSEMBLE)
1
FC D+3
FC D+5
both
200
BSS
0
0
-10
-5
0
5
10
-10
-5
THRESHOLD [dam]
0
THR = 1 mm
0.8
5
10
THRESHOLD [dam]
HIT RATE
800
0.6
0.4
0.2
TP ACC 24 H DJF 1999-2000
AREA 28°N - 60°N; 18°W - 45°E
(WHOLE ENSEMBLE)
TP ACC 24 H DJF 1999-2000
AREA 28°N - 60°N; 18°W - 45°E
(5 RM COMPLETE LINKAGE)
300
200
0
100
0
0
0
5
-100
FC D+3
FC D+5
both
200
100
10
ROC
0
300
FC D+3
FC D+5
both
0
5
FC D+3
FC D+5
both
10
0.2
0.4
0.6
0.8
FALSE ALARM RATE
Appreciable effect of
member reduction
-100
BSS
BSS
-200
-200
THRESHOLD [mm]
THRESHOLD [mm]
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
1
Soverato flood
EPS TL255-51
members
EPS TL159- 51
members
TEPS TL159- 51
members
ECMWF
probability
maps
for 24h
WORKSHOP ON
SHORT-RANGE ENSEMBLE
PREDICTION
USINGcumulated
LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
precipitation
exceeding P threshold at +60h
E-mean distribution in the first 2 EOF space (approx 50% e.v.)
45 cases during 1999
D+3
analysis
E-mean
average
D+5
E-mean
average
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002
Precip outliers
91 EPS DJF 1999/2000
TP ACC 24 H
EPS
5 RM
(c.linkage)
BELOW ABOVE OUTLIERS
MIN
MAX
%
D+3
8.3
5.6
13.9
D+5
4.6
2.8
7.3
both
4.1
1.7
5.8
D+3
D+5
both
13.6
9.7
9.4
24.9
17.2
16.8
38.6
26.9
26.1
WORKSHOP ON SHORT-RANGE ENSEMBLE PREDICTION USING LIMITED-AREA MODELS
Instituto National de Meteorologia, Madrid, 3-4 October 2002