Transcript ppt - Cosmo

Medium-range flood forecasting and warning
Basin-oriented verification of
COSMO-LEPS system
Andrea Montani
ARPA-SIM Hydrometeorological service of Emilia-Romagna, Italy
Thanks to H. Asensio, R. Buizza, F. Pappenberger, B. Ritter, J.W. Schipper
X General COSMO meeting
Cracow, 15-19 September 2008
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Workpackage Medium-Range Plain Flood
of PREVIEW project
Aims:
1.
Set-up and validation of the probabilistic medium-range flood
forecasting using the meteorological products for the UpperDanube in the hydrological year 2002, which includes the large
flood of August 2002; to achieve this,
•
•
2.
reruns of a number of state-of-the-art of atmospheric models were
performed,
“convenient” (basin-oriented) scoring techniques for probabilistic
forecasts were developed.
In terms of operational applications, the goal is to demonstrate
the usefulness of probabilistic medium-range flood forecasting as
a sound basis for early warning and decision making.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
The region of interest
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Verification methodology
MAIN FEATURES:
•
Verification performed in terms of 24-hour
precipitation (from 6UTC to 6UTC);
•
fcst ranges: 18-42h, 42-66h, 66-90h, 90-114h, …;
•
observations: gridded observations (about 5
km of horizontal resolution) provided by JRC;
•
verification domain: full upper-Danube river
basin and 3 (out of 19) sub-basins;
•
verification period: 20 July - 31 August 2002.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Sub-basin verification
ID
name of Basin
area (km2)
no. of gridded obs
1
Bratislava
131.978
5278
13
Wiblingen
2.247
80
16
Passau-Ingling
25.977
1045
20
Hofkirchen
47.534
1896
For each gridded observation, it
was identified the sub-basin it
belonged to.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Deterministic measures for verification
• Catchment-based measures
• centre of gravity: it is a specific point at which the system's mass behaves as if it were concentrated.
The centre of mass/gravity is a function only of the positions and masses of the particles of the system.
CoG 
cf:
dist c f  cobs 
CoG: Centre of Gravity measure
distmean
forecasted Centre of gravity
cobs: observed Centre of Gravity
dist: distance
distmean: mean distance of grid elements to catchment outlet
Npo int s
pr
cf=
i 1
Npo int s
p
i 1
ri
i i
i
are the geographical coordinates at grid point i
pi : forecast precipitation at grid point i
CoG (dimensionless) ranges from 0 to infinity … the lower the better (in practice the score is limited by the value 2,
indicating a location error twice the mean catchment distance of the grid elements to the outlet).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Ensemble systems in PREVIEW
–
–
–
– VarEPS by ECMWF (global, ensemble)
– COSMO-LEPS by ARPA-SIM (limited-area, ensemble)
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
CentreofGravity distance



Aim: quantify the "location error" of the precipitation forecast (by the ensemble mean);
the CoG is scaled with the "mean distance" in a catchment  a value of cog=0.3 for the catchment
"Passau-Ingling" (mean distance =113 km) indicates a location error of the forecasted centre of gravity of
precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km.
with the scaling, we get a dimensionless number (…the lower, the better…) to compare the absolute
location error with respect to the catchment size.
 The absolute figures are very low
(distances of the order of a few tens
of km).
 Little dependence of the distance
from the forecast range.
 Slightly higher distances for the
Passau-Ingling
basin
(possibly
related to observation problems in
the Alpine region).
 Best results for the smallest basin.
CoG distance
CoG
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Brier Score for VarEPS and COSMO-LEPS
 BS measures the mean squared difference between forecast and observation in probability space.
 equivalent to MSE for deterministic forecast.
 BS between 0 and 1; the lower the better ….
 the largest (Bratislava) and the smallest (Wiblingen) basins are considered (tp > 80% of obs distribution).
BS
 ECMWF
: solid.
 COSMO-LEPS: dotted.
 Low values for both models: GOOD!
 COSMO-LEPS performs better over
the smaller basins (not well resolved
by VarEPS).
 Similar performance of the two
systems over the largest basin.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
For the other results, come and see the
poster!
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Thank you !
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
The 32-day unified VarEPS at ECMWF
Unification of the 15d VarEPS (50+1 members) and the 32d monthly
forecast (MOFC) systems into the unified 32d VarEPS:
old system
Twice-a-day (at 00 and 12 UTC):
• d 0-10: TL399L62 uncoupled
• d 10-15: TL255L62 uncoupled
Once a week:
– d 0-32: TL159L62 coupled
NEW SYSTEM (since 11/3/2008)
15d VAREPS
T0
+240
+360
32d MOFC
+768
T0
15d and 32d VAREPS
T0
+240
Twice-a-day (at 00 and 12 UTC):
• d 0-10: TL399L62 uncoupled
• d 10-15: TL255L62 coupled at 00
Once a week:
• d 0-10: TL399L62 uncoupled
A.Montani;
Basin-oriented verification of COSMO-LEPS
• d 10-32: TL255L62
coupled
X COSMO meeting – Cracow – 15-19 September 2008
+360
+768
Skill of ECMWF predictions for hydrological modelling
 Nash_Sutcliffe efficiency coefficient: normally used to assess the predictive power of hydrological models.
 … the higher, the better (ranges from –  to 1)
 Timing of forecasts is
good,
although
underestimation occurs.
 The ensemble spread
brackets
observations
only partially.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
17th of July 2007
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
7 Panel version
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Atmospheric models in PREVIEW
–
–
–
–
– COSMO-LEPS by ARPA-SIM (limited-area, ensemble)
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
COSMO-LEPS (developed at ARPA-SIM)
integration domain
MAIN FEATURES:
•
•
•
•
initial time: 12 UTC (i.e. once a day);
bc and ic: “selected” VarEPS members;
COSMO-LEPS configuration
– 10 members;
– hor. res. = 10 km; 32 vertical levels;
– forecast length: 132h;
– archived variables: surf and plev up to
+132h, every 3h;
– output fields archived at ECMWF;
rerun period: 20 July – 31 August 2002.
COSMO-LEPS: the Limited-area Ensemble Prediction System (LEPS), based on
COSMO-model and implemented within COSMO (COnsortium for Small-scale
MOdelling, including Germany, Greece, Italy, Poland, Romania, Switzerland).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Brier Skill Score vs forecast range
 BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems if BSS > 0.
 BS measures the mean squared difference between forecast and observation in probability space.
 Equivalent to MSE for deterministic forecast.
BSS
 For low thresholds, better
performance over the smallest
basin.
 For higher thresholds, more
difficult to assess a clear trend
(possible sampling problems
over small basins).
 Worse than climatology only
for d+5 range (10 and 20 mm
threshold).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Brier Skill Score vs increasing threshold
 BSS is plotted vs increasing threshold to assess the COSMO-LEPS skill as a function of rainfall intensity.
BSS +42h
BSS +90h
 BSS positive for all thresholds at both forecast ranges.
 At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin.
 Stable skill of the system for intermediate basins.
 For higher thresholds, more difficult to assess a clear trend (sampling problems over the small basin).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Ranked Probability Skill Score
 A sort of BSS, but “cumulated” over all thresholds.
 Useful forecast system, if RPSS > 0.
RPSS
 RPSS always positive.
 Better (worse) performance of
the system over the smallest
(largest) basin.
 Almost identical scores using
either nearest grid-point (NGP)
or bilinear interpolation (BILIN).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
ROC area vs increasing threshold
 ROC area is plotted vs increasing threshold to assess the dependence of COSMO-LEPS HIT/FAR diagram on
rainfall intensity.
ROC +42h
ROC +90h
 ROC area always above 0.7 for all thresholds at both forecast ranges.
 At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin (the same as BSS).
 Scores increase with thresholds for intermediate and large basins.
 For higher thresholds, more difficult to assess a clear trend (sampling problems over the small basin).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
ROC area vs forecast range
 Area under the curve in the HIT rate vs FAR diagram.
 Valuable forecast systems have ROC area values > 0.6.
ROC
 ROC area always above 0.6;
 at low thresholds, better COSMOLEPS performance over the smallest
basin;
 possible sampling problems at 20
mm thresholds.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Outliers
 How many times the analysis is outside the forecast interval spanned by the ensemble members.
 … the lower the better …
OUTL
 The absolute figures are quite large
(about 20% at d+5 range).
 Lower outliers for the smallest basin.
 Similar percentages for all basins at
the longest forecast range.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Main results and future plans
•
Output files are of COSMO-LEPS, for the period 20/7/ to 31/8/2002, are archived
on ECFS (retrieval script disseminated) and ready to be used for hydrological
purposes; for any kind of help with grib files, just ask.
•
Verification against gridded observations (about 5300 in the full upper-Danube
basin) indicates better performance of the system over the smallest basin,
especially for low thresholds.
•
OLD RESULT (Offenbach, 25-26-/9/2007): Verification against SYNOP stations
(about 100 in the full upper-Danube basin) indicates slightly better performance of
the system over larger basins, although results are not statistically robust.
•
Difficulty to understand and use the CoG measure for a probabilistic system.
•
Finish the work on the verification report.
•
Provide COSMO-LEPS fields for 2-month real-time testing (no “bureaucratic”
problems envisaged).
•
Real-time testing of services will be done using the “improved” COSMO–LEPS (16
members; 40 vertical levels; physics perturbations).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Main results
•
During the hydrological year 2002, reruns of a number of NWP systems, both
global-scale and limited-area, both deterministic and probabilistic.
•
Development of new
– basin-oriented scores (e.g. centre of gravity);
– hydrological-oriented products (e.g. rivergrams);
•
All systems seem to provide useful guidance for the possible occurrence of flood
events also for forecast ranges up to 7 (5) days for global (limited-area) systems.
•
Verification vs gridded observations (about 5300 in the full upper-Danube basin)
indicates better performance of the high-res system over the smaller basin.
•
The verification period is probably too short to draw general (and statistically
solid) conclusions about the overall skill of the different forecast systems; the
“Plain-flood campaign” clearly shows the potential of state-of-the-art NWP
systems in the field of weather forecasting for river flooding.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification
• Brier Score
The BS is the mean-squared error of the probability forecasts:
1
BS 
N
N
 f
k 1
 ok 
2
k
where:
• the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur);
• fk is the fraction of ensemble members which forecast a precipitation amount exceeding that threshold
• k denotes a numbering of the N forecast/event pairs.
BS ranges from 0 to 1, the perfect forecast having BS = 0.
BS is computed for a fixed precipitation threshold.
• ROC
area
It is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram
The integral under the curve is used to indicate the skill of the forecast.
ROC area ranges from 0 to 1 … the higher the better …
Useful forecast systems have ROC area values greater than 0.6
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Summary of measures
•
Deterministic measures:
• Root Mean Squared Error
• Mean Error
• Probability of detection
• Probability of false detection
• True Skill Statistics
• Centre of gravity
• Coverage
• Probabilistic measures:
• Brier score
• ROC area
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Performance measures
1)
continuous measures (RMSE and MAE for the ensemble mean);
2)
catchment-based measures (CentreofGravity for the ensemble mean);
3)
probabilistic measures (Brier Skill Score, ROC area, Percentage of Outliers, …).
Centre of Gravity:
the centre of gravity of a system of raster cells is a specific point at which, for many
purposes, the system's mass behaves as if it were concentrated. The centre of mass is a function
only of the positions and masses of the particles that comprise the system.
CoG: Centre of Gravity measure
c:
Centre of gravity
dist:
Distance
f:
forecast
distmean: mean distance of grid elements to catchment outlet
obs:
observed
p: precipitation
Npo int s
c=
pr
i 1
Npo int s
i i
this score indicates a location error of the forecast, a perfect score having the value of 0.
p
i 1
i
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Catchment Information System
River-grams
Sylvie Lamy-Thepaut, Enrico Fucile
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Main results
•
During the hydrological year 2002 (1/10/2001 to 30/9/2002), reruns of a number
of NWP systems, both global-scale and limited-area, both deterministic and
probabilistic.
•
Coordinated efforts to provide state-of-the-art weather forecasts over the
Danube river basin.
•
Development of basin-oriented scores
•
Development of hydrological-oriented products (e.g. rivergrams)
•
All systems seem to provide useful guidance for the occurrence of flood events.
Time-series verification scores indicate the following trends:
 increase in ROC area scores and reduction in outliers percentages;
 positive impact of increasing the population from 5 to 10 members (June 2004);
 some deficiency in the skill of the system can be identified after the system upgrades
occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT
A.Montani;throughout
Basin-oriented verification
of COSMO-LEPS
 scores are encouraging
DPHASE
Operations Period.
X COSMO meeting – Cracow – 15-19 September 2008
What is left?
•
“Close the gap” between hydrological and meteorological communities.
•
Learn from MAP D-PHASE experience
•
Exploit the wealth of information provided by probabilistic forecasts:
– assess performance over different domains (North and South of the Alps),
– study individual case studies,
– consider basin-by-basin performance.
•
“Think about” increasing horizontal resolution to 7 km.
•
Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel , Meteoswiss).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Sub-basin verification
ID
name of Basin
area (km2)
no. of “stations”
approx no of grid points
1
Bratislava
131.978
5278
1320
13
Wiblingen
2.247
80
20
16
Passau-Ingling
25.977
1045
260
20
Hofkirchen
47.534
1896
470
For each gridded observation,
it was identified the sub-basin
it belonged to.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Ranked Probability Skill Score
 A sort of BSS, but “cumulated” over all thresholds.
 Useful forecast system, if RPSS > 0.
RPSS
 RPSS always positive.
 Better (worse) performance of
the system over the smallest
(largest) basin.
 Almost identical scores using
either nearest grid-point (NGP)
or bilinear interpolation (BILIN).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
CentreOfGravity distance


Scaled (that is, dimensionless) distance between predicted (by COSMO-LEPS ensemble mean) and
observed centre of gravity for each catchment.
the "cog" is an attempt to quantify the "location error" of the precipitation forecast; we have scaled the
"cog" with the "mean distance" in a catchment - so a value of cog=0.3 for the catchment "Passau-Ingling"
with a mean distance distmean=113 km indicates a location error of the forecasted centre of gravity of
precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km. With the
scaling we get a (dimensonless) number to compare the absolute location error with respect to the
catchment size.
… the lower the better …

 The absolute figures are very low
(distances of the order of a few tens
of km).
 Little dependence of the distance
from the forecast range.
 Slightly higher distances for the
Passau-Ingling basin.
 Best results for the Wiblingen basin
(the smallest one)
 Similar distances for the other
basins.
CoG distance
CoG
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Catchment Information System River-grams
• Proto type, currently implemented operational.
• Database of catchments will be extended to more
catchments in Europe and to include all major World
catchments.
• Variables are currently static (always the same for all
catchments), but will be ‘dynamic’ – reflecting the
usage of catchments.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Outliers
 How many times the analysis is outside the forecast interval spanned by the ensemble members.
 … the lower the better …
OUTL
 The absolute figures are quite large
(about 20% at d+5 range).
 Lower outliers for the smallest basin.
 Similar percentages for all basins at
the longest forecast range.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Catchment-based verification of GME
(2)
period 2002-01-01 – 2002-09-30
Verification of GME forecasts against rain gauge data from high density observation
network for the four sub-catchments of the Danube with different catchment sizes:
• increasing centre of gravity score („location error“) with increasing forecast time;
• a positive mean coverage error for this period.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Catchment-based verification of DWD models
(2)
period 2002-07-19 – 2002-08-20
Verification of GME and COSMO-EU forecasts against rain gauge observations for the
catchment Hofkirchen (upper Danube):
• increasing centre of gravity score („location error“) with increasing forecast time
• mostly a negative mean coverage error for this period
the verification period might be too short for scores to be statistically significant.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
COSMO-1km – cont. measures
A.Montani;
A.Montani;
Basin-oriented
Noveltiesverification
in weather of
forecasting
COSMO-LEPS
PREVIEW
X COSMO
trainingmeeting
workshop
– Cracow
- Mosonmagyaróvár
– 15-19 September
- 19-202008
June 2008
COSMO-LEPS methodology
Possible
evolution
scenarios
ensemble size
reduction
Cluster members chosen
as representative
members (RMs)
Initial conditions
Dim 1
LAM scenario
LAM scenario
LAM integrations driven by
RMs
LAM scenario
Initial conditions
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Dim 1
COSMO-LEPS (developed at ARPA-SIM)
• What is it?
It is a Limited-area Ensemble Prediction System (LEPS),
based on COSMO-model and implemented within COSMO
(COnsortium for Small-scale MOdelling, which includes
Germany, Greece, Italy, Poland, Romania, Switzerland).
• Why?
It was developed to combine the advantages of global-model
ensembles with the high-resolution details gained by the
LAMs, so as to identify the possible occurrence of severe
and localised weather events (heavy rainfall, strong winds,
temperature anomalies, snowfall, …)
generation of COSMO-LEPS to improve the Late-Short
(48hr) to Early-Medium (132hr) range forecast of
severe weather events.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
ROC area vs forecast range
 Area under the curve in the HIT rate vs FAR diagram.
 Valuable forecast systems have ROC area values > 0.6.
ROC
 ROC area always above 0.6.
 Similar results to those obtained in
terms of BSS: at low thresholds,
better COSMO-LEPS performance
over the smallest basin.
 Possible sampling problems at 20
mm thresholds.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification (1)
The probabilistic measures we used, are the Brier Score and the ROC area. The thresholds
for their computation are based on both absolute values (>1mm) and on the percentiles of the
observed cumulative precipitation distribution (> 80% of observed).
• Brier Score
The BS is the mean-squared error of the probability forecasts; it averages the squared difference between pairs
of forecast probabilities and the correspondent binary observations:
1
BS 
N
N
 f
k 1
 ok 
2
k
where:
•the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur);
•k denotes a numbering of the N forecast/event pairs.
BS ranges from 0 to 1, the perfect forecast having BS = 0.
BS is computed for a fixed precipitation threshold and fk is the fraction of the ensemble members which forecast
a precipitation amount exceeding that threshold.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification (2)
• ROC
area
It is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram
Hit rate (HR) and false alarm rate (FAR) are computed for each probability class k:
HRk 
ak
N o
 k k
a k  ck
No
FARk 
bk
N (1  ok )
 k
bk  ck
N (1  o )
where:
• the verification sample is subdivided into subsamples of size Nk, according to the probability with which the event is forecasted,
• ok is the frequency with which the event is observed, being forecasted with a given probability and is the sample climatology.
The cumulative HRk are plotted against the corresponding cumulative FARk, generating the ROC curve.
The integral under the curve is used to indicate the skill of the forecast.
ROC area ranges from 0 to 1, the higher the better.
Useful forecast systems have ROC area values greater than 0.6
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
COSMO-1km – Skill Scores (IMK)
A.Montani;
A.Montani;
Basin-oriented
Noveltiesverification
in weather of
forecasting
COSMO-LEPS
PREVIEW
X COSMO
trainingmeeting
workshop
– Cracow
- Mosonmagyaróvár
– 15-19 September
- 19-202008
June 2008
17th of July 2007
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Outline
• Introduction
• Measure for model performance (precipitation only)
• Atmospheric models re-run for PREVIEW:
–
–
–
–
–
GME by DWD (global, deterministic),
COSMO-EU by DWD (limited-area, deterministic),
VarEPS by ECMWF (global, ensemble),
COSMO-LEPS by ARPA-SIM (limited-area, ensemble),
COSMO-1km by IMK (limited-area, deterministic)
(tomorrow’s talk by J.W. Schipper)
• Application to the Danube sub-basins
• Lesson learnt
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Deterministic measures for verification (1)
•
Continuous measures
The Mean Error (ME, bias) of the precipitation forecast against the measurement (F: Forecast; O:
Observations; N: sample size):
1
ME 
N
 (F  O )
i 1
i
ME ranges from –infinity to infinity; the closer to zero, the better …
i
Categorical measures
Consider thresholds (e.g. tp >1mm and tp > 80% of observed mean).
Scores are generated with the help of a contingency table:
observed
Forecast
•
N
observation >
threshold
observation 
threshold
forecast > threshold
hits
false alarms
forecast  threshold
misses
correct negatives
Probability of detection POD 
True Skill Statistic
hits
false alarms
; Probability of false detection POFD 
hits  misses
correct negatives  false alarms
TSS  POD  POFD
(useful forecast for 0 < TSS < 1 … the higher the better …)
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Atmospheric models in PREVIEW
– GME by DWD (global, deterministic)
– COSMO-EU by DWD (limited-area, deterministic)
–
–
–
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
DWD operational NWP models
COSMO-DE:
grid spacing: 2.8 km
421*461 * 50 grid elements
time step: 25 sec
forecasts up to 21 hours
COSMO-EU:
grid spacing: 7 km
665*657 * 40 grid elements
time step : 40 sec.
forecasts up to 78 hours
GME:
grid spacing: 40 km
368642 * 40 grid elements
time step : 133 sec.
forecasts up to 7 days
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
DWD meteorological forecasts
• forecasts with the global model GME for the
hydrological year 2002 (2001-10-01  2002-09-30).
• forecasts with the high-resolution regional model
COSMO-EU for the PREVIEW special period
(2002-07-19  2002-08-20).
Example: DWD forecast of 24h accumulated
precipitation with GME (global) and COSMO-EU
(regional) for 2002-08-12 06 UTC to 2002-08-13 06
UTC, the forecast start time is 2002-08-11 12 UTC.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
(1)
DWD meterorological forecasts
(2)
24h catchment mean precipitation for the Hofkirchen catchment (upper Danube) forecasts with GME and
COSMO-EU and corresponding adjusted radar data and high-density network rain gauge observations in the
period 2002-08-01 to 2002-08-20.
There are differences between the two observing systems  the overall evolution for the flooding event is
well captured by both forecast models (COSMO-EU slightly better).
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Catchment-based verification of GME
period 2002-01-01 – 2002-09-30
GME forecasts vs rain-gauge data from high-density network for the four subcatchments of the Danube with different catchment sizes:
• positive bias for all four catchments in this period (the model “rains” too much);
• positive TSS (true skill statistics), i.e. the forecasts are skilful.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
Catchment-based verification of DWD models
period 2002-07-19 – 2002-08-20
GME and COSMO-EU forecasts vs rain-gauge observations for the catchment
Hofkirchen:
• negative bias for both models (the models “rain” too little);
• GME slightly better for low thresholds
• positive TSS (true skill statistics), i.e. the forecasts are skilful.
the verification period might be too short for scores to be statistically significant.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008
What is left?
• “Close the gap” between hydrological and meteorological communities.
• Consider the outcome of MAP D-PHASE experimentation (model
performance may vary considerably from basin to basin).
• Develop and exploit “basin-oriented” calibration of NWP systems (being
implemented in these months for ECMWF varEPS and COSMO-LEPS).
• Exploit the wealth of information provided by probabilistic forecasts:
– get more and more acquainted with the concept of probability;
– consider the “gain” (€, $, Ft,…)  presentation by D. Richardson.
A.Montani; Basin-oriented verification of COSMO-LEPS
X COSMO meeting – Cracow – 15-19 September 2008