Transcript ppt - Cosmo

COSMO LEPS system:
status, verification, applications
Andrea Montani,
C. Marsigli, T. Paccagnella
ARPA-SIM, Bologna, Italy
F. Fundel, A. Walser, M. A. Liniger, C. Appenzeller
Meteoswiss
X General COSMO meeting
Cracow, 15-19 September 2008
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Outline
• Introduction
• Present status and recent upgrades
• Verification results (SYNOP on the GTS):
– performance from 2002 onwards
• Application of COSMO-LEPS: the reforecast
exercise
• Main results
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS (developed at ARPA-SIM)
• What is it?
It is a Limited-area Ensemble Prediction System (LEPS),
based on COSMO-model and implemented within COSMO
(COnsortium for Small-scale MOdelling, which includes
Germany, Greece, Italy, Poland, Romania, Switzerland).
• Why?
It was developed to combine the advantages of global-model
ensembles with the high-resolution details gained by the
LAMs, so as to identify the possible occurrence of severe
and localised weather events (heavy rainfall, strong winds,
temperature anomalies, snowfall, …)
generation of COSMO-LEPS to improve the Late-Short
(48hr) to Early-Medium (132hr) range forecast of
severe weather events.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS suite @ ECMWF
before and during D-PHASE Operations Period (from 1 June to 30 November 2007)
d
d-1
d+1
d+2
older EPS
00
d+3
d+4
d+5
4 variables
ZUVQ
3 levels
500 700 850 hPa
Cluster Analysis and RM identification
younger EPS
12
2
time
steps
European
area
Complete
Linkage
•
COSMOLEPS
clustering
area
COSMOLEPS
Integratio
n Domain
•
•
•
•
suite runs as a ”time-critical
application”
managed
by
ARPA-SIM;
Δx ~ 10 km; 40 ML;
COSM0 v3.20;
fc length: 132h;
Computer time (4.7 million BU
for 2007) provided by the
COSMO partners which are
ECMWF member states.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Recent updates
Changes to the operational suite on 1 December 2007:
•
•
•
•
new model version (COSMO-LM 4.0);
new numerics (Runge-Kutta scheme);
use of multi-layer soil model;
new perturbations introduced relative to the maximal turbulent length scale (tur_len) and to the
length scale of thermal surface patterns (pat_len)  to increase spread and maintain skill.
 Old system and new system ran
in parallel for 52 days (October
and November 2007);
 Larger T2m spread in new
system for all forecast ranges;
 Comparable skill (in terms of
T2m root-mean-square error) of
the two systems.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS suite @ ECMWF
after D-PHASE Operations Period (since December 2007)
d
d-1
d+1
d+2
older EPS
00
d+3
d+4
d+5
4 variables
ZUVQ
3 levels
500 700 850 hPa
Cluster Analysis and RM identification
younger EPS
12
2
time
steps
European
area
Complete
Linkage
•
COSMOLEPS
clustering
area
COSMOLEPS
Integratio
n Domain
•
•
•
suite runs as a ”time-critical
application” managed by ARPASIM;
Δx ~ 10 km; 40 ML; fc+ 132h;
COSM0 v4.0 since Dec07 (with
RK + multi-layer);
computer time (5.8 million BU
for 2008) provided by the
COSMO partners which are
ECMWF member states.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Objective verification of COSMO-LEPS
–
SYNOP on the GTS
Main features:
variable:
12h cumulated precip (18-06, 06-18 UTC);
period :
from Dec 2002 to Aug 2008;
region:
43-50N, 2-18E (MAP D-PHASE area);
method:
nearest grid point; no-weighted fcst;
obs:
synop reports (about 470 stations/day);
fcst ranges: 6-18h, 18-30h, …, 102-114h, 114-126h;
thresholds: 1, 5, 10, 15, 25, 50 mm/12h;
system:
COSMO-LEPS;
scores:
ROC area, RPSS, Outliers
both (69) monthly and (23) seasonal scores were computed
verification over the full domain (about 1400 stations/day) is underway
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area
 Area under the curve in the HIT rate vs FAR diagram.
 Valuable forecast systems have ROC area values > 0.6.
ROC
 The positive impact
of increasing the
ensemble size in
2004 is evident for
all thresholds and for
different
forecast
ranges.
 Poor performance of
the system in Spring
and Summer 2006
(both
particularly
dry), despite system
upgrades.
 Good performance
during DOP 2007.
 Small positive impact
of
2007
system
upgrades.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Seasonal scores of ROC area
 Performance of the system assessed for the last 5 summers (JJA) and the last 5 autumns (SON).
 Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6.
ROC
 Need to take into
account
the
different statistics
for each season
(JJA 2004 and SON
2006 less rainy
than the others).
 Good
scores
obtained
throughout the full
DOP in 2007.
 Best performance
for JJA 2008, more
evident for longer
ranges.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Outliers: time series + seasonal scores
 How many times the analysis is out of the forecast interval spanned by the ensemble members.
 … the lower the better …
 Performance of the system assessed as time series and for the last 5 summers (JJA).
OUTL
 Evident seasonal cycle
(more
outliers
in
winter), but overall
reduction of outliers in
the years up to 2007.
 Reduction of outliers
from one summer to
the other, related to the
increase of ensemble
size (evident for the
10m to 16m increase).
 Need to take into
account the different
statistics
for
each
season.
 Best results for the last
2 summers.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS reforecasts (v 4.0)
Motivations:
•
•
COSMO-LEPS is often not reliable;
probabilities might be wrong!  need for
calibration
Setup
•
•
•
•
•
•
•
Period of 30 years (1971-2000)
1 deterministic run of COSMO-LEPS
fc+90h, 12:00 initial time (every 3rd day)
ERA40 as initial/boundary conditions.
Calculated and archived at ECMWF
Convect. scheme = Tiedtke/Kain-Fritsch
Random physics (turlen & patlen)
Limitations
•
•
New climatology needed with each model
version change
Needs time and is costly (~ 2000 KBU/yr)
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Verification of calibrated vs raw forecasts
Observation Data
model topography [m]
• 24h Tot_Prec (0600-0600 UTC)
• Domain: Switzerland
• interp. on CLEPS Grid (C.Frei)
• Apr06 – Aug07 (verification)
• 1971-2000 (calibration)
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
BSSD 24h precipitation
 BSSD is written as 1-BS/(BSref +D). Useful forecast systems if BSSD > 0.
 BS measures the mean squared difference between fcst and obs in probability space.
 Equivalent to MSE for deterministic forecast.
winter
summer
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
calibrated
raw
Main results
(1)
•
Since November 2002, COSMO-LEPS system has been running on a daily basis.
•
Since June 2008, COSMO-LEPS data are disseminated to JRC for the use in the
European Flood Alert System (EFAS) as input to the flood forecasting system over
the major European basins ==> success for COSMO both from a scientific and a
visibility point of view.
•
COSMO-LEPS products routinely used in met-ops rooms across COSMO community
as well as in research Projects (e.g. COPS, D-PHASE, PREVIEW: see the poster).
Time-series verification scores indicate the following trends:
 increase in ROC area scores and reduction in outliers percentages;
 positive impact of increasing the population from 5 to 10 members (June 2004);
 some deficiency in the skill of the system can be identified after the system upgrades
occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT
 scores are encouraging throughout DPHASE Operations Period;
 system upgrades of Dec07 (RK + multi-layer + new perturbations) brought
small but positive impact.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Main results
(2)
•
Calibrating with reforecasts improves the forecast skill significantly (especially in
winter).
•
Calibrated warnings without using observations are possible.
•
Calibrating frequent precipitation events does not require a large calibration period,
while calibrating extreme events does (not shown).
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Thank you !
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Feasibility study: COSMO-LEPS at 7 km
(the answer to forecasters’ dream?)
Present system
x = 10 km
z = 40 ML
t = 90 s
ngp = 306x258x40 = 3.157.920
fcst range = 132h
cost = 640 BU x run
elapsed time = 45 min
New system
x = 7 km
z = 40 ML
t = 66-72 s
ngp = 510x405x40 = 8.262.000
fcst range = 132h
cost = 1925 BU x run
elapsed time = 138 min
… cosmoleps_7 is about 3 times more expensive than present configuration
new computer at ECMWF being installed
Computer resources for each ECMWF member state will increase by a
factor of 5 (five)  and ….
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
… the dream is possible
COSMO-LEPS 10 km
COSMO-LEPS 7 km
 the grid of cosmoleps_7
would be almost identical to
that of COSMO-EU, this
making easier and cleaner the
use of initial fields provided
by DWD (e.g. soil moisture
analysis).
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Future plans (2008 and 2009)
•
test the use of the Soil Moisture Analysis fields provided by DWD;
•
run cleps_7 for ~ 40 days in autumn 2008 and assess the impact;
•
within TIGGE-LAM, develop coding of COSMO-LEPS output files in GRIB2 format;
•
migration to the new machine at ECMWF;
•
use a better snow analysis (possibly provided by DWD or Meteoswiss);
•
extend the cluster analysis so as to consider not only ECMWF EPS, but also
UKMO MOGREPS as global ensemble providing ic’s and bc’s (first tests);
•
implement cosmoleps_7;
•
gaining from COSMO-SREPS experience, introduce more model perturbations;
•
test COSMO-LEPS nested on the under-development ECMWF EDA over MAP
D-PHASE period;
•
optimise use of reforecasts + calibration of wind gust;
•
support CONSENS.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Feasibility study of COSMOLEPS at 7 km
Motivations:
 Provide a more detailed description of mesoscale
processes by incresing the horizontal resolution.
 Do not lose a “reasonable advantage” against ECMWF
EPS, which will go to x=25 km during 2009.
 “Keep the pace” with deterministic model (x~ 2-3
km): if the gap in resolutions between deterministic
and probabilistic systems is too large, the two
systems do not support each other and go for
different solutions (that is, they forecast different
weather!).
from 10 to 7 km (plus small domain extensions) does not seem a lot
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
but …
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Present methodology
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Operational set-up
Core products:
 16 perturbed COSMO-model runs (ICs and 3-hourly
BCs from 16 EPS members) to generate, “via weights”,
probabilistic output (start at 12UTC; t = 132h);
Additional products:
 1 deterministic run (ICs and 3-hourly BCs from the highresolution deterministic ECMWF forecast) to “join”
deterministic and probabilistic approaches (start at
12UTC; t = 132h);
 1 “hindcast” run (ICs and 3-hourly BCs from ECMWF
analyses) to “downscale” ECMWF information (start at
00UTC; t = 36h).
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Recent updates
Changes to the operational suite on 1 December 2007:
•
•
•
•
new model version (COSMO-LM 4.0);
new numerics (Runge-Kutta scheme);
use of multi-layer soil model;
new perturbations introduced relative to the maximal turbulent length scale (tur_len) and to the
length scale of thermal surface patterns (pat_len)  to increase spread and maintain skill.
 Old system and new system ran
in parallel for 52 days (October
and November 2007);
 Larger T2m spread in new
system for all forecast ranges;
 Comparable skill (in terms of
T2m root-mean-square error) of
the two systems.
back to plans
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area for COSMO-LEPS
(2)
 Area under the curve in the FAR vs HIT diagram.
 Valuable forecast systems have ROC area values > 0.6
ROC
fc step: 78-90h
 the positive impact of
increasing the ensemble
size in 2004 is evident
for all thresholds and for
different fcst ranges.
 poor performance of the
system in Spring and
Summer 2006, despite
system upgrades.
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Time series of Brier Skill Score
 BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems if BSS > 0.
 BS measures the mean squared difference between forecast and observation in probability space.
 Equivalent to MSE for deterministic forecast.
BSS
fc step: 30-42h
 improvement of
performance detectable
for all thresholds along
the years;
 still problems with high
thresholds, but good
trend in 2007.
Jun04: 5m  10m
Feb06: 10m16m; 32ML  40 ML
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Semi-diurnal cycle in COSMO-LEPS scores
 BSS score … the higher the better …
 Performance of the system assessed for 5 different Summers (JJA).
BSS
 Evident 12-hour cycle in BSS
scores (the same holds for
RPSS, while less evident for
ROC area scores).
 Better performance of the
system
for
“night-time”
precipitation, that is for rainfall
predicted between 18Z and 6Z
(ranges 30-42h, 54-66h, …).
 The amplitude of the cycle is
somewhat reduced throughout
the years and with increasing
forecast range.
 The bad performance in
Summer 2006 is confirmed.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Score dependence on the domain size (1)
 Verification of COSMO-LEPS against synop reports over the MAP D-PHASE area (~ 470 stations; MAPDOM)
and the full domain (~ 1500 stations; fulldom):
 different statistics of the verification samples;
 up to now, performance of the system over the 2 domains assessed only for the last 6 months
(March-August 2007).
 difficult to draw general conclusions
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Score dependence on the domain size (2)
 RPSS score… the higher the better… (and positive).
 ROC area… the higher the better… (and above 0.6).
 Smoother transitions from
month to month in “fulldom”
scores.
 Slightly better performance of
COSMO-LEPS
over
the
MAPDOM, but the signal
varies from month to month.
 Higher predictability with
orographic forcing?
 Need to check individual
regions and/or to stratify
for type of stations.
 Outliers percentage … the lower the better.
RPSS
OUTL
ROC
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
MAM06
Average values (boxes 0.5 x 0.5)
tp > 1mm/24h
tp > 5mm/24h
COSMO-LEPS
16-MEMBER EPS
51-MEMBER EPS
As regards AVERAGE precipitation above
these two threshols, the 3 systems have
similar performance.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Main results
•
COSMO-LEPS system runs on a daily basis since November 2002 and it has
become a “member-state time-critical application” at ECMWF ( partial
involvement of ECMWF operators in the suite management).
•
Time series scores identify positive trends in the performance of the system:
•
•
•
increase in ROC area scores and reduction in outliers percentages;
positive impact of increasing the population from 5 to 10 members (June 2004);
some deficiency in the skill of the system identified after the system upgrades occurred
on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS
upgrade!!!)  need of further investigation.
•
High-resolution verification shows better scores of COSMO-LEPS with respect to
EPS in forecasting precipitation maxima within boxes.
•
No clear impact of the weighting procedure as regards precipitation, BUT
COSMO-LEPS ensemble mean for 2m temperature has better skill (lower standard
deviation to observations) with weighting the ensemble members according to
cluster populations compared to the mean with unweighted members.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area
 Area under the curve in the HIT rate vs FAR diagram.
 Valuable forecast systems have ROC area values > 0.6.
ROC
 the positive impact
of increasing the
ensemble size in
2004 is evident for
all thresholds and
for different forecast
ranges.
 poor performance of
the system in Spring
and Summer 2006
(both
particularly
dry),
despite
system upgrades.
 good performance
during DOP 2007.
fc step: 30-42h
DOP
fc step: 78-90h
Jun04: 5m  10m
Jun04: 5m  10m
DOP
Feb06: 10m  16m; 32ML  40 ML
Feb06: 10m  16m; 32ML  40 ML
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Seasonal scores of ROC area
 Performance of the system assessed for 5 different summers (JJA) and 5 different autumns (SON).
 Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6.
ROC
 Need to take into
account
the
different statistics
for each season
(JJA 2003 and
SON 2006 less
rainy than the
others).
 Best
scores
obtained during
DOP 2007 for
either seasons.
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Time series of Ranked Probability Skill Score
 A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference
system. BS measures the mean squared difference between forecast and observation in probability space.
 Useful forecast system, if RPSS > 0.
RPSS
DOP
 the improvement of the
system performance is
detectable
for
all
forecast ranges along the
years;
 the poor performance of
the system in summer
2006 is confirmed;
 RPSS
always
positive
throughout 2007 and
especially high during
DOP.
Jun04: 5m  10m
Feb06: 10m16m; 32ML  40 ML
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Outliers: time series + seasonal scores
 How many times the analysis is out of the forecast interval spanned by the ensemble members.
 … the lower the better …
 Performance of the system assessed as time series and for 5 different summers (JJA).
OUTL
 Evident seasonal cycle
(more
outliers
in
winter), but overall
reduction of outliers in
the years.
 Reduction of outliers
from one summer to the
other, related to the
increase of ensemble
size (more evident for
the 5 to 10 increase).
 Need to take into
account the different
statistics for each season
(JJA 2003 less rainy than
the others).
 Best results for D-PHASE
summer!
DOP
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Future plans
•
Verification vs “DPHASE observations”:
– assess performance over different domains (North and South of the Alps),
– study individual case studies,
– consider basin-by-basin performance.
•
“Think about” increasing horizontal resolution to 7 km.
•
Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel , Meteoswiss).
•
Implement “seamless COSMO ensemble system” merging COSMO-LEPS
and COSMO-SREPS.
•
Develop “hybrid” clustering technique (take boundary conditions from a
“grand-global” ensemble provided by mixing ECMWF-EPS and UKMO-MOGREPS).
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Dissemination during DOP




probabilistic products (www.smr.arpa.emr.it/prodotti and www.d-phase.info )
“deterministic” products (individual 16 COSMO-LEPS runs)
derived probability products (EM, ES)
images and alerts
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Time series of Ranked Probability Skill Score
 A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference
system. BS measures the mean squared difference between forecast and observation in probability space.
 Useful forecast system, if RPSS > 0.
RPSS
 the improvement of the
system performance is
detectable
for
all
forecast ranges along the
years;
 the poor performance of
the system in summer
2006 is confirmed;
 RPSS
always
positive
throughout 2007 and
especially high during
DOP.
 Lesser skill in 2008 (so
far).
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS methodology
Possible
evolution
scenarios
ensemble size
reduction
Cluster members chosen
as representative
members (RMs)
Initial conditions
Dim 1
LAM scenario
LAM scenario
LAM integrations driven by
RMs
LAM scenario
Initial conditions
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Dim 1
Why calibrate?
OBS-CLEPS/(OBS+CLEPS)/2
Q0.8
1971-2000 LT: 42h
Q0.95
Jan
COSMO-LEPS is not reliable,
Probabilities might be wrong!
Need for calibration
Jul
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Verification
Reliability Diagram:
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Winter 06/07 Q0.8 24h precipitation
+ (18-42h)
calibrated
raw
+ (66-90h)
• raw forecast overconfident, very limited skill
• strong improvement in reliability
• long lead-time forecasts more reliable
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
Summer 06 & 07 Q0.8 24h precipitation
+ (18-42h)
calibrated
raw
+ (66-90h)
• raw forecast overconfident
• modest improvement of reliability
• long lead-time forecasts more reliable
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008
The BSS debiased
BSS D  1 
BS
Weigel et al. 2006,
Mon. Wea. Rev.
BS REF  D
Special case BSS:
1
D
o (1  o )
M
M: Ensemble size
A.Montani; COSMO-LEPS status, verification, …
X COSMO meeting - Cracow - 15-19 September 2008