Transcript ppt - Cosmo

Performance of the COSMO-LEPS system
during D-PHASE Operations Period
Andrea Montani,
Chiara Marsigli and Tiziana Paccagnella
ARPA-SIM
Hydrometeorological service of Emilia-Romagna, Italy
X General COSMO meeting
Cracow, 15-19 September 2008
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Outline
• Introduction
• Methodology of COSMO-LEPS
• Verification results (SYNOP on the GTS):
– performance from 2002 onwards
– performance during DOP
• Future plans
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS (developed at ARPA-SIM)
• What is it?
It is a Limited-area Ensemble Prediction System (LEPS),
based on COSMO-model and implemented within COSMO
(COnsortium for Small-scale MOdelling, which includes
Germany, Greece, Italy, Poland, Romania, Switzerland).
• Why?
It was developed to combine the advantages of global-model
ensembles with the high-resolution details gained by the
LAMs, so as to identify the possible occurrence of severe
and localised weather events (heavy rainfall, strong winds,
temperature anomalies, snowfall, …)
generation of COSMO-LEPS to improve the Late-Short
(48hr) to Early-Medium (132hr) range forecast of
severe weather events.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS methodology
Possible
evolution
scenarios
ensemble size
reduction
Cluster members chosen
as representative
members (RMs)
Initial conditions
Dim 1
LAM scenario
LAM scenario
LAM integrations driven by
RMs
LAM scenario
Initial conditions
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Dim 1
COSMO-LEPS suite @ ECMWF
during D-PHASE Operations Period (from 1 June to 30 November 2007)
d
d-1
d+1
d+2
older EPS
00
d+3
d+4
d+5
4 variables
ZUVQ
3 levels
500 700 850 hPa
Cluster Analysis and RM identification
younger EPS
12
2
time
steps
European
area
Complete
Linkage
•
COSMOLEPS
clustering
area
COSMOLEPS
Integratio
n Domain
•
•
•
•
suite running as a ”timecritical application” managed
by ARPA-SIM;
Δx ~ 10 km; 40 ML;
COSM0 v3.20;
fc length: 132h;
Computer time (4.7 million BU
for 2007) provided by the
COSMO partners which are
ECMWF member states.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS suite @ ECMWF
after D-PHASE Operations Period (since December 2007)
d
d-1
d+1
d+2
older EPS
00
d+3
d+4
d+5
4 variables
ZUVQ
3 levels
500 700 850 hPa
Cluster Analysis and RM identification
younger EPS
12
2
time
steps
European
area
Complete
Linkage
•
COSMOLEPS
clustering
area
COSMOLEPS
Integratio
n Domain
•
•
•
suite runs as a ”time-critical
application” managed by ARPASIM;
Δx ~ 10 km; 40 ML; fc+ 132h;
COSM0 v4.0 since Dec07 (with
RK + multi-layer);
computer time (5.8 million BU
for 2008) provided by the
COSMO partners which are
ECMWF member states.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Dissemination during DOP




probabilistic products (www.smr.arpa.emr.it/prodotti and www.d-phase.info )
“deterministic” products (individual 16 COSMO-LEPS runs)
derived probability products (EM, ES)
images and alerts
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Objective verification of COSMO-LEPS
–
SYNOP on the GTS
Main features:
variable:
12h cumulated precip (18-06, 06-18 UTC);
period :
from Dec 2002 to Aug 2008;
region:
43-50N, 2-18E (MAP D-PHASE area);
method:
nearest grid point; no-weighted fcst;
obs:
synop reports (about 470 stations/day);
fcst ranges: 6-18h, 18-30h, …, 102-114h, 114-126h;
thresholds: 1, 5, 10, 15, 25, 50 mm/12h;
system:
COSMO-LEPS;
scores:
ROC area, RPSS, Outliers
both (69) monthly and (23) seasonal scores were computed
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area
 Area under the curve in the HIT rate vs FAR diagram.
 Valuable forecast systems have ROC area values > 0.6.
ROC
 the positive impact
of increasing the
ensemble size in
2004 is evident for
all thresholds and
for different forecast
ranges.
 poor performance of
the system in Spring
and Summer 2006
(both
particularly
dry),
despite
system upgrades.
 good performance
during DOP 2007.
 limited impact of
system upgrades in
2008
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Seasonal scores of ROC area
 Performance of the system assessed for the last 5 summers (JJA) and the last 5 autumns (SON).
 Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6.
ROC
 Need to take into
account
the
different statistics
for each season
(JJA 2004 and
SON 2006 less
rainy than the
others).
 Good
scores
obtained
during
DOP 2007 for
either seasons.
 Good scores for
JJA 2008, more
evident for longer
ranges.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Time series of Ranked Probability Skill Score
 A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference
system. BS measures the mean squared difference between forecast and observation in probability space.
 Useful forecast system, if RPSS > 0.
RPSS
 the improvement of the
system performance is
detectable
for
all
forecast ranges along the
years;
 the poor performance of
the system in summer
2006 is confirmed;
 RPSS
always
positive
throughout 2007 and
especially high during
DOP.
 Lesser skill in 2008 (so
far).
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Outliers: time series + seasonal scores
 How many times the analysis is out of the forecast interval spanned by the ensemble members.
 … the lower the better …
 Performance of the system assessed as time series and for the last 5 summers (JJA).
OUTL
 Evident seasonal cycle
(more
outliers
in
winter), but overall
reduction of outliers in
the years up to 2007.
 Reduction of outliers
from one summer to
the other, related to the
increase of ensemble
size (evident for the
10m to 16m increase).
 Need to take into
account the different
statistics
for
each
season.
 Best results for the last
2 summers.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Main results
•
Since November 2002, COSMO-LEPS system has been running on a daily basis.
•
Since December 2005, COSMO-LEPS is a “member-state time-critical application”.
•
COSMO-LEPS products are routinely used in met-ops rooms across COSMO
community as well as in research Projects (e.g. PREVIEW: see the poster).
•
During D-PHASE Operations Period, COSMO-LEPS products were used both in the
Visualisation Platform and to feed hydrological models.
Time-series verification scores indicate the following trends:
 increase in ROC area scores and reduction in outliers percentages;
 positive impact of increasing the population from 5 to 10 members (June 2004);
 some deficiency in the skill of the system can be identified after the system upgrades
occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT
 scores are encouraging throughout DPHASE Operations Period;
 system upgrades of Dec07 (RK + multi-layer + new perturbations) brought
modest positive impact.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Thank you !
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Present methodology
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Operational set-up
Core products:
 16 perturbed COSMO-model runs (ICs and 3-hourly
BCs from 16 EPS members) to generate, “via weights”,
probabilistic output (start at 12UTC; t = 132h);
Additional products:
 1 deterministic run (ICs and 3-hourly BCs from the highresolution deterministic ECMWF forecast) to “join”
deterministic and probabilistic approaches (start at
12UTC; t = 132h);
 1 “hindcast” run (ICs and 3-hourly BCs from ECMWF
analyses) to “downscale” ECMWF information (start at
00UTC; t = 36h).
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Recent updates
Changes to the operational suite on 1 December 2007:
•
•
•
•
new model version (COSMO-LM 4.0);
new numerics (Runge-Kutta scheme);
use of multi-layer soil model;
new perturbations introduced relative to the maximal turbulent length scale (tur_len) and to the
length scale of thermal surface patterns (pat_len)  to increase spread and maintain skill.
 Old system and new system ran
in parallel for 52 days (October
and November 2007);
 Larger T2m spread in new
system for all forecast ranges;
 Comparable skill (in terms of
T2m root-mean-square error) of
the two systems.
back to plans
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area for COSMO-LEPS
(2)
 Area under the curve in the FAR vs HIT diagram.
 Valuable forecast systems have ROC area values > 0.6
ROC
fc step: 78-90h
 the positive impact of
increasing the ensemble
size in 2004 is evident
for all thresholds and for
different fcst ranges.
 poor performance of the
system in Spring and
Summer 2006, despite
system upgrades.
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Time series of Brier Skill Score
 BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems if BSS > 0.
 BS measures the mean squared difference between forecast and observation in probability space.
 Equivalent to MSE for deterministic forecast.
BSS
fc step: 30-42h
 improvement of
performance detectable
for all thresholds along
the years;
 still problems with high
thresholds, but good
trend in 2007.
Jun04: 5m  10m
Feb06: 10m16m; 32ML  40 ML
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Semi-diurnal cycle in COSMO-LEPS scores
 BSS score … the higher the better …
 Performance of the system assessed for 5 different Summers (JJA).
BSS
 Evident 12-hour cycle in BSS
scores (the same holds for
RPSS, while less evident for
ROC area scores).
 Better performance of the
system
for
“night-time”
precipitation, that is for rainfall
predicted between 18Z and 6Z
(ranges 30-42h, 54-66h, …).
 The amplitude of the cycle is
somewhat reduced throughout
the years and with increasing
forecast range.
 The bad performance in
Summer 2006 is confirmed.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Score dependence on the domain size (1)
 Verification of COSMO-LEPS against synop reports over the MAP D-PHASE area (~ 470 stations; MAPDOM)
and the full domain (~ 1500 stations; fulldom):
 different statistics of the verification samples;
 up to now, performance of the system over the 2 domains assessed only for the last 6 months
(March-August 2007).
 difficult to draw general conclusions
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Score dependence on the domain size (2)
 RPSS score… the higher the better… (and positive).
 ROC area… the higher the better… (and above 0.6).
 Smoother transitions from
month to month in “fulldom”
scores.
 Slightly better performance of
COSMO-LEPS
over
the
MAPDOM, but the signal
varies from month to month.
 Higher predictability with
orographic forcing?
 Need to check individual
regions and/or to stratify
for type of stations.
 Outliers percentage … the lower the better.
RPSS
OUTL
ROC
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
MAM06
Average values (boxes 0.5 x 0.5)
tp > 1mm/24h
tp > 5mm/24h
COSMO-LEPS
16-MEMBER EPS
51-MEMBER EPS
As regards AVERAGE precipitation above
these two threshols, the 3 systems have
similar performance.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Main results
•
COSMO-LEPS system runs on a daily basis since November 2002 and it has
become a “member-state time-critical application” at ECMWF ( partial
involvement of ECMWF operators in the suite management).
•
Time series scores identify positive trends in the performance of the system:
•
•
•
increase in ROC area scores and reduction in outliers percentages;
positive impact of increasing the population from 5 to 10 members (June 2004);
some deficiency in the skill of the system identified after the system upgrades occurred
on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS
upgrade!!!)  need of further investigation.
•
High-resolution verification shows better scores of COSMO-LEPS with respect to
EPS in forecasting precipitation maxima within boxes.
•
No clear impact of the weighting procedure as regards precipitation, BUT
COSMO-LEPS ensemble mean for 2m temperature has better skill (lower standard
deviation to observations) with weighting the ensemble members according to
cluster populations compared to the mean with unweighted members.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area
 Area under the curve in the HIT rate vs FAR diagram.
 Valuable forecast systems have ROC area values > 0.6.
ROC
 the positive impact
of increasing the
ensemble size in
2004 is evident for
all thresholds and
for different forecast
ranges.
 poor performance of
the system in Spring
and Summer 2006
(both
particularly
dry),
despite
system upgrades.
 good performance
during DOP 2007.
fc step: 30-42h
DOP
fc step: 78-90h
Jun04: 5m  10m
Jun04: 5m  10m
DOP
Feb06: 10m  16m; 32ML  40 ML
Feb06: 10m  16m; 32ML  40 ML
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Seasonal scores of ROC area
 Performance of the system assessed for 5 different summers (JJA) and 5 different autumns (SON).
 Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6.
ROC
 Need to take into
account
the
different statistics
for each season
(JJA 2003 and
SON 2006 less
rainy than the
others).
 Best
scores
obtained during
DOP 2007 for
either seasons.
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Time series of Ranked Probability Skill Score
 A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference
system. BS measures the mean squared difference between forecast and observation in probability space.
 Useful forecast system, if RPSS > 0.
RPSS
DOP
 the improvement of the
system performance is
detectable
for
all
forecast ranges along the
years;
 the poor performance of
the system in summer
2006 is confirmed;
 RPSS
always
positive
throughout 2007 and
especially high during
DOP.
Jun04: 5m  10m
Feb06: 10m16m; 32ML  40 ML
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Outliers: time series + seasonal scores
 How many times the analysis is out of the forecast interval spanned by the ensemble members.
 … the lower the better …
 Performance of the system assessed as time series and for 5 different summers (JJA).
OUTL
 Evident seasonal cycle
(more
outliers
in
winter), but overall
reduction of outliers in
the years.
 Reduction of outliers
from one summer to the
other, related to the
increase of ensemble
size (more evident for
the 5 to 10 increase).
 Need to take into
account the different
statistics for each season
(JJA 2003 less rainy than
the others).
 Best results for D-PHASE
summer!
DOP
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008
Future plans
•
Verification vs “DPHASE observations”:
– assess performance over different domains (North and South of the Alps),
– study individual case studies,
– consider basin-by-basin performance.
•
“Think about” increasing horizontal resolution to 7 km.
•
Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel , Meteoswiss).
•
Implement “seamless COSMO ensemble system” merging COSMO-LEPS
and COSMO-SREPS.
•
Develop “hybrid” clustering technique (take boundary conditions from a
“grand-global” ensemble provided by mixing ECMWF-EPS and UKMO-MOGREPS).
A.Montani; Performance of the COSMO-LEPS system during DOP
COSMO meeting - Cracow - 15-19 September 2008