Transcript WWRP/WGNE Joint Working Group on Forecast Verification

WWRP/WGNE Joint Working Group
on Forecast Verification Research
Co-Chairs: Beth Ebert, Bureau of Meteorology
Laurie Wilson, Meteorological Service of Canada
WGNE-29, Melbourne, Australia, 10-14 March 2014
Aims
Verification component of WWRP, in collaboration with
WGNE, WCRP, CBS
• Develop and promote new verification methods
• Training on verification methodologies
• Ensure forecast verification is relevant to users
• Encourage sharing of observational data
• Promote importance of verification as a vital part of
experiments
• Promote collaboration among verification scientists,
model developers and forecast providers
2
Relationships / collaboration
WGCM
WGNE
TIGGE
SDS-WAS
HyMeX
Polar
Prediction
SWFDP
YOTC
Subseasonal to
Seasonal Prediction
WGSIP
High Impact
Weather (proposed)
CG-FV
3
Forecast & Research
Demonstration Projects
Sydney 2000 FDP
MAP D-PHASE
Beijing 2008 FDP/RDP
Typhoon Landfall FDP
SNOW-V10 RDP
Severe Weather FDP
FROST-14 FDP/RDP
SCMREX RDP
4
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Forecast Verification Framework of
The Sochi 2014 Winter Olympics
( FROST-2014 ~ Forecast and Research in the Olympic Sochi Testbed )
Pertti Nurmi
WMO WWRP JWGFVR aka Joint Working Group on Forecast Verification Research
Acknowledgements : FMI verification system development team & WMO FROST-2014 Expert Team
[email protected]
[email protected]
FROST-2014 Verification Framework
5
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
FROST-2014 : Models - Contributors
Model
Contributors
FMI Verification
Notes
Deterministic Forecasts
COSMO-RU - 7 km
COSMO-RU - 2 km
COSMO-RU - 1 km
GEM - 2.5 km
GEM - 1 km
GEM - 0.25 km
HARMONIE-SOCHI - 1 km
NMMB - 1 km
KMA
ARPA (Sochimini)
INCA (+48 hr)
Joint (+48 hr)
HMC,Russia
*
*
Env. Canada
*
*
*
*
*
FMI, Finland
NOAA,USA
KMA, S-Korea
ARPA SIMC, Italy
ZAMG, Austria
“consensus”
*
to be considered as RDP
under processing as of early-March
to be considered as RDP
Ensemble Forecasts
Aladin-LAEF-EPS - 7 km
GLAMEPS - 11 km
HarmonEPS - 2.5 km
COSMO-RU-EPS - 2 km
COSMO-S14-EPS - 7 km
NMMB-EPS - 7 km
ZAMG,Austria
HIRLAM, Norway
(*)
(*)
HMC,Russia
ARPA SIMC, Italy
NOAA, USA
under processing as of early-March
to be considered as RDP
to be considered as RDP
Nowcasting
MeteoExpert
Joint
INCA
NTW
CARDS
ABOM
RW Model, Harmonie-driven
[email protected]
IRAM,Russia
“consensus”
ZAMG,Austria
Env.Canada
Env.Canada
Env.Canada
FMI, Finland
*
*
to be considered as RDP
to be considered as RDP
to be considered as RDP
Verification within CoMoSeF project
FROST-2014 Verification Framework
6
Finnish Meteorological Institute
Ski
Biathlon
Ski Jump
Valley
Alpine Ski
[email protected]
FROST-2014 Verification Framework
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
MAE
[email protected]
FROST-2014 Verification Framework
8
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Sochi 2014 Road Weather Pilot: End-user Feedback
1 = Fully disagree / 2 = Disagree / 3 = Neutral / 4 = Agree / 5 = Fully agree
Date
Friction forecast
Storage terms Road weather
gave a good
(water, snow, ice) forecast was
estimate of
are realistic
useful
slipperiness
16.2.2014
20.2.2014
21.2.2014
5
23.2.2014
5
The
Impact
Issue
Notes (free text)
5
4
Precipitation, however road surface dried towards Adler
5
5
2
3
Dry road surface, i.e. No real need for RW forecast
5
2
Dry road surface, i.e. No real need for RW forecast
Variable road surface humidity
End-user ( = Finnish skiing service team ) comments:
•
•
•
•
[email protected]
”HARMONIE
”HARMONIE
”HARMONIE
useful”
”HARMONIE
was superior”
visibility forecasts were excellent”
precipitation and cloud height forecasts were highly
five-panel user interface was really good”
FROST-2014 Verification Framework
9
XXII Olympic Winter Games
Feb - Mar 2014
Finnish Meteorological Institute
Future actions and activities
 Comprehensive diagnostic verification
 Re-run of statistics after full data sets available
 Compare with others’ verification results
 Joint reporting and publishing with WMO FROST-2014 expert group
 Extension to societal aspects  The Impact Issue  SERA group ?
 Presentation of results:
 WWOSC, Montreal; FROST Fall Meeting, ECAM etc…
[email protected]
FROST-2014 Verification Framework
10
Outreach and training
• Workshops and
tutorials
• EUMETCAL training
modules
• SWFDP training
• Verification web page
• Sharing of tools
http://www.cawcr.gov.au/projects/verification/
11
International Verification
Methods Workshops
• Tutorial
•
•
•
•
~35 participants
Lectures and exercises
Tools to take home
Group projects,
presented at scientific
workshop
• Scientific workshop
• ~100 participants
• Talks, posters,
keynotes, discussions
12
Promotion of best practice
Recommendations for the verification and
intercomparison of QPFs and PQPFs from
operational NWP models (2008)
Recommended methods for evaluating
cloud and related parameters (2012)
Verification methods for tropical cyclone
forecasts (2013)
Suggested methods for the verification of
high resolution precipitation forecasts
against high resolution limited area
observations (in preparation)
13
Verification of deterministic
TC forecasts
15
Verification of probabilistic
TC forecasts
16
Beyond track and intensity…
Track error distribution
Precipitation
Model 1
Model 2
Wind speed
Also:
• Storm structure
• Storm surge
• Waves
• Genesis
• Seasonal forecasts
• Experimental methods
17
Spatial Verification Method
Intercomparison Project
• Phase 1 – understanding
the methods
Tier 3
• Phase 2 – testing the
methods
Tier 2a
– Point and gridded
observations including
ensemble observations
– MAP D-PHASE / COPS
dataset
Sensitivity tests
to method parameters
– Deterministic & ensemble
forecasts
Core
Determ. precip
+ VERA anal
+ JDC obs
Ensemble wind
+ VERA anal
+ JDC obs
Other variables ensemble
+ VERA ensemble
+ JDC obs
– "MesoVICT" – precipitation
and rain in complex terrain
Tier 1
Tier 2b
• Model reruns??
18
Verification research
priorities
• High resolution NWP
• Ensembles
• Seamless forecasts – nowcasts  short-medium range
 sub-seasonal  seasonal  …
• Warnings (intensity, timing, spatial extent, etc.)
• Polar forecasts
• Urban forecasts
• Hazard impacts / user focus
19
High Resolution Assessment (HiRA)
17
3 xx7
7
317
Make use of spatial verification methods which compare
single observations to a forecast neighbourhood
around the observation location.
x
Forecast
neighbourhood
Observation
NOT upscaling
or smoothing!
© Crown copyright 2013 Met Office
HiRA framework outline @ grid scale
•
Use standard synoptic observations
and a range of neighbourhood sizes
•
Use 24h persisted observations as
reference
•
The method needs to be able to
compare:
 Deterministic vs deterministic
(different resolutions, and test vs
control of the same resolution)
 Deterministic vs EPS
Variable
Old
New
Temp
RMSESS
CRPSS
MAE
Vector wind
(wind
speed)
RMSVESS
MAE
RPSS
Cloud cover
ETS
BSS
PC
CBH
ETS
BSS
PC
Visibility
ETS
BSS
PC
ETS
PC
BSS
 EPS vs EPS

Test whether differences are
statistically significant (Wilcoxon)
1h precip

Grid scale calculated for reference
 NOT main focus.
RMS(V)ESS = Root Mean Square (Vector) Error Skill Score
ETS = Equitable Threat Score
BSS = Brier Skill Score
RPSS = Ranked Probability Skill Score
CRPSS = Continuous Ranked Probability Skill Score
MAE = Mean Absolute Error
PC = Proportion Correct
© Crown copyright 2013 Met Office
Mittermaier, WAF 2013
Approach
2.2 km MOGREPS-UK ensemble
• Deterministic
forecast with/
without
neighbourhood
or
• Ensemble
members
with/without
neighbourhoods
Comparisons:
1 gridpoint with 12 single ensemble gridpoints
or
9 gridpoint with 12 * 9 ensemble gridpoints  enhanced sampling
© Crown copyright 2013 Met Office
MOGREPS-UK @ 2.2 km
UKV @ 1.5 km
Time series – skill against persistence
© Crown copyright 2013 Met Office
Mittermaier 2014 in press
Seamless verification
Spatial scale
Seamless forecasts - consistent across space/time scales
single modelling system or blended
probabilistic / ensemble
seasonal decadal climate
subseasonal prediction prediction change
NWP prediction
global
regional
local
very
short
range
nowcasts
Ebert, E., et al., 2013: Progress and challenges in
forecast verification. Meteorol. Appl., 20, 130–139.
point
minutes
hours
days
weeks
months
forecast aggregation time
years
decades
Multi-temporal verification
Zhu et al., 2013: Seamless precipitation prediction skill in the tropics and extratropics from a global
model, Monthly Weather Review
1. Use precipitation for fair global comparison.
2. Compute skill globally for a large range of lead times.
3. As we increase the lead time, we also increase the time-averaging
window for a seamless transition from weather to climate.
Application to long range forecasts
a. POAMA-2 ensemble forecast system
–
–
–
–
Initialized with realistic atmospheric, land, and ocean initial conditions
Coupled breeding scheme to produce a burst ensemble of 11 members
3 versions of the model to provide in total 33 members
Hindcasts from the 1st, 11th, and 21st of each month (out to 120 days)
b. Observations
– GPCP daily precipitation (blended station and satellite), 1996-2009.
c. Measures of prediction skill
– Tried different verification measures (ROC score, Brier score, correlation)
– In the end chose the simplest: the correlation of the ensemble mean
– Two versions:
–
CORt - using total precipitation values
–
CORa - using anomalies with respect to separate climatologies for the
hindcasts and observations.
– CORt is more usual for weather prediction; CORa is more usual for seasonal
prediction.
CORt
1d1d: Extratropics better than tropics; winter extratropics better than summer.
4w4w: ENSO dominates. Something strange around 50-65ºS in DJF?
Zonally-averaged CORt
ENSO peak at equator is apparent at all lead times.
Extratropical skill drops rapidly from 1d1d to 1w1w and then levels-off.
CORa: plotted as a function of the log(time)
Skill in tropics (10ºS-10ºN) overtakes skill in extratropics for 4d4d in DJF and 1w1w
in JJA.
Comparison with persistence
An important component of predictability is the prediction skill that can come from
persistence. What is its contribution here?
1 week average
1 day average
Use the most recent observed
precipitation anomalies to
predict future anomalies.
Initial condition
CORa for persistence and model
1d1d, DJF
Persistence
Model
4w4w, DJF
Next few years
• Promote verification research for high resolution NWP,
ensembles, seamless, warnings, polar, urban, hazard impacts
• 7th International Verification Methods Workshop
• SWFDP verification training
• New FDPs and RDPs (e.g., 2018 Olympics, Lake Victoria, La
Plata Basin, …)
• THORPEX legacy projects (PPP, S2S, HIWeather)
• Spatial Verification Methods Intercomparison (Phase 2)
• Liaison with CIMO
32