Transcript Dia 1 - Copernicus.org

Verification techniques for
high resolution NWP
precipitation forecasts
Emiel van der Plas ([email protected])
Kees Kok
Maurice Schmeits
[email protected] EMS 2013 (Reading UK)
Introduction
NWP has come a long way…
It was:
Then it became Hirlam:
Now it is Harmonie
It should be GALES
(or so)
It looks better…
But how is it better?
Does it perform better?
That remains to be seen…
2/15
2
[email protected] EMS 2013 (Reading UK)
Representation: “double penalty”
Forecast localised phenomena: False alarm + Miss = double penalty
Station (gauge) data:
Forecast
vs
Radar data:
When we take point-by-point errors (ME/RMSE):
3/15
3
[email protected] EMS 2013 (Reading UK)
This talk
HARP: Hirlam Aladin R-based verification Packages
Tools for spatial, ensemble verification
Based on R
FSS, SAL, …
Relies on eg SpatialVX package (NCAR)
Generalized MOS approach
Comparison high vs low resolution
Hirlam (11 km, hydrostatic)
Harmonie (2.5 km, non-hydrostatic, w/ & w/o Mode-S)
ECMWF (T1279, deterministic)
Lead times: +003, +006, +009, +012
Accumulated precipitation vs (Dutch) radar, synop
4/15
4
[email protected] EMS 2013 (Reading UK)
Neo-classical: neighborhood methods, FSS
• Options: FSS, ISS, SAL, …
Fraction Skill Score
(fuzzy verification)
(Roberts & Lean, 2008)
Straightforward interpretation
‘Resolves’ double penalty
observation
But
‘smoothes’ away
resolution that may
contain information!
( Vstorm D t )
== upscaling
Baserate , FSS 
5/15
5
[email protected] EMS 2013 (Reading UK)
forecast
FSS results:
Differences are sometimes
subtle:
• 1x1
• 3x3
7/15
7
[email protected] EMS 2013 (Reading UK)
FSS: more results
Higher resolutions: higher thresholds?
DMO!
8/15
8
[email protected] EMS 2013 (Reading UK)
Model Output Statistics
How would a trained meteorologist look at direct model
output?
Learn for each model, location, … separately!
9/15
9
/15
[email protected] EMS 2013 (Reading UK)
Model Output Statistics
• Construct a set of predictors (per model, station, starting and lead time):
For now: use precipitation only
Use various ‘areas of influence’: 25,50,75,100 km
DMO, coverage, max(DMO) within area,
distance to forecasted precipitation, …
Apply logistic regression
Forward stepwise selection, backward deletion
Probability of threshold exceedance!
Verify probabilities based on DMO, coefficients of selected predictors
Training data: day 1-20, `independent’ data: day 21 – 28/31
10/15
10
[email protected] EMS 2013 (Reading UK)
Model (predictor) selection
Based on AIC (Akaike Information Criterion)
Take the predictor with highest AIC in training set (day 1 - 20)
Test on independent set (day 21 – 28/31)
Sqrt(max)_100
More predictors != more skill
Sqrt(tot_100)
distext_100
exp2int_100
11/15
11
[email protected] EMS 2013 (Reading UK)
Model comparison (April – October 2012)
• Hirlam,
• Harmonie
(based on Hirlam)
ECMWF
12UTC+003
12UTC+006
12UTC+009
13/15
13
[email protected] EMS 2013 (Reading UK)
Discussion, to do
MOS method:
Stratification per station, season, …
More data necessary, reforecasting under way
Representation error: take (small) radar area
Use ELR, conditional probabilities for higher thresholds
Extend to wind, fog/visibility, MSG/cloud products, etc
FSS:
Use OPERA data
15/15
[email protected] EMS 2013 (Reading UK)
Conclusion/Discussion
Comparison between NWP’s of different resolution is, well, fuzzy
Realism != Score
Fraction Skill Score yields numbers,
but sometimes hard to draw conclusions
MOS method:
Resolution/model independent
Takes into account what we know
Doubles (potentially) as predictive guide
Thank you for your attention!
16/15
[email protected] EMS 2013 (Reading UK)
Extended Logistic Regression (ELR)
Binary predictand yi (here: precip > q)
Probability: logistic:
Joint likelihood:
L2 penalisation
minimise
(using R: stepPLR by Mee Young Park and Trevor Hastie, 2008):
Use threshold (sqrt(q)) as predictor:
complete distribution function (Wilks, 2009)
Few cases, many potential predictors: pool stations, max 5 terms
17
17/15
[email protected] EMS 2013 (Reading UK)