Verification of SREF Aviation Forecasts at Binghamton, NY
Download
Report
Transcript Verification of SREF Aviation Forecasts at Binghamton, NY
Verification of SREF Aviation
Forecasts at Binghamton, NY
Justin Arnott
NOAA / NWS Binghamton, NY
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Motivation
Ensemble information making impacts throughout the
forecast process
Examples SREFs, MREFs, NAEFS, ECMWF Ensemble
SREF resolution is reaching the mesoscale (32-45 km), a
scale at which some aviation impacts may be resolvable
Can SREFs provide useful information in the aviation
forecast process?
The 10th annual Northeast Regional Operational Workshop, Albany, NY
SREF
21 member multi-model ensemble
10 ETA(32 km, 60 vertical levels)
3 NCEP - NMM WRF (40 km, 52 vertical levels)
3 NCAR - ARW WRF (45 km, 35 vertical levels)
5 NCEP - RSM (45 km, 28 vertical levels)
Various IC/BCs, physical parameterizations
The 10th annual Northeast Regional Operational Workshop, Albany, NY
SREF Aviation Forecasts
Numerous Aviation parameters are created from the
9 and 21Z simulations
•Some outputted directly,
some derived
•Include: CIG, VSBY, icing,
turbulence, jet stream, shear,
convection, precipitation,
freezing level, fog, etc.
For creating TAFs,
CIG/VSBY fields may
provide the most potential
http://wwwt.emc.ncep.noaa.gov/mmb/SREF/SREF.html
use
The 10th annual Northeast Regional Operational Workshop, Albany, NY
SREF Aviation Forecasts
Verification of SREF CIG/VSBY forecasts has
been minimal
Alaska Region has completed a study using SREF
MEAN values
No verification study has been conducted over the
lower 48
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Expectations
CIGS/VSBYS, can vary greatly on scales far less than the 32-45 km
scale of the SREFs
Some MVFR/IFR events
are more localized than others
Summer MVFR/IFR tends
to be more localized
Winter MVFR/IFR is
typically more widespread
~40 km
Bottom Line: Expect relatively poor SREF performance during the
warm season, with improvements during the cool season
The 10th annual Northeast Regional Operational Workshop, Albany, NY
The Study So Far…
Gather SREF CIG/VISBY data daily starting July 1, 2008
Data provided specifically for the project by Binbin Zhou at
NCEP
Compute POD/FAR/CSI/BIAS statistics for JulySeptember at KBGM
MVFR and IFR (due to small sample size)
Investigate using different probabilities to base forecast on
50%, 30%, 20%, 10%
The 10th annual Northeast Regional Operational Workshop, Albany, NY
The Study So Far…continued
Compare SREF results to WFO Binghamton, NY
and GFS MOS forecasts
Use stats-on-demand web interface to obtain this data
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results
Very little MVFR/IFR at KBGM in July-September
IFR or MVFR only ~10% of the time
So, we’re aiming at a very small target!
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – MVFR/IFR CIGS
SREF SREF SREF SREF SREF BGM GFS
Mean 50% 30% 20% 10%
MOS
POD 0.61
0.48
0.75
0.86
0.99
0.72
0.60
FAR
0.66
0.46
0.55
0.66
0.74
0.34
0.37
CSI
0.27
0.34
0.39
0.32
0.26
0.53
0.44
BIAS 1.80
0.89
1.67
2.57
3.84
1.08
0.96
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – MVFR/IFR CIGS
WFO BGM/GFS MOS more skillful than the
SREF mean or any SREF probability threshold
30% probability threshold shows best skill
Large false alarm ratios with nearly all SREF
forecasts
Large positive biases for SREF mean and nearly
all probability thresholds
IE over forecasting MVFR/IFR CIGS
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Comparing Apples with Oranges?
These results compare 9-21 hr SREF forecasts
with 0-6 hour WFO BGM forecasts and 6-12 hr
GFS forecasts
Due to later availability of SREF data (9Z SREFS
not available for use until 18Z TAFs)
How well does a 9-24 hr GFS MOS (or BGM)
forecast perform?
21 hr not available using stats-on-demand
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – MVFR/IFR CIGS
SREF SREF SREF SREF SREF BGM
Mean 50% 30% 20% 10% 9-24hr
GFS
MOS
9-24hr
POD 0.61
0.48
0.75
0.86
0.99
0.63
0.64
FAR
0.66
0.46
0.55
0.66
0.74
0.33
0.27
CSI
0.27
0.34
0.39
0.32
0.26
0.48
0.52
BIAS 1.80
0.89
1.67
2.57
3.84
0.93
0.88
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – MVFR/IFR CIGS
WFO BGM / GFS MOS performance does not
decrease substantially by changing the
comparison time window
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – MVFR/IFR VSBYS
SREF SREF SREF SREF
Mean 30% 20% 10%
BGM
GFS
MOS
BGM
9-24hr
GFS
MOS
9-24hr
POD 0.04
0.14
0.31
0.58
0.66
0.65
0.52
0.54
FAR
0.00
0.46
0.59
0.64
0.53
0.58
0.60
0.63
CSI
0.04
0.12
0.21
0.28
0.38
0.34
0.29
0.28
BIAS 0.04
0.25
0.74
1.62
1.42
1.53
1.30
1.46
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results MVFR/IFR VSBYS
SREF Mean as well as 30 and 20% thresholds
fail to identify enough cases to be useful
10% threshold shows greatest skill and is
comparable to GFS MOS forecasts!
There is a significant positive bias at this threshold
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – IFR CIGS
SREF SREF SREF SREF
Mean 30% 20% 10%
BGM
GFS
MOS
BGM
9-24hr
GFS
MOS
9-24hr
POD 0.29
0.48
0.58
0.83
0.51
0.53
0.25
0.38
FAR
0.42
0.42
0.62
0.73
0.24
0.48
0.40
0.48
CSI
0.24
0.36
0.30
0.26
0.44
0.36
0.22
0.28
BIAS 0.50
0.83
1.50
3.03
0.67
1.00
0.42
0.73
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – IFR CIGS
SREF Mean poor at identifying IFR CIGS
CSI scores for SREF probability fields are an
improvement on WFO BGM/GFS MOS
Bias scores indicate underforecasting at a 30%
threshold but large overforecasting for 20,10%
thresholds
WFO BGM/ GFS MOS tend to underforecast IFR
CIGS
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – IFR VSBYS
SREF SREF SREF SREF
Mean 30% 20% 10%
BGM
GFS
MOS
BGM
9-24hr
GFS
MOS
9-24hr
POD 0.04
0.15
0.26
0.67
0.45
0.29
0.28
0.29
FAR
0
0.78
0.88
0.87
0.46
0.70
0.62
0.76
CSI
0.04
0.10
0.09
0.12
0.33
0.18
0.19
0.15
BIAS 0.04
0.67
2.22
5.25
0.85
0.97
0.73
1.22
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Results – IFR VSBYS
SREF can only readily identify IFR VSBY
situations except at the 10% threshold
Tremendous biases indicate, however that these
forecasts are not useful
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Summary
SREF performance occasionally comparable to GFS MOS
potentially useful guidance
Promising for “~direct” model output
Hampered by later arrival time at WFO
MEAN fields show little/no skill
Different probability thresholds show best skill for different
variables/categories
CIGS:
SREFS frequently over forecast MVFR/IFR CIGS
SREFS perform surprisingly well with IFR CIGS
Best performing probability thresholds are 20-30% balancing
BIAS with CSI
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Summary, continued
VSBYS:
SREFS have trouble identifying VSBY restrictions
10% probability threshold necessary to get any
signal, but this may be useful for MVFR/IFR (not
IFR alone)
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Future Plans
Continue computing statistics through the
upcoming cool season
Expect improved results given more widespread (i.e.
resolvable) restrictions
Expand to other WFO BGM TAF sites
Work with NOAA/NWS/NCEP in improving
calculations of CIG/VSBY
The 10th annual Northeast Regional Operational Workshop, Albany, NY
Acknowledgements
Binbin Zhou – NOAA/NWS/NCEP
For providing access to SREF data in near real-time
The 10th annual Northeast Regional Operational Workshop, Albany, NY