Transcript Application of the CRA Method

Application of the CRA
Method
William A. Gallus, Jr.
Iowa State University
Beth Ebert
Center for Australian Weather and
Climate Research
Bureau of Meteorology
Idealized cases - geometric
Observed
(1) 50 pts to the right
(2) 200 pts to the right
(3) 125 pts to the right
and big
(4) 125 pts to the right
and rotated
(5) 125 pts to the right
and huge
Which forecast is best?
Traditional verification yields same
statistics for cases 1 and 2
forecast
5th case – traditional verification
forecast
forecast
1st case – CRA verification
2nd case – CRA verification
CRA Technique yields similar
results with cases 3 and 4
case – CRA
verification
th
5
RESULTS ARE SENSITIVE TO SEARCH BOX FOR DISPLACEMENTS
Increase of size of rectangle (extra 90 instead of 30 pts) affects results
Further increase from 90 pts to 150 pts does not result in additional change
1st case vs. 5th case
Observed
Area (# gridpoints)
Case 1 Case 5
7815
7815
62789
Average rain rate (mm/h)
.36
.36
10.05
Maximum rain rate (mm/h)
25.4
25.4
25.4
Rain volume (km3)
1.88
1.88
14.45
Displacement east
2.34°
6.65°
Displacement north
0°
0°
RMS error after shift
0
11.92
1.00
0.26
100%
12%
Volume error (%)
0%
47%
Pattern error (%)
0%
41%
Correlation coefficient after shift
Displacement error (%)
Perturbed cases
(2) Shift 12 pts right, 20 pts
down, intensity*1.5
"Observed"
Which forecast is better?
1000 km
(1) Shift 24 pts right, 40 pts
down
1st case – traditional verification
(1) Shift 24 pts right, 40 pts down
2nd case – traditional verification
(2) Shift 12 pts right, 20 pts down,
intensity*1.5
CRA verification
Case 2
Case 1
Threshold=5 mm/h
CRA verification
Case 2
Case 1
Threshold=5 mm/h
CRA verification
Case 2
Case 1
Threshold=5 mm/h
Problem?
System far from boundary in Case 1 – small shift – behaves as expected
System closer to boundary yields unexpected results – not all error is displacement
Problem is more serious for smaller system at edge of domain
Central system works well through medium displacements
But…. Larger displacement yields odd results
Summary
• CRA requirement for forecast and
observed systems to be contiguous
may limit some applications
• Problems occur for systems near the
domain boundaries – not yet clear what
causes the problems
Results from separate study
using object-oriented techniques
to verify ensembles
• Both CRA and MODE have been applied
to 6-hr forecasts from two 15km 8
member WRF ensembles integrated for
60 h for 72 cases
• This results in 10 x 16 x 72 = 11,520
evaluations (plots, tables….) from each
approach
• Results were compared to Clark et al
(2008) study
Clark et al. study
• Clark et al. (2008) looked at two 8 member
WRF ensembles, one using mixed IC/LBC,
the other mixed physics/dynamic cores
• Spread & skill initially may have been
better in mixed physics ensemble vs.
IC/LBC one, but spread grew much faster
in the IC/LBC one, and it performed better
than the mixed physics ensemble at later
times (after 30-36 h) in these 120 h
integrations.
0.5
mm
2.5
mm
Areas under ROC curves for both ensembles (Clark et al. 2007)
Skill initially better in mixed ensemble but IC/LBC becomes better after hour
30-36
Diurnal Cycle
Variance continues to grow in IC/LBC ensemble but levels off after hour 30 in
mixed ensemble. MSE always worse for mean of mixed ensemble – and
performance worsens with time relative to IC/LBC ensemble.
0.5
mm
2.5
mm
Spread Ratio also shows dramatically different behavior with
increasing spread in IC/LBC ensemble but little or no growth in
mixed ensemble after first 24 hours
Questions:
• Do the object parameters from the CRA
and MODE techniques show the
different behaviors between the Mix
and IC/LBC ensembles?
• Do the object parameters from the CRA
and MODE techniques show an
influence from the diurnal trends in
observed precipitation?
Diurnal signal not pronounced,
only weak hint of IC/LBC
tendency to have increasing
spread with time – and only in
MODE results
0.45
0.4
Rain Rate Standard
Deviation (in.) –
mean usually around
.5 inch
0.35
0.3
0.25
Mix-MODE
0.2
0.15
IC/LBC-MODE
0.1
Mix-CRA
0.05
IC/LBC-CRA
0
06
1
12
2
18
3
24
4
30
5
36
6
Forecast Hour
42
7
48
8
549
60
10
Wet
times
in
blue
1.8
1.6
1.4
Standard Deviation of Rain Volume
(km3) – MODE values multiplied by
10 (mean ~ 1)
No diurnal signal, hard
to see different trends
between 2 ensembles
Mix-MODE
Rain Volume
1.2
1
0.8
IC/LBC-CRA
0.6
0.4
Mix-CRA
IC/LBCMODE
0.2
0
106
12
2
18
3
244
30
5
36
6
Time
42
7
48
8
54
9
60
10
CRA results show both ensembles
with growing spread, and IC/LBC
having faster growth
600
Number of Points
500
Areal Coverage
Standard Deviation
(number of points
above .25 inch) –
Mean ~ 800 pts
400
IC/LBCCRA
300
Mix-CRA
200
Mix-MODE
100
0
IC/LBC-MODE
1
06
2
12
3
18
4
24
5
30
Time
6
36
7
42
8
48
549
10
60
No clear diurnal
signal, both CRA
& MODE show
max in 24-48 h
16
14
Number of Cases (SD)
12
10
Mix-MODE
8
6
IC/LBCMODE
4
Mix-CRA
2
IC/LBC-CRA
0
1
06
2
12
3
18
244
5
30
Time
6
36
427
488
9
54
10
60
20
Mix-CRA
18
16
IC/LBC-MODE
Displacement SD (km)
Mix-MODE
14
12
10
IC/LBC-CRA
8
6
No diurnal signal, no obvious
differences in behavior of Mix and
IC/LBC
4
2
0
1
06
2
12
3
18
4
24
5
30
6
36
Time
427
488
9
54
6010
Other questions:
• Is the mean of the ensemble’s
distribution of object-based parameters
a good forecast (better than ensemble
mean put into CRA/MODE)?
• Does an increase in spread imply less
predictability?
• How should a forecaster handle a case
where only a subset of members show
an object?
These questions have been examined using CRA results
0.15
wet
0.1
Rain Rate error (in)
Mix Ensemble – in general,
slight positive bias in rain rate,
with Probability Matching
forecast slightly less intense
than mean of rates from
members (PM usually better
but not by much). Only during
06-18 period does observed
rate not fall within forecasted
range.
0.05
mean
PM
0
1
2
3
4
5
6
7
8
9
10
dry
-0.05
-0.1
06 12
18 24 30 36 42
48
54 60
Time
0.04
wet
0.02
0
Rain Rate Error (in)
IC/LBC – usually too dry with
rain rate (at all hours except 0618), Probability Matching
forecast exhibits much more
variable behavior, again its
performance is comparable to
mean of rates of members
1
2
-0.02
3
4
5
6
7
8
9
10
mean
-0.04
PM
-0.06
dry
-0.08
-0.1
-0.12
06 12
18 24 30 36 42
Time
48
54 60
0.15
Rain Rate Error (in)
0.1
0.05
0
1
2
3
4
5
6
7
8
9
10
-0.05
-0.1
-0.15
Time
Notice that at all times, the observed rain rate falls within the
range of values from the full 16 member ensemble – indicating
potential value for forecasting
0.8
Mix Ensemble – clear diurnal
signal, usually too much rain
volume except at times of
observed peak, when it is too
small. Probability Matching
equal in skill to mean of
member volumes
wet
0.6
Rain Volume Error
0.4
mean
PM
0.2
0
1
2
3
4
5
6
-0.2
7
8
9
10
dry
-0.4
-0.6
06 12
18 24 30 36 42
48
54 60
Time
0.3
wet
0.2
IC/LBC Ensemble – also clear
diurnal signal, less volume than
Mix ensemble, Probability
Matching usually a little wetter
but generally comparable to
mean of members
Rain Volume Error (km**3)
0.1
0
1
2
3
4
5
6
7
8
9
10
-0.1
mean
PM
-0.2
-0.3
dry
-0.4
-0.5
-0.6
-0.7
06 12
18 24 30 36 42
Time
48
54 60
NOTE: Even with all 16 members, there are still times when
observed volume does NOT fall within range of predictions --not enough spread (indicated with red bar)
0.8
0.6
Mix
Mix-PM
Rain Volume Error (km**3)
0.4
0.2
0
1
-0.2
2
3
4
5
6
7
8
9
24
30
36
42
48
54
10
Mix
IC/LBC
-0.4
Mix
IC/LBC-PM
IC/LBC
IC/LBC
-0.6
06
12
18
-0.8
Time
60
Percentage of times the observed value fell within the min/max of the ensemble
70
Successful Forecasts (%)
60
50
40
30
Rate Mix
VolumeMix
VolumeIC/LBC
RateIC/LBC
20
IC/LBC
10
Areal
Coverage
Mix
0
1
2
Day
3
2.5
Skill (MAE) as a function of spread (> 1.5*SD cases vs < .5*SD cases)
CRA applied to Mix Ensemble (IC/LBC similar)
2
Area/1000 big SD
1.5
Vol big SD
1
Rate*10
low SD
0.5 Rate*10
big SD
Vol low SD
Area/1000 low SD
0
06
1
12
2
18
3
24
4
30
5
36
6
Time
427
48
8
54
9
60
10
• It thus appears that total system rain
volume and total system areal coverage
of rainfall show a clear signal for better
skill when spread is smaller
• Rain rate does not show such a clear
signal – (especially when 4 bins of SDs
are examined).
• Perhaps average rain rate for systems
is not as big a problem in the forecasts
as areal coverage (and thus volume)?
Seems to be ~5-10% error for rate, 1020% for volume, 10-20% for area
Summary
• Ensemble spread behavior for objectoriented parameters may not behave
like traditional ensemble measures
• Some similarities but some differences
also in output from CRA vs MODE
• Some suggestion that ensembles may
give useful information on probability
of systems having a particular size,
intensity, volume
Acknowledgments
• Thanks to Eric (and others?) for
organizing the workshop
• Thanks to John Halley-Gotway and
Randy Bullock for help with MODE runs
for ensemble work, and Adam Clark for
precip forecast output
• Partial support for the work was
provided by NSF Grant ATM-0537043
Gulf near-boundary system with increased rainfall