#### Transcript Verification of Spatial Forecasts - ISAC

Verifying Satellite Precipitation Estimates for Weather and Hydrological Applications Beth Ebert Bureau of Meteorology Research Centre Melbourne, Australia 1st IPWG Workshop, 23-27 September 2002, Madrid (val' dat' ) tr.v. 1. To declare or make legally valid. 2. To mark with an indication of official sanction. 3. To substantiate; verify. val.i.date -e (ver' fi') tr.v. 1. To prove the truth of by -e ver.i.fy the presentation of evidence or testimony; substantiate. 2. To determine or test the truth or accuracy of, as by comparison, investigation, or reference: "Findings are not accepted by scientists unless they can be verified" (Norman L. Munn) The American Heritage Dictionary of the English Language. William Morris, editor, Houghton Mifflin, Boston, 1969. Satellite precipitation estimates -what do we especially want to get right? Climatologists - mean bias NWP data assimilation (physical initialization) - rain location and type Hydrologists - rain volume Forecasters and emergency managers rain location and maximum intensity Everyone needs error estimates! Short-term precipitation estimates • High spatial and temporal resolution desirable • Dynamic range required • Motion may be important for nowcasts • Can live with some bias in the estimates if it's not too great • Verification data need not be quite as accurate as for climate verification • Land-based rainfall generally of greater interest than ocean-based Some truths about "truth" data • No existing measurement system adequately captures the high spatial and temporal variability of rainfall. • Errors in validation data artificially inflate errors in satellite precipitation estimates Rain gauge observations Advantages True rain measurements Disadvantages May be unrepresentative of aerial value Verification results biased toward regions with high gauge density Most obs made once daily Radar data Advantages Excellent spatial and temporal resolution Disadvantages Beamfilling, attenuation, overshoot, clutter, etc. Limited spatial extent TRMM PR Rain gauge analyses Advantages Grid-scale quantities Overcomes uneven distribution of rain gauges Disadvantages Smoothes actual rainfall values Stream flow measurements Advantages Integrates rainfall over a catchment Many accurate measurements available Hydrologists want it Disadvantages Depends on soil conditions, hydrological model Time delay between rain and outflow Blurs spatial distribution observed Discharge (m3/hr) estimated time Verification strategy for satellite precipitation estimates Use (gauge-corrected) radar data for local instantaneous or very short-term estimates Use gauge or radar-gauge analysis for larger spatial and/or temporal estimates Focus on methods, not results • What scores and methods can we use to verify precipitation estimates? • What do they tell us about the quality of precipitation estimates? • What are some of the advantages and disadvantages of these methods? • Will focus on spatial verification Does the satellite estimate look right? • Is the rain in the correct place? • Does it have the correct mean value? • Does it have the correct maximum value? • Does it have the correct size? • Does it have the correct shape? • Does it have the correct spatial variability? Spatial verification methods • • • • Visual ("eyeball") verification Continuous statistics Categorical statistics Joint distributions "standard" -------------------------------------------------------------- "scientific" or "diagnostic" • Scale decomposition methods • Entity-based methods Step 1: Visual ("eyeball") verification Visually compare maps of satellite estimates and observations Advantage: "A picture tells a thousand words…" Disadvantages: Labor intensive, not quantitative, subjective Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Rozumalski, 2000 Continuous verification statistics Measure the correspondence between the values of the estimates and observations Examples: • mean error (bias) • mean absolute error • root mean squared error • skill score • linear error in probability space (LEPS) • correlation coefficient Advantages: Simple, familiar Disadvantage: Not very revealing as to what's going wrong in the forecast Mean error (bias) 1 N Mean Error ( Fi Oi ) N i 1 Measures: Average difference between forecast and observed values Mean absolute error 1 N MAE | Fi Oi | N i 1 Measures: Average magnitude of forecast error Root mean square error RMSE 1 N N 2 ( F O ) i i i 1 Measures: Error magnitude, with large errors having a greater impact than in the MAE Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Time series of error statistics 24-hr rainfall from NRL Experimental Geostationary algorithm validated against Australian operational daily rain gauge analysis 0.25° grid boxes, tropics only Linear error in probability space (LEPS) 1 N LEPS | CDFo ( Fi ) CDFo (Oi ) | N i 1 Measures: Probability error - does not penalise going out on a limb when it is justified. error { Cumulative probability of observations CDFo Oi Fi Value Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Correlation coefficient r ( F F ) (O O ) ( F F ) (O O ) 2 2 Measures: Correspondence between estimated spatial distribution and observed spatial distribution, independent of mean bias Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Danger... Rozumalski, 2000 AutoEstimator validated against Stage III 8x8 km grid boxes Skill score Skill score scoreestimate scorereference scoreperfect scorereference Measures: Improvement over a reference estimate. When MSE is the score used in the above expression then the resulting statistic is called the reduction of variance. The reference estimate is usually one of the following (a) random chance (b) climatology (c) persistence but it could be another estimation algorithm. Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Cross-validation - useful when observations are included in the estimates score score(Yi* , Oi , i 1,...,N ) where Yi* is the estimate at point i computed with Oi excluded from the analysis Measures: Expected accuracy at the scale of the observations. The score is usually bias, MAE, RMS, correlation, etc. Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Categorical statistics Measure the correspondence between estimated and observed occurrence of events Examples: Advantages: Simple, • bias score familiar • probability of detection • false alarm ratio Disadvantage: Not • threat score very revealing • equitable threat score • odds ratio • Hanssen and Kuipers score • Heidke skill score Categorical statistics Correct negatives Estimated yes no False alarms Observed Misses Hits Estimated Observed yes hits misses no false alarms correct negatives Bias score BIAS hits false alarms hits misses Measures: Ratio of estimated area (frequency) to observed area (frequency) Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability hits hits misses Probability of Detection POD False Alarm Ratio false alarm s FAR hits false alarm s Threat score (critical success index) TS CSI hits hits m isses false alarm s Equitable threat score ETS hits hitsrandom hits m isses false alarm s hitsrandom Odds ratio hits * correct negatives OR m isses * false alarm s Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Hanssen and Kuipers discriminant (true skill statistic) hits false alarm s HK hits m isses false alarm s correct negatives Measures: Ability of the estimation method to separate the "yes" cases from the "no" cases. Heidke skill score (hits correct negatives) correct random N correct random 1 correct random (Obs yes * Est yes ) (Obsno * Estno ) N HSS Measures: Fraction of correct yes/no detections after eliminating those which would be correct due purely to random chance this attribute? Verifies Location Size Shape Mean value Maximum value Spatial variability Categorical verification of daily satellite precipitation estimates from GPCP 1DD algorithm during summer 2000-01 over Australia North (tropics) Southeast (mid-latitudes) Rain threshold varies from light to heavy Real-time verification example 24-hr rainfall from NRL Experimental Geostationary algorithm Real-time verification example 24-hr rainfall from NRL Experimental blended microwave algorithm Distributions oriented view Estimated category Observed category 1 2 … K total 1 n11 n12 … n1K No1 2 n21 n22 … n2K No2 … … … … … … K nK1 nK2 … nKK NoK total Ne1 Ne2 … NeK N Advantage: Much more complete picture of forecast performance Disadvantage: Lots of numbers Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability OBSERVED (mm/d) 24-hr rainfall from NRL Experimental Geostationary algorithm validated against Australian operational daily rain gauge analysis on 21 Jan 2002 0.0 | 0.1 | 0.2 | 0.5 | 1 | 2 | 5 | 10 | 20 | 50 | 100 | 200 total PREDICTED (mm/d) .0--.1--.2--.5---1---2---5--10--20--50--100--200 total 4134 130 267 136 111 83 28 23 18 6 0 4936 206 25 45 42 30 15 3 4 4 1 0 375 281 17 52 36 25 29 12 6 3 3 0 464 260 6 34 17 17 31 16 20 6 3 1 411 229 13 41 28 28 61 20 26 29 4 1 480 259 22 77 50 51 55 53 43 38 6 1 655 182 21 59 37 48 76 66 68 80 15 0 652 104 21 27 47 54 106 112 127 134 27 5 764 42 6 19 13 41 96 125 158 325 127 9 961 7 1 0 0 0 1 7 8 46 45 13 128 0 0 0 0 0 0 0 0 0 8 1 9 5704 262 621 406 405 553 442 483 683 245 31 9835 Scatterplot Shows: Joint distribution of estimated and observed values NRL geo 20020121 R=0.63 Probability distribution function Shows: Marginal distributions of estimated and observed values NRL geo 20020121 geo anal Heidke skill score (K distinct categories) HSS K K k 1 k 1 P ( Fk ,Ok ) P ( Fk ) P ( Ok ) 1 K P ( Fk ) P ( Ok ) k 1 Measures: Skill of the estimation method in predicting the correct category, relative to that of random chance Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Scale decomposition methods Measure the correspondence between the estimates and observations at different spatial scales Examples: • 2D Fourier decomposition • wavelet decomposition • upscaling Advantages: Scales on which largest errors occur can be isolated, can filter noisy data Disadvantages: Less intuitive, can be mathematically tricky Discrete wavelet transforms Concept: Decompose fields into scales representing different detail levels. Test whether the forecast resembles the observations at each scale. Measures, for each scale: • % of total MSE • linear correlation • RMSE • categorical verification scores • others... Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Casati and Stephenson (2002) technique Step 1: "Recalibrate" forecast using histogram matching errortotal = errorbias + errorrecalibrated Step 2: Threshold the observations and recalibrated forecast to get binary images Step 3: Subtract to get error (difference) image Step 4: Discrete wavelet decomposition of error to scales of resolution x 2n Step 5: Compute verification statistics on error field at discrete scales. Repeat for different rain thresholds. Odds ratio Multiscale statistical organization Zepeda-Arce et al. (J. Geophys. Res., 2000) Concept: Observed precipitation patterns have multiscale spatial and spatio-temporal organization. Test whether the satellite estimate reproduces this organization. Method: Start with fine scale, average to coarser scale Measures: • TS vs. scale • depth vs. area • spatial scaling parameter • dynamic scaling exponent Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability fcst obs + fcst obs Std. dev. + + Depth (mm) Threat score + fcst * * * * obs ++ ++ Scale (km) Area (km2) Scale (km) Upscaling verification of IR power law rainrate 16 September 2002, Melbourne mm hr-1 IR IR radar radar GMSRA validated against rain gauge analyses at different spatial scales (Ba and Gruber, 2001) Entity-based methods Use pattern matching to associate forecast and observed entities ("blobs"). Verify the properties of the entities. Examples: • CRA (contiguous rain area) verification Advantages: Intuitive, quantifies "eyeball" verification Disadvantage: May fail if forecast does not sufficiently resemble observations Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability CRA (entity) verification Ebert and McBride (J. Hydrology, Dec 2000) Concept: Verify the properties of the forecast (estimated) entities against observed entities Method: Pattern matching to determine location error, error decomposition, event verification Measures: • location error • size error • error in mean, max values • pattern error Verifies this attribute? Location Size Shape Mean value Maximum value Spatial variability Determine the location error using pattern matching: • Horizontally translate the estimated blob until the total squared error between the estimate and the observations is minimized in the shaded region. Other possibilities: maximum correlation, maximum overlap • The displacement is the vector difference between the original and final locations of the estimate. Observed Estimated CRA error decomposition The total mean squared error (MSE) can be written as: MSEtotal = MSEdisplacement + MSEvolume + MSEpattern The difference between the mean square error before and after translation is the contribution to total error due to displacement, MSEdisplacement = MSEtotal – MSEshifted The error component due to volume represents the bias in mean intensity, MSEvolume (F X )2 where F and X are the CRA mean estimated and observed values after the shift. The pattern error accounts for differences in the fine structure of the estimated and observed fields, MSEpattern = MSEshifted - MSEvolume 24-hr rainfall from NRL Experimental Geostationary algorithm validated against Australian operational daily rain gauge analysis Diagnosis of systematic errors NRL Experimental Geostationary algorithm 289 CRAs April 2001March 2002 Displacement (km) Diagnosis of systematic errors Estimate Analyzed NRL Experimental Geostationary algorithm 289 CRAs April 2001March 2002 Tropical Rain Potential (TRaP) verification? TRaP 24 h rain from 20001208_16 Which methods verify which attributes? Visual (“eyeball”) Location Size Shape Mean value Maximum value Spatial variability Continuous statistics Categorical statistics Joint distribution Scale decomposition Entitybased Conclusions • The most effective diagnostic verification method is still visual ("eyeball") verification. • Categorical statistics based on yes-no discrimination are probably the least informative of all of the verification methods, although they remain very useful for quantitative algorithm intercomparison. • The newer diagnostic verification methods (scale decomposition, entity-based) give a more complete and informative diagnosis of algorithm performance • Need methods to deal with observational uncertainty http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.shtml __________________ UNDER CONSTRUCTION __________________