Transcript Textor_060314Ispra

GEMS
Global Earth-system Monitoring
using Space and in-situ data
GEMS– Overview


Atmospheric Composition and Dynamics
Build an operational thoroughly-validated assimilation
system for atmospheric composition and dynamics,
by 2008.







Daily global monitoring of dynamics & composition
Improvements in daily regional air quality forecasts
Monthly / seasonal estimates of surface fluxes
for CO2 and other species
Extended reanalyses of composition & dynamics
Integrated Project co-funded by
European Commission, 6th FP
GMES (EC&ESA) Atmosphere theme
31 consortium members
4 years (started in March 2005)
Goals of GEMS:
Global Earth-system Monitoring using
Space and in-situ data
 Coordinator
 Greenhouse
 Reactive
Gases P.Rayner (LSCE)
Gases
 Aerosols
 Regional
A.Hollingsworth (ECMWF)
M.Schultz (Juelich)
O.Boucher (MetOff)
Air Quality V-H.Peuch (Meteo.Fr)
 Validation
 Production
H.Eskes (KNMI)
System A.Simmons (ECMWF)
Link between main elements of GEMS
Model inter-comparisons in GEMS

GHG: 2 models
IFS (ECMWF), LMDzT (LSCE)

GRG: 3 models
MOZART-3 (MPI-M), TM5 (KNMI), MOCAGE
(MeteoFr)

AER: 1 model
IFS (-> AeroCom)

RAQ: 10 models
MOCAGE (MeteoFr), BOLCHEM (CNR-ISAC),
EURAD (FRIUUK), CHIMERE (CNRS), SILAM
(FMI), MATCH (FMI), CAC (DMI), MM5-UAM-V
(NKUA), EMEP (MetNo), REMO (MPI-M),
UMAQ-UKCA (UKMO)
RAQ: Ensemble forecasts
Analysis:
Centralized vs Decentralized





Same analysis
applied to all models
Communication
platform
A lot of work for
analyzing team
Progress of work
guaranteed
Large storage
facilities needed





Increased
implication of
individual groups
Duplication of work
Distribution by topic
Progress depends on
many people
Large data transfer
Observational data sets
Methodologies
Subject of Comparison?
 Fields / Fluxes / Processes
What to compare?
 Continuous behavior
 Categorical behavior (Threshold exceedance)
 Averaging in time & space
Limited area / time verification
Which method to use?
 “Eyeball” methods
 Basis statistical evaluation
 Sophisticated skill scores
Topics to think about
Influence of model resolution
 Interpolation techniques
 Reference state (e.g. Observation,
Climatology, Persistence, Median)
 Errors of reference state / observations
 Representativity of stations

Mixing of model skills
 Maintenance of data base

Eyeball Methods
Comparison of time series at a given location
P. Agnew
(Educated) Eyeball Methods
Comparison of fields at a given time (period)
Plots taken from talk of Adrian Simmons at the GEMS Annual Assembly, Feb. 2006
Basis Statistic Evaluation
HERBS (M. Chin)

How well does the distribution of model results corresponds to
the distribution of observed quantities?


What is the average error of the model compared to the
observations?


Correlation Coefficient R
What is the model bias?


Mean error E
How well do the model calculated values correspond to the
observed values?


Histogram H
Mean bias B
What is the overall model skill?

Skill score S
M. Chin
Basic statistical evaluation









(Rank) Correlation coefficient between observations
and reference state
Slope and offset in scatter plots
(Normalized) Root-mean square errors
Bias (absolute and relative to reference values)
RMSE (absolute and relative to reference values)
Variability ratio (i.e. standard deviation of modelled
values versus standard deviation of refecence values)
Contingency tables defined with respect to thresholds
Histograms of - absolute and relative - errors
…
P. Agnew
B 'n 
2  fi  oi 


N i  fi  oi 
Basic statistical evaluation (RAQ)
(continous behaviour)

measure of overall forecast error
fractional gross error
normalized RMSE not used:
• errors not symmetric,
• overweighting larger errors due to squaring
P. Agnew
B 'n 
2  fi  oi 


N i  fi  oi 


Basic statistical evaluation (RAQ)
(continous behaviour)
extent of over/under prediction
Modified mean bias: symmetric around 0, -1 -> 1,
degree of pattern match: Correlation Coefficient
no offset
P. Agnew
Taylor Diagramme



condense info of spatio-temporal varying fields
Use geometric relation between RMS – STDDEV – CORRELATION
Graphic display of model skill (RMS or others)
Reference
M. Schulz
Taylor skill scores



Skill score should
 increase monotonically with correlation
 increase with match of modeled and observed variance
 vary between 0-1
S1 = 4(1+R) / [(f +1/ f )2(1+R0)]
S2 = 4(1+R)4 / [(f +1/ f )2(1+R0)4] (+ penalty for low corr.)
Where R0=max attainable R, f =std_dev (model)/std_dev (data)
Categorical Skill Scores
 Definition
of an event or a threshold
 Number of a certain event (‘hit’)
 Basis: 2x2 contingency table
O Yes
O No
F Yes
Hits
a
False Alarms
F No
Misses
c
a+c
b
Correct
Rejections
d
b+d
a+b
c+d
a+b+c+d=n
P. Agnew
Model forecast
Radar
X>u
X<u
Y>u
Hits
a
False Alarms
Y<u
Misses
c
a+c
Radar > 1 mm
b
Correct
Rejections
d
b+d
a+b
c+d
a+b+c+d=n
Forecast > 1 mm
Source: Marion Mittermaier, derived from Casati (2004)
Binary error image
P. Agnew
Categorial Skill Scores:
Odds Ratio (Stephensen, 2000)
 ‘Odds
Ratio’ defined as
ratio of probability that event occurs to
probability that event does not occur
 Easily calculated from contingency table
 Significance testing possible
P. Agnew
How to compare ?
M. Sofiev
Evaluation tools used/discussed
within GEMS
MetPy (ECMWF)
 MMAS (FMI)
 AeroCom (LSCE)
 several other tools at partner institutes


CDO, MetView?, CDAT, nco,…
MetPy







gridded data – gridded data
(gridded data – station data)
(station data – stat ion data)
Python-based scripts
user-friendly front end (“Verify”)
“all” formats which Python supports
to be run in batch mode
designed for operational use
additional visualization tool required
C. Gibert et al., ECMWF
MetPy
compute(
param = Z,
levtype = pl,
levelist = (1000,500,100),
score = (ancf,ref),
steps = StepSequence(12,240,12),
area = (‘europe’, ‘north hemisphere’),
forecast = forecast (
)
persistence = persistence(
)
analysis = analysis (
expver = ‘0001’,
date = DateSequence(20040101,20040131),
)
)
C. Gibert
Model and Measurement Analysis
Software (MMAS)





Point data sets, NO MAPS
station data – station data (ASCII)
easy menu-driven for individual use
to be run in Microsoft Windows
environments
output: ASCII & GraDS bin
additional visualization tool needed
M.Sofiev
Finnish Meteorological Institute
M. Sofiev
MMAS strategy
Input data sets
Output data set
Statistical
characteristics
Measurements
Merged Model/Measurements
~~~~
~~~~
~~~~
~~~~
Model
Binary files
with mapped
statistics



merges two arbitrary time-dependent data sets
computes statistics/skill scores for the merged sets
presents the results in numerical and graphic-ready format
M. Sofiev
MANY THANKS TO
Paul Agnew
Olivier Boucher
Mian Chin
Fadoua Eddounia
Hendrik Elbern
Claude Gibert
Kathy Law
Dimitris Melas
Martin Schultz
Michael Schulz
Mikhael Sofiev
Leonor Tarrason
Odds Ratio Skill Score
A
skill score can be derived by a simple
transformation:
ORSS=(OR-1)/(OR+1)
This mapping produces a skill score in
the range -1 to +1
When ORSS=-1 forecasts and
observations are independent
Providing
number of forecasts is
statistically significant, ORSS
approaching +1 indicates a skillful
forecast

- different approaches around to do the data
handling
- software tools
-regridding
-visualisation
-maximizing the use of 'ensemble' data versus
individual models
-involvement of participants.
-dissemination of data
-typical problems encountered during
intercomparison and how to avoid them.
- whatever you think is important to share
with your collegues along this concept.
GEMS Research and Operational Goals
Build an operational thoroughly-validated assimilation
system for atmospheric composition and dynamics, by 2008.

Delivering





Using




Daily global monitoring of dynamics & composition
Improvements in daily regional air quality forecasts
Monthly / seasonal estimates of surface fluxes
for CO2 and other species
Extended reanalyses of composition & dynamics for validation,
and in support of GCOS
Best available models, assimilation systems
Best available in-situ data
Best available satellite data and algorithms
Collaborating
with EU-IPs MERSEA & GEOLAND to implement IGOS_P Themes on

Carbon Cycle

Atmospheric Chemistry
T. Hollingsworth
GEMS– Overview






Atmospheric Composition and Dynamics
Build an operational thoroughly-validated assimilation
system for atmospheric composition and dynamics,
by 2008.
Integrated Project co-funded by
European Commission, 6th FP
GMES (EC&ESA) Atmosphere theme
17 M€ budget, 12.5 M€ EC-contribution
31 consortium members
4 years (started in March 2005)
T. Hollingsworth