Transcript Comments on the DELTA tool statistics relating to local

Comments on the DELTA tool statistics
relating to local and urban models
David Carruthers & Jenny Stocker
FAIRMODE 4th Plenary and Working Group meeting
14th -16th June 2011
Swedish Meteorological and Hydrological Institute
Norrköping, Sweden
Contents
•
•
•
•
•
What do we mean by ‘local’ and ‘urban’ models?
Challenges in predicting local concentrations
Influence of ‘background’ concentrations
Influence of averaging times
Comments on the NO2 statistics used by the
DELTA tool
• Comment on data capture
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
What do we mean by ‘local’ and ‘urban’ models?
Hourly estimates of
‘background’ concentrations
PM 2.5 µg/m³
8.5 - 11.0
11.0 - 13.0
13.0 - 15.0
15.0 - 18.0
18.0 - 20.0
20.0 - 22.0
22.0 - 25.0
25.0 - 33.0
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
What do we mean by ‘local’ and ‘urban’ models?
PM 2.5 µg/m³
8.5 - 11.0
11.0 - 13.0
13.0 - 15.0
15.0 - 18.0
18.0 - 20.0
Clearly there are different challenges for
models that are grid based to those the
resolve spatially to within tens of metres
20.0 - 22.0
22.0 - 25.0
25.0 - 33.0
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Challenges in predicting local concentrations
• Plots show modelled against observed concentrations for
the AQD NO2 24-hourly maximum statistic
• It is complicated to model the local effects – further away
concentrations are made up of elements from different
sources, inaccuracies in emissions etc are smoothed out.
Roadside
Kerbside
400
Urban background
300
200
300
200
Modelled
Modelled
Modelled
200
100
100
100
0
0
0
100
200
Observed
300
400
0
0
100
200
Observed
300
0
50
100
Observed
150
200
Note different scales
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Challenges in predicting local concentrations
• Plots show modelled against observed concentrations for
the AQD NO2 24-hourly maximum statistic
• It is complicated to model the local effects – further away
concentrations are made up of elements from different
sources, inaccuracies in emissions etc are smoothed out.
• Should the performance criteria for local model
predictions be less strict than for the urban and
regional models?
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Influence of ‘background’ concentrations
• Background concentration data taken from ADMS-Urban
model input file (rural measurements) for London, 2008
• Variances are for observed concentrations and relate to
the DELTA tool interpretation of the AQD statistics
Pollutant
Ozone
PM10
NO2
Typical background
concentration (µg/m³)
55
25
9
Typical variance (µg/m³)
20
20
20-100
Magnitude of the variance depends
heavily on the site type i.e. kerbside,
urban background etc
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Influence of ‘background’ concentrations
• The DELTA tool indicates that ADMS performs better for PM10
and Ozone than it does for NO2. Related to the background
concentrations.
300
35
Modelled NOx
concentrations
250
200
Modelled
Background
150
100
50
0
Urban
background
Roadside
Total concentration (µg/m³)
Total concentration (µg/m³)
350
30
Modelled PM10
concentrations
25
20
15
10
5
0
Urban background Roadside
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Influence of ‘background’ concentrations
• The DELTA tool indicates that ADMS performs better for PM10
and Ozone than it does for NO2. Related to the background
concentrations; are there other reasons?
DELTA target
plot for NO2
DELTA target
plot for PM10
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Influence of averaging times
• Models predict the ensemble mean. In reality, pollutant
concentrations are subject to fluctuations in time and space.
400
Example quantilequantile plot for hourly
NO2 concentrations
Models cannot be
expected to predict
the higher end of the
observations,
although an attempt
at predicting the
likelihood of these
values is possible
Modelled
300
200
100
0
0
100
200
300
400
Observed
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Influence of averaging times
• Models predict the ensemble mean. In reality, pollutant
concentrations are subject to fluctuations in time and space.
• Average statistical measures such as the daily average PM10
and 8-hour running mean for Ozone smooth out the effect of
fluctuations, which makes these measures easier to model.
• Another reason for criteria for NO2 be less strict compared
to the other pollutants?
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
DELTA tool criteria and some statistics
Pollutant
Region
|MFB|
(%)
MFE
(%)
FAC2
(%)
R (-)
IOA (-)
TARGET
(-)
RDE &
RPE (%)
Ozone
Regional/
Urban/ Local
30-15
45-30
50-60
0.65-0.78
0.65-0.78
1.0-0.8
50-42
PM10
Regional/
Urban/ Local
60-30
75-50
50-60
0.40-0.48
0.60-0.72
1.0-0.8
50-42
Regional
42-32
50-40
50-60
0.25-0.30
0.50-0.60
Urban
40-32
50-40
50-60
0.25-0.30
0.50-0.60
1.0-0.8
50-42
Local
40-32
50-40
50-60
0.25-0.30
0.50-0.60
NO2
• Target
1 N
2


M

O
O

i
i
N i 1
N = number of values; Oi / Mi observed
/ modelled concentrations;  O =
standard deviation of observations.
• RDE
OLV  M LV LV
LV = Limit value; OLV closest
observation to LV; MLV corresponding
ranked modelled value.
• RPE
OP  M P OP
p = percentile corresponding to the
allowed number of exceedences of the
limit value..
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Target
• The DELTA tool indicates that ADMS performs better for PM10
and Ozone than it does for NO2. But clearly this is related to
the background concentrations.
DELTA target
plot for NO2
DELTA target
plot for PM10
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Comment on NO2 statistics – target
standard deviation: daily max or hourly
30
25
20
Frequency
• Normal distribution does
not fits the observations
well.
• Skewed distribution would
fit the hourly data better
Normal
distribution
15
10
5
1000
0
900
0
800
10
30
40
50
60
70
80
90 100 110 120 130 140 150 160
NO2 daily maximum concentration
700
Frequency
20
600
O
500
400
300
200
100
0
0
10
20
30
40
50
60
70
80
90
100 110 120 130 140 150 160
NO2 hourly concentration
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Comments on the NO2 statistics
RPE and RDE
• The DELTA tool interprets the NO2 EU air quality objective:
‘18 exceedences of 200 µg/m³ allowed per year’ (EUstat –
calculated in Excel)
by analysing statistics relating to the
‘1-hr daily maximum concentration’ (DTstat – calculated in DELTA)
• It seems harder to achieve the criteria and goals using the
latter statistic compared to the former. For example:
Statistic (exact agreement – statistic = 0)
Statistics from
example sites
Target
RPE
RDE
EUstat
DTstat
EUstat
DTstat
EUstat
DTstat
Kerbside
1.1
1.2
0.06
0.05
0.04
0.03
Roadside
1.1
1.2
0.06
0.68
0.19
0.27
Urban
background
1.0
1.3
0.16
0.61
0.53
0.49
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
DELTA tool criteria and some statistics
Pollutant
Region
|MFB|
(%)
MFE
(%)
FAC2
(%)
R (-)
IOA (-)
TARGET
(-)
RDE &
RPE (%)
Ozone
Regional/
Urban/ Local
30-15
45-30
50-60
0.65-0.78
0.65-0.78
1.0-0.8
50-42
PM10
Regional/
Urban/ Local
60-30
75-50
50-60
0.40-0.48
0.60-0.72
1.0-0.8
50-42
Regional
42-32
50-40
50-60
0.25-0.30
0.50-0.60
Urban
40-32
50-40
50-60
0.25-0.30
0.50-0.60
Local
40-32
30?
50-40
50-60
0.25-0.30
0.55?
0.50-0.60
NO2
• Target
1 N
2


M

O
O

i
i
N i 1
1.0-0.8
50-42
N = number of values; Oi / Mi observed
/ modelled concentrations;  O =
standard deviation of observations.
• RDE
OLV  M LV LV
LV = Limit value; OLV closest
observation to LV; MLV corresponding
ranked modelled value.
• RPE
OP  M P OP
p = percentile corresponding to the
allowed number of exceedences of the
limit value..
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Comment on data capture
• In similar inter-model comparison studies within the UK
(quote), minimal percentage data capture criteria have been
applied, as monitors with poor data capture are less likely to be
reliable.
• Should the DELTA tool impose a minimum data capture
criteria of, for example, 75%?
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Any questions?
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Influence of ‘background’ concentrations
• The DELTA tool indicates that ADMS performs better for PM10
and Ozone than it does for NO2.
• If we believe the input data for different pollutants to be of
similar accuracy, we would expect the model output to be of
similar order of accuracy (neglecting chemistry).
• Why is the target for NO2 predictions so much harder to
achieve for ADMS?
– Is it related to ADMS being a local model?
– Or is the NO2 statistic harder to achieve for all models?
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011
Comments on the NO2 statistic
• The DELTA tool interprets the NO2 EU air quality objective:
‘18 exceedences of 200 µg/m³ allowed per year’ (EUstat –
calculated in Excel)
by analysing statistics relating to the
‘1-hr daily maximum concentration’ (DTstat – calculated in DELTA)
• It seems harder to achieve the criteria and goals using the
latter statistic compared to the former.
• Is there any way of using a stricter interpretation of the EU
air quality directive in the tool?
FAIRMODE 4th Plenary and Working Group meeting, 14th -16th June 2011