Transcript Methods to estimate uncertainties - European Topic Centre for Air

Methods to estimate uncertainties
EU Workshop on uncertainties in greenhouse gas inventories
5 to 6th September 2005, Helsinki, Finland.
John Watterson1, Justin Goodwin1, Melissa Downes1, Alistair Manning2, and
Lorna Brown3
With thanks to John Abbott1 and Neil Passant1
1National
Environmental Technology Centre - Netcen - Harwell Science Park, Didcot, Oxfordshire, OX11 0QJ, UK.
Met Office, FitzRoy Road, Exeter, Devon, EX1 3PB, UK.
3Institute of Grassland and Environmental Research (IGER), North Wyke Research Station, Okehampton, Devon, EX20 2SB, UK.
2The
What is in this presentation
Topics








Overview of methods and guidance – initial thoughts
Estimation of uncertainties in activity data (AD)
Use of IPCC default uncertainties in national inventories
Estimation of uncertainty in national emission factors (EFs)
Verification of emission data: How can comparison of different
models/methods be used to estimate uncertainties?
Estimation of uncertainties in models
Combining uncertainties
Treatment of correlations
Some general problems with
uncertainty analysis

Strictly uncertainties in inventories
cannot be exactly quantified
 Unknown sources
 Gaps in understanding of existing sources
 Measurement for emission factors are
inadequate to quantify uncertainties
 Emission factors may be inappropriate for
specific sources
 Expert elicitation has a role – workshops later
this afternoon
However


We need to understand the likely
magnitude of uncertainties and their
impacts
But there is hope!!
 We do have some knowledge and
understanding of uncertainties
 We need to identify major uncertainties to
direct improvements in GHG inventories
Overview of methods and
guidance

‘Approach 1’
 emission sources aggregated up to level similar to
IPCC Summary Table 7A
 uncertainties then estimated for these categories
 uncertainties calculated based on error propagation
equations
 Provides basis for Key Source analysis

‘Approach 2’
 corresponds to Monte Carlo approach
 Can use software such as @RISK and MS excel
spreadsheets – or write your own MC code

Recommend reading the 2006 IPCC Guidelines –
Volume 1 Chapter 3 “Uncertainties”
Estimation of uncertainties in
Activity Data
Estimation of uncertainties in
activity data (AD) - examples
Digest of UK Energy
Statistics
(UK Department for
Trade and Industry)
Energy
Energy statistics for the
UK (imports, exports,
production,
consumption, demand)
of liquid, solid and
gaseous fuels
Calorific values of fuels
and conversion factors
Agriculture
UK Defra - Institute of
Grassland and
Environmental
Research (IGER)
Estimation of uncertainties in
activity data (AD)
Pollution
Inventory(Environment
Agency)
Scottish Environmental
Protection Agency
United Kingdom Petroleum
Industry Association
United Kingdom Offshore
Operators Association
Iron and Steel Statistics
Bureau
Etc.
Industrial
Processes
Uncertainties in UK fuel activity
data




Fuel activity data taken from Digest of UK Energy
Statistics
Uncertainties used for the fuel activity data
estimated from the statistical difference between
supply and demand for each fuel
Effectively the residuals when a mass balance is
performed on the production, imports, exports
and consumption of fuels
For solid and liquid fuels both positive and
negative results are obtained indicating that these
are uncertainties rather than losses
Uncertainties in UK fuel activity
data


Quoted uncertainty refers to the total fuel
consumption rather than the consumption
by a particular sector, e.g. residential coal
To avoid underestimating uncertainties, it
was necessary to correlate the
uncertainties used for the same fuel in
different sectors
Uncertainties in UK fuel activity
data


For gaseous fuels uncertainties include losses and
tended to be negative. For natural gas, a
correction was made to take account of leakage
from the gas transmission system but for other
gases this was not possible.
The uncertainties in activity data for minor fuels
(colliery methane, orimulsion, SSF, petroleum
coke) and non-fuels (limestone, dolomite and
clinker) were estimated based on judgement
comparing their relative uncertainty with that of
the known fuels.
Time series of variability in the
supply and demand of coal
Difference in supply and demand
of coal
Difference between supply and demand of coal
Trend suggests
improvement in accuracy of
estimates of supply and
demand over time?
2000
Difference (k tonnes)
1500
1000
500
0
1980
-500
Coal
1985
1990
1995
-1000
-1500
-2000
Year
2000
2005
Difference in supply and demand
of natural gas
Difference between supply and demand of gas
30000
Difference (GWh)
25000
20000
15000
10000
5000
0
1984
Supply greater than demand
– is this all due to losses
(fugitive emissions) in the
gas transmission system?
Could apply a correction if
estimated fugitive emissions
are known
Gas
Decline in difference reflects
measures implemented in
the UK to reduce fugitive
emissions in the gas
transmission system
1986
1988
1990
1992
1994
Year
1996
1998
2000
2002
2004
Uncertainty in coal activity data
Time series of uncertainties as 95% Confidence Intervals expressed as a
percentage of the central estimate for UK coal supply and demand
2.5%
2 SD / mean
2.0%
1.5%
Yearly values
Rolling 5 year mean
1.0%
0.5%
0.0%
1980
1985
1990
1995
Year
2000
2005
Some general comments on using
statistical differences derived from energy
balance data


Uncertainties in the fuel combustion data for
specific sectors or applications, are probably
higher than the uncertainty suggested by the
statistical difference between supply and demand
Warning - if a statistical difference is zero it is
likely that the data are of uncertain quality and
this does not imply zero uncertainty. In these
instances, the data quality should be examined for
QA/QC purposes and the relevant statistical
agencies should investigate
Use of IPCC default uncertainties
in national inventories
Using IPCC default uncertainties

Where possible, uncertainty data for EFs should
be derived from published country specific studies
 estimate values of uncertainties
 you may be able to derive the PDF from available
data

If such data are unavailable, then use default
values from guidelines
 suggest it will be best to refer to the 2006 IPCC
guidelines which should be available in early 2006
 unless you have other evidence, assume PDF
normal
 before using defaults, try using expert judgement /
elicitation to produce more applicable data
Typical data available in 2006
IPCC guidelines
TABLE 2.12 DEFAULT UNCERTAINTY ESTIMATES FOR STATIONARY COMBUSTION EMISSION FACTORS
Sector
CH4
N2O
Public Power, co-generation and district heating
Commercial, Institutional & Residential combustion
Industrial combustion
50-150%
50-150%
50-150%
Order of magnitude*
Order of magnitude
Order of magnitude
*i.e. having an uncertainty range from one-tenth of the mean value to ten times the mean value.
Source: IPCC Good Practice Guidance and Uncertainty Management in National Greenhouse Gas Inventories(2000)
 GLs suggest an overall uncertainty value of 7 % for the CO2
emission factors of Energy

Data above taken from Stationary Combustion
Chapter in 2006 Guidelines
 Derived from EMEP/CORINAIR Guidebook
 Very limited sector specific information and wide
range of uncertainty quoted
Using uncertainties in the IPCC
guidelines
e.g. Uncertainties in CH4 emissions from Table 2.12 in
previous slide
Probability Distribution
50
100
150
2 = 50
95% CI
= 2 / E
= 50 / 100
= 50 %
 = 1 standard deviation
of the mean, E
Using uncertainties in the IPCC
guidelines
e.g. uncertainties in N2O emissions from Table 2.12
These are order of magnitude uncertainties – need
to use Approach 2 (i.e. MC simulation) and define a
suitable PDF
Estimation of uncertainty in
national emission factors (EFs)
Example from data to support UK review of
carbon emission factors (CEF)
Large
number of
samples
used to
estimate CEF
Checks to
see if a
weighted
mean
approach
produces a
more
accurate CEF
estimate
Uncertainty values can be used
directly from this report
Verification of emission data:
how can comparison of
different models/methods be
used to estimate uncertainties?
Initial considerations

One a very basic level comparisons using different
models/methods can be used to assess
uncertainties by
 (a) The closeness of the estimates gives a feel for
potential gross errors. It depends on how
independent the methods are and the potential
errors in each method - both estimation and
modeling approaches could have problems, but for
different reasons.
 (b) By comparison across a wide number of
pollutants a qualitative feel for the uncertainty for
any particular pollutant can be gauged.
Verification of the UK GHG
inventory



The approach uses the Lagrangian dispersion
model NAME (Numerical Atmospheric dispersion
Modelling Environment)
Sorts the observations made at Mace Head into
those that represent Northern Hemisphere
baseline air masses and those that represent
regionally-polluted air masses arriving from
Europe. The Mace Head observations and the
hourly air origin maps are applied in an inversion
algorithm to estimate the magnitude and spatial
distribution of the European emissions that best
support the observations
The technique has been applied to 2-yearly rolling
subsets of the data and used to estimate longer
term averages
Verification of the UK GHG
inventory


The inversion (best-fit) technique, simulated
annealing, is used to fit the model emissions to
the observations.
It assumes that the emissions from each grid box
are uniform in both time and space over the
duration of the data. This in turn implies that the
release
 is assumed independent of meteorological factors
such as temperature and diurnal or annual cycles,
and
 that, in so far as the emission relates to industrial
production or other anthropogenic activity, use
there is no definite cycle or intermittency.

The estimated releases will include any natural
release as well as anthropogenic emissions.
Baseline analysis


Based on meteorological
analyses
NAME model derived air origin
maps
 Darker shade – Greater
contribution from area


All possible surface sources
over previous 10 days
Maps generated for each hour
1995-2004
Mace
Head
Inverse modelling





Aim: generate emission estimates from ‘polluted’ observations (above baseline)
Use NAME to predict concentration time series at Mace Head from each source
Scale emissions to obtain best match between model and observations
 Simulated Annealing
 Iterative technique
 No prior information
Apply to all monitored species
Independent verification of emissions
Equation: A e = m
Minimise: m - A e
A: the dilution matrix
m: observed concentrations
(- baseline)
e: emissions
NAME model predictions of
emissions of N2O across Europe
Nitrous oxide – comparison of GHG
inventory estimates and model estimates
200
GHGI
NAME - inc part North Sea
NAME
Nitrous Oxide Emissions (kt/yr)
180
160
Thermal
oxidiser
abatement
system
fitted to
adipic acid
plant
140
120
100
80
60
40
20
0
1995-96
1996-97
1997-98
1998-99
1999-00
2000-01
2001-02
2002-03
2003-04
Quality of agreement between
UK GHGI estimates and model




Reasonable agreement between modelled
and measured which gives confidence of
the inventory estimates
But fitment of abatement to adipic acid
plant not reflected in NAME model trend
This problem investigated with
representatives from the adipic acid plant
and the meteorological office
Where was the problem – GHG inventory
or model?
Answer – probably mostly the model,
but check the GHG inventory also

The NAME model assumes no definite cycle or
intermittency in emissions – this was not the case
– periods were the adipic acid plant was shut
down and periods where abatement not operating
 So, make adjustments to the model

The oxidised nitrogen from wastewater is not
currently included in the GHG – this (small)
source could be added to improve the accuracy of
the N2O estimate
 So, make checks on the inventory
Estimation of uncertainties in
models
Initial considerations


Model is a representation of a ‘real world’
system – but can never exactly mimic the
‘real world’
Key considerations in model uncertainty
 Has the correct ‘real world’ been identified –
for example, the ‘real world’ in a GHG
inventory would be a complete and unbiased
inventory
 Is the model an accurate representation of
this ‘real world’?
Example using N2O from agriculture
in the UK GHG inventory
Recent detailed study into the uncertainty of the
model used to estimate emissions from the UK
GHG inventory
An inventory of nitrous oxide emissions from agriculture
using the IPCC methodology: emission estimate, uncertainty
and sensitivity analysis (2001). Brown, L., Amstrong Brown,
S., Jarvis, S.C., Syed, B., Goulding, K.W.T., Philips, V.R.,
Sneath, R.W. and Pain, B.F. Atmos Environ., 35, 1439-1449.
Approach

Monte Carlo approach used to estimate
the uncertainty in the model
 26 parameters were included
 For some parameters, a beta pert distribution
used derived from IPCC minima, maxima and
most likely (default) values. No information in
IPCC GLs to suggest alternative distribution.

Sensitivity analysis performed using
multivariate stepwise regression using
@RISK software
Results





N2O emissions from UK agriculture were
estimated to be 87 Gg N2O-N for both 1990 and
1995 using the IPCC default EFs
Total estimate shown to have high overall
uncertainty of 62%
Comparisons of results from this study and other
UK-derived inventories suggests the default IPCC
inventory may overestimate emissions
Uncertainty in individual components determined
This has identified the components of the model
where improvements could be made since
 emissions are a significant fraction of the total and
 the associated uncertainties are high
Uncertainty associated with
parameters
Parameter
Direct sector - soil
% of total N2O
from UK
agriculture
54%
Parameter /
Uncertainty
EF1 (direct emission
from soil)
 31%
EF3PRP (emission from
pasture range and
paddock)
+ 11% to –17%
Indirect sector –
leached N and
deposited
ammonia
29%
 126%
Treatment of correlations
Correlations
When to use a correlation

Activity Data are calculated via mass balance
 Supply and demand of fuels in energy statistics

Emission Factors are shared across activities
 Natural gas or gas/diesel oil used by different sources

Emission Factors are calculated or extrapolated
across a time series
 Methane from livestock
Correlations (ii)
How to use a correlation



Can be used in combination
 Activity and EF’s correlated
If correlations occur, the easiest and most effective
method is to use a Monte Carlo Simulation
NB: Correlations may not have an effect. Will only
affect areas where the inventory is sensitive and/or the
dependencies are very strong
Combining uncertainties
Combining uncertainties

For Tier 2 analysis - a Monte Carlo approach is necessary.
 Uncertainties are set and the correlations marked. The software is
then set up and run and automatically takes these into account.

For component uncertainties <60%, a sum of squares approach
can be used.
 UT =  (UE2 + UA2)

For component uncertainties >60% all that is possible is to
combine limiting values to define an overall range
 U% = (E+A+E*A/100) and L% = (E+A-E*A/100)
U=Uncertainty, T=total, E=Emission Factor, A=Activity Data, U%=upper limit, L%= lower limit
Example Monte Carlo model
Calculating uncertainties
Fuel/Activity Uncertainty
Emission Uncertainty
Emission Factor Uncertainty
Frequency
Probability Distribution
Min
Range
Max
Value
Distribution Types:
normal
Lognormal
Uniform
Triangular
Tier 2 - Monte-Carlo Method
Input 2
10
8
6
4
2
0
S1
1
Model
8
6
4
2
0
S1
Result
10
8
6
4
2
0
S1
Input 1
Tier 2 - Monte-Carlo Method

Step 1: Assess component uncertainties
 Expert Judgement & Data

• Maximum, Minimum
• Distribution type
Step2: Run the analysis
 up to 20,000 iterations
Factors
Min
Activity
Min
Max
Max
Emission

Step 3: Results
 5th - 95th percentile = Range as % of the mean
Comments



Correlations do affect the overall uncertainty
result – suggest approach is to start identifying
inputs that are correlated, rather than setting up
the model with the input level at the lowest level
of aggregation and examining the correlations in
each parameter individually
It is easy to produce MC output that superficially
looks credible – but carefully check underlying
assumptions
You can write a programme to complete a MC
analysis – you do not need to use an expensive
commercial package
Demonstration of MC model

UK has set up MC model to illustrate
certain key points




Suggested layout of a simple MC model
Defining non-correlated PDFs
Considering correlations
How to deal with emissions and associated
uncertainty where individual EF and AD
uncertainties are unknown
 Example output table
Final thoughts
Guidance
- Read the IPCC
guidance
- Consider comments
made by Expert
Reviewers and in
Peer Reviews
Background
reading
first!
Data
- Gather country
specific
information on
EFs and AD
- Use IPCC
defaults only if
sufficient
information
cannot be found
Gather
sufficient
information
Implement
- Careful with
Monte Carlo
analysis –
easy to
produce poor
quality work
Review
- Ask for peer review
- Reflect on output of the
uncertainty analysis – is
it sensible?
- Get the help
of a
statistician
Follow IPPC
Good
Practice
Try to be
open to
criticism!
Acknowledgements


The UK GHG inventory is funded by UK
Defra and the Devolved Administrations
UK Defra




Jim Penman – Head Response Strategies
Susan Donaldson – GHG Science Advisor
Joanne Halliday – GHG Science Advisor
Steve Cornellius - GHG Science Advisor