Transcript Methods for Developing Input Distributions for

Methods and Applications of Uncertainty and
Sensitivity Analysis
H. Christopher Frey, Ph.D.
Professor
Department of Civil, Construction, and Environmental Engineering
North Carolina State University
Raleigh, NC 27695
Prepared for:
Workshop on Climate Change
Washington, DC
March 7, 2005
Outline
• Why are uncertainty and sensitivity analysis
needed?
• Overview of methods for uncertainty analysis
– Model inputs
» Empirical data
» Expert judgment
– Model uncertainty
– Scenario uncertainty
• Overview of methods for sensitivity analysis
• Examples
– Technology assessment
– Emissions Factors and Inventories
– Air Quality Modeling
– Risk Assessment
• Findings
• Recommendations
Why are uncertainty and sensitivity
analysis needed?
• Strategies for answering this question:
–what happens when we ignore uncertainty and
sensitivity?
–what do decision makers want to know that
motivates doing uncertainty and sensitivity
analysis?
–what constitutes best scientific practice?
• Program and research managers may not care
about all three, but might find at least one to
be convincing (and useful)
When is Probabilistic
Analysis Needed or Useful?
• Consequences of poor or biased estimates are unacceptably
high
• A (usually conservative) screening level analysis indicates a
potential concern, but carries a level of uncertainty
• Determining the value of collecting additional information
• Uncertainty stems from multiple sources
• Significant equity issues are associated with variability
• Ranking or prioritizing significance of multiple pathways,
pollutants, sites, etc.
• Cost of remediation or intervention is high
• Scientific credibility is important
• Obligation to indicate what is known and how well it is known
When is a Probabilistic Approach Not
Needed?
• When a (usually conservative) screening level analysis
indicates a negligible problem
• When the cost of intervention is smaller than the cost of
analysis
• When safety is an urgent and/or obvious issue
• When there is little variability or uncertainty
Myths: Barriers to Use of Methods
• Myth: it takes more resources to do uncertainty analysis, we have
deadlines, we don’t know what to do with it, let’s just go with what we
have…
• Hypothesis 1: poorly informed decisions based upon misleading
deterministic/point estimates can be very costly, leading to a longer
term and larger resource allocation to correct mistakes that could have
been avoided or to find better solutions
• Hypothesis 2: Uncertainty analysis helps to determine when a robust
decision can be made versus when more information is needed first
• Hypothesis 3: Uncertainty and sensitivity analysis help identify key
weaknesses and focus limited resources to help improve estimates
• Hypothesis 4: Doing uncertainty analysis actually reduces overall
resource requirements, especially if it is integrated into the process of
model development and applications
Role of Modeling in Decision-Making
• Modeling should provide insight
• Modeling should help inform a decision
• Modeling should be in response to clearly
defined objectives that are relevant to a
decision.
Questions that Decision-Makers and
Stakeholders Typically Ask
• How well do we know these numbers?
– What is the precision of the estimates?
– Is there a systematic error (bias) in the estimates?
– Are the estimates based upon measurements,
modeling, or expert judgment?
• How significant are differences between two
alternatives?
• How significant are apparent trends over time?
• How effective are proposed control or management
strategies?
• What is the key source of uncertainty in these numbers?
• How can uncertainty be reduced?
Application of Uncertainty to Decision
Making
• Risk preference
–Risk averse
–Risk neutral
–Risk seeking
• Utility theory
• Benefits of quantifying uncertainty: Expected
Value of Including Uncertainty
• Benefits of reducing uncertainty: Expected
Value of Perfect Information (and others)
Variability and Uncertainty
• Variability: refers to the certainty that
–different members of a population will have
different values (inter-individual variability)
–values will vary over time for a given member
of the population (intra-individual variability)
• Uncertainty: refers to lack of knowledge
regarding
–True value of a fixed but unknown quantity
–True population distribution for variability
• Both depend on averaging time
Variability and Uncertainty
• Sources of Variability
–Stochasticity
–Periodicity, seasonality
–Mixtures of subpopulations
–Variation that could be explained with better
models
–Variation that could be reduced through control
measures
Variability and Uncertainty
• Sources of Uncertainty:
– Random sampling error for a random sample of data
– Measurement errors
» Systematic error (bias, lack of accuracy)
» Random error (imprecision)
– Non-representativeness
» Not a random sample, leading to bias in mean (e.g., only
measured loads not typical of daily operations)
» Direct monitoring versus infrequent sampling versus
estimation, averaging time
» Omissions
– Surrogate data (analogies with similar sources)
– Lack of relevant data
– Problem and scenario specification
– Modeling
Overview of “State of the Science”
• Statistical Methods Based Upon Empirical
Data
• Statistical Methods Based Upon Judgment
• Other Quantitative Methods
• Qualitative Methods
• Sensitivity Analysis
• Scenario Uncertainty
• Model Uncertainty
• Communication
• Decision Analysis
Statistical Methods
Based Upon Empirical Data
• Frequentist, classical
• Statistical inference from sample data
–Parametric approaches
» Parameter estimation
» Goodness-of-fit
–Nonparametric approaches
–Mixture distributions
–Censored data
–Dependencies, correlations, deconvolution
Statistical Methods
Based Upon Empirical Data
• Variability and Uncertainty
– Sampling distributions for parameters
– Analytical solutions
– Bootstrap simulation
Propagating Variability and Uncertainty
– Analytical techniques
» Exact solutions (limited applicability)
» Approximate solutions
– Numerical methods
» Monte Carlo
» Latin Hypercube Sampling
» Other sampling methods (e.g., Hammersley,
Importance, stochastic response surface method,
Fourier Amplitude Sensitivity Test, Sobol’s method,
Quasi-Monte Carlo methods, etc.)
Monte Carlo Simulation
• Probabilistic approaches are widely used
• Monte Carlo (and similar types of) simulation
are widely used.
• Why?
–Extremely flexible
» Inputs
» Models
–Relatively straightforward to conceptualize
Tiered Approach to Analysis
• Purpose of Analyses (examples)
– Screening to prioritize resources
– Regulatory decision-making
– Research planning
• Types of Analyses
– Screening level point-estimates
– Sensitivity Analysis
– One-Dimensional Probabilistic Analysis
– Two-Dimensional Probabilistic Analysis
– Non-probabilistic approaches
Methods
Based Upon Expert Judgment
• Expert Elicitation
– Heuristics and Biases
» Availability
» Anchoring and Adjustment
» Representativeness
» Others (e.g., Motivational, Expert, etc.)
– Elicitation Protocols
» Motivating the expert
» Structuring
» Conditioning
» Encoding
» Verification
– Documentation
– Individuals and Groups
– When Experts Diasagree
An Example of Elicitation Protocols:
Stanford/SRI Protocol
Motivating
(Es tablish Rapport)
Structuring
(Identify Variables )
Conditioning
(Get Expert to Think About Evidence)
Encoding
(Quantify Judgment About Uncertainty)
Verify
(Tes t the Judgment)
Key Ongoing Challenges
• Expert Judgment vs. Data
– Perception that judgment is more biased than analysis
of available data
– Unless data are exactly representative, they too could
be biased
– Statistical methods are “objective” in that the results
can be reproduced by others, but this does not
guarantee absence of bias
– A key area for moving forward is to agree on conditions
under which expert judgment is an acceptable basis for
subjective probability distributions, even for rulemaking
situations
Appropriate Use of Expert Judgment in
Regulatory Decision Making
• There are examples…e.g.,
– analysis of health effects for EPA standards
– Uncertainty in benefit/cost analysis (EPA, OMB)
– Probabilistic risk analysis of nuclear facilities
• Key components of credible use of expert judgment:
– Follow a clear and appropriate protocol for selecting experts and
for elicitation
– For the conditioning step, consider obtaining input via workshop,
but for encoding, work individually with experts – preferably at their
location
– Document (explain) the basis for each judgment
– Compare judgments: identify key similarities and differences
– Evaluate the implications of apparent differences with respect to
decision objectives – do not “combine” judgments without first
doing this
– Where possible, allow for iteration
Statistical Methods
Based Upon Expert Judgment
• Bayesian methods can incorporate expert judgment
– Prior distribution
– Update with data using likelihood function and Bayes’
Theorem
– Create a posterior distribution
• Bayesian methods can also deal with various complex
situations:
– Conditional probabilities (dependencies)
– Combining information from multiple sources
• Appears to be very flexible
• Computationally, can be very complex
• Complexity is a barrier to more widespread use
Other Quantitative Methods
• Interval Methods
–Simple intervals
–Probability bounds
–Produce “optimally” narrow bounds – cannot be
any narrower and still enclose all possible
outcomes, including dependencies among
inputs
–Bounds can be very wide in comparison to
confidence intervals
Other Quantitative Methods
• Fuzzy methods
– Representation of vagueness, rather than uncertainty
– Approximate/semi-quantitative
– Has been applied in many fields
• Meta-analysis
– Quantitatively combine, synthesize, and summarize data and
results from different sources
– Requires assessment of homogeneity among studies prior to
combining
– Produces data with larger sample sizes than the constituent inputs
– Can be applied to summary data
– If raw data are available, other methods may be preferred
Scenario Uncertainty
• A need for formal methods
• Creativity, brainstorming, imagination
• Key dimensions (e.g., human exposure assessment)
– Pollutants
– Transport pathways
– Exposure routes
– Susceptible populations
– Averaging time
– Geographic extent
– Time Periods
– Activity Patterns
• Which dimensions/combinations matter, which ones don’t?
• Uncertainty associated with mis-specification of a scenario –
systematic error
• Scenario definition should be considered when developing and
applying models
Model Uncertainty
• Model Boundaries (related to
scenario)
• Simplifications
–Aggregation
–Exclusion
•
•
•
•
•
Resolution
Structure
Calibration
Validation, Partial validation
Extrapolation
Model Uncertainty
• Methods for Dealing with Model Uncertainty
–Compare alternative models, but do not
combine
–Weight predictions of alternative models (e.g.,
probability trees)
–Meta-models that degenerate into alternative
models (e.g., Y = a(|x-t|)n , where n determines
linear/nonlinear and t determines threshold or
not)
Probability Density
Probability Density
Weighting vs. Averaging
Each Model has Equal Weight
Model B
Model A
Average of
Both Models
Output of Interest
Neither Model
Supports This
Range of Outcomes
Output of Interest
Sensitivity Analysis
• Objectives of Sensitivity Analysis (examples):
– Help identify key sources of variability (to aid management
strategy)
» Critical control points?
» Critical limits?
– Help identify key sources of uncertainty (to prioritize additional
data collection to reduce uncertainty)
– What causes worst/best outcomes?
– Evaluate model behavior to assist verification/validation
– To assist in process of model development
• Local vs. Global Sensitivity Analysis
• Model Dependent vs. Model Independent Sensitivity Analysis
• Applicability of methods often depends upon characteristics of a
model (e.g., nonlinear, thresholds, categorical inputs, etc.)
Examples of Sensitivity Analysis Methods
• Mathematical Methods
Assess sensitivity of a model
output to the range of variation
of an input.
• Statistical Methods
Effect of variance in inputs on
the output distribution.
• Graphical Methods
Representation of sensitivity
in the form of graphs, charts,
or surfaces.
Nominal Range
Sensitivity Analysis (NRSA)
Differential Sensitivity Analysis
Regression Analysis (RA)
Analysis of Variance (ANOVA)
Classification and Regression
Trees (CART)
Scatter Plots
Conditional
Sensitivity
Sensitivity Analysis Methods (Examples)
•
•
•
•
•
•
•
•
•
•
•
•
Nominal Range Sensitivity Analysis
Differential Sensitivity Analysis
Conditional Analysis
Correlation coefficients (sample, rank)
Linear regression (sample, rank, variety of basis functions
possible)
Other regression methods
Analysis of Variance (ANOVA)
Categorical and Regression Trees (CART) (a.k.a. Hierarchical
Tree-Based Regression)
Sobol’s method
Fourier Amplitude Sensitivity Test (FAST)
Mutual Information Index
Scatter Plots
Sensitivity Analysis: Displays/Summaries
•
•
•
•
Scatter plots
Line plots/conditional analyses
Radar plots
Distributions (for uncertainty or variability in
sensitivity)
• Summary statistics
• Categorical and regression trees
• Apportionment of variance
Guidance on Sensitivity Analysis
Guidance for Practitioners, with a focus on food
safety process risk models (Frey et al., 2004):
• When to perform sensitivity analysis
• Information needed depending upon objectives
• Preparation of existing or new models
• Defining the case study/scenarios
• Selection of sensitivity analysis methods
• Procedures for application of methods
• Presentation and interpretion of results
Summary of Evaluation Results for
Selected Sensitivity Analysis Methods
Example of Guidance on Selection of
Sensitivity Analysis Methods
Source: Frey et al., 2004, www.ce.ncsu.edu/risk/
Example of Guidance on Selection of
Sensitivity Analysis Methods
Communication
• Case Studies (scenarios)
• Graphical Methods
–Influence Diagrams
–Decision Tree
–Others
• Summary statistics/data
• Evaluation of effectiveness of methods for
communication (e.g., Bloom et al., 1993;
Ibrekk and Morgan, 1987)
Example Case Studies
•
•
•
•
Technology Assessment
Emission Factors and Inventories
Air Quality Modeling
Risk Assessment
Role of Technology Assessment in
Regulatory Processes (Examples)
• Assessment of ability of technology to achieve
desired regulatory or policy goals (emissions
control, safety, efficiency, etc.)
• Evaluation of regulatory alternatives (e.g.,
based on model cost estimates)
• Regulatory Impact Analysis – assessment of
costs
An Example of Federal Decision Making:
Process Technology RD&D
Research and
Developm ent
Screening
R&D Cost
Estim ate
No
Reject?
Yes
Cost High
Cost Reasonable
Yes
Rework
Cost Estim ate?
No
Comm it
$ for Scope
Def.
Yes
P roj ect
Definition
Unr esolved
Technical
Uncertainties?
No
R&D
Yes
No
Yes
Comm ercialization
R&D
or
P ilot Plant?
Dem onstration
P lant
No
Good
Results?
A Probabilistic Framework for Federal
Process Technology Decision-Making
P OLICY
CONCERNS
SET OF
TECHNOLOGIES
IDENTIFY
DECISION
CRITERIA
INITIAL
SCREENING
NO, Reject
YES
Design Studies
ENGINEERING
MODELS
Test Data
CHARACTERIZE
UNCERTAINTIES
Experts
PROBABILISTIC ANALYSES
Characterize
P erferm ance,
Emissions,
and Cost
Uncertainties
P robabilistic
Design
Analy sis
P rioritization
of
Uncertainties
Comparative
Risk
Assessm ent
DECISION ANALYSIS
• DESIGN TRADE-OFFS
• TECHNOLOGY SELECTION
• RESEARCH STRATEGIES
NO, reject
SECOND
SCREENING
YES, commit further
resources to R&D
Methodology for Probabilistic Technology
Assessment
• Process simulation of process technologies in probabilistic
frameworks
– Integrated Environmental Control Model (IECM) and
derivatives
– Probabilistic capability for ASPEN chemical process
simulator
• Quantification of uncertainty in model inputs
– Statistical analysis
– Elicitation of expert judgment
• Monte Carlo simulation
• Statistical methods for sensitivity analysis
• Decision tree approach to comparing technologies and
evaluating benefits of additional research
Conceptual Diagram of Probabilistic
Modeling
Input
Uncertainties
Engineering Performance
and Cost Model of a
New Process Technology
Exhaust Gas
Blowdown
Raw
water
Output
Uncertainties
Boiler Feedwater
Boiler
Feedwater
Treatment
HRSG
& Steam
Cycle
Return Water
Steam
Turbine
Steam
Performance
Gasifier Steam
Performance
Inputs
Shift &
Regeneration
Steam
Coal
Coal
Handling
Gasification, Cyclone
CoalParticulate &
Ash Removal,
Fines Recycle
Raw
Syngas
Gas Turbine Exhaust
Cyclone
Hot Gas
Desulfurization
Gas
Turbine
Clean
Syngas
Emissions
Ash
Gasifier Air
Ash
Fines
Fines
Sulfuric
Acid
Plant
Cost Inputs
Tailgas
Sulfuric Acid
Air
Air
Electricity
Cost
Comparison of Probabilistic and PointEstimate Results for an IGCC System
Cumulative Probability
1
0.8
0.6
All Uncertainties
Point Estimate
0.4
Key Uncertainties
0.2
0
1600
1700
1800
1900
2000
Cumulative Probability
Total Capital Requirement (1998 $/kW)
1
0.8
0.6
All Uncertainties
0.4
Point Estimate
0.2
Key Uncertainties
0
47
48
49
50
51
52
53
54
55
56
Levelized Cost of Electricity, (1998 mills/kWh)
57
58
Cumulative Probability
Example of a Probabilistic Comparison of
Technology Options
1
0.8
0.6
Probabilistic
Deterministic
0.4
0.2
0
-7.5
-7.0
-6.5
-6.0
-5.5
-5.0
COE Difference of IGCC-7H and IGCC-7FA, miils/kWh
Uncertainty in the difference in cost between two technologies,
taking into account correlations between them
Example: Engineering Study of
Coal-Gasification Systems
• DOE/METC Engineers
• Briefing Packets:
- Part 1: Uncertainty Analysis (9 pages)
- Part 2: Process Area Technical Background
- Lurgi Gasifier:
12 p., 16 ref.
- KRW Gasifier:
19 p., 25 ref.
- Desulfurization:
9 p., 19 ref.
- Gas Turbine:
23 p., 36 ref.
- Part 3: Questionnaire
• Follow-Up
• Fines Carryover
P r o b a b il i ty D e n sit y
Examples of the Judgments
of One Expert
0.2
0.1
0.0
0
5
10
15
20
25
30
• Carbon Retention
P r ob a b i l it y D e nsi ty
Fines Ca rryove r, % of coal fee d
0.3
Expert LG-1's Judgment Regarding Fines Carry over
0.2
0.1
0.0
0
5
10
15
20
3
4
• Air/Coal Ratio
P ro b a b i l i t y D e n si t y
Carbon Rete ntion in Bottom Ash, % of c oa l fee d ca rbon
3
Expert LG-1's Judgment Regarding Carbon Retention
2
1
0
0
1
2
Air-to-Coal Ratio, lb air/lb DAF coal
Expert LG-1's Judgment Regarding Gasifier Air/Coal Ratio
Examples of the Judgments
of Multiple Experts
Expert
Sorbent Replacement
Sorbent Loading
0. 3
Probability Density
ZF-1
Probability Density
1.5
1.0
0.5
0
5
10
15
20
Sorben t Replacemen t Rate, wt-%/cy cle
0
25
15
20
25
30
35
0. 12
Probability Density
Probability Density
10
Expert ZF-1's Judgment Regardi ng Sorbent Sulfur Loadi ng
Judgment of Expert ZF-1 Regarding Sorbent A ttrition
0.5
0.4
0.3
0.2
0.1
0. 10
0. 08
0. 06
0. 04
0. 02
0. 00
0.0
0
5
10
15
20
Sorben t Replacemen t Rate, wt-%/cy cle
0
25
5
10
15
20
25
30
35
So rbent Sulfur Loadin g, wt-%
E xp ert ZF-2 's Jud gmen t Reg ard in g So rbent Su lfu r L oadi ng
Judgment of Expert ZF-2 Regarding Sorbent Life
8
0. 10
Probability Density
Probability Density
5
So rb en t Su lfu r Loadin g, wt-%
0.6
ZF-3
0. 1
0. 0
0.0
ZF-2
0. 2
6
4
2
0
0
5
10
15
20
Sorben t Replacemen t Rate, wt-%/cy cle
25
Judgment of Expert ZF-3 Regarding Sorbent A ttrition
0. 08
0. 06
0. 04
0. 02
0. 00
0
5
10
15
20
25
30
35
So rbent Sulfur Loadin g, wt-%
E xp ert ZF-3 's Jud gmen t Reg ard in g So rbent Su lfu r L oadi ng
Cumulative Pro bability
Do Different Judgments Really Matter?
1.0
0.8
0.6
Input Uncertainty Assumptions
Judgm ent s LG-1 and ZF-1
Judgm ent s LG-1 and ZF-2P
Judgm ent s LG-1 and ZF-2R
Judgm ent s LG-1 and ZF-3
0.4
0.2
0.0
40
50
60
70
80
90
100
Levelized Cost of Electricity, Constant 1989 Mills/kWh
110
Data from "Data ALH LG-1,ZF-1 research"
• Specific Sources
of Disagreement:
- Sorbent Loading
- Sorbent Attrition
• Qualitative Agreement in Several Cases
120
Technology Assessment:
Findings (1)
• Interactions among uncertain inputs, and nonlinearities in
model, contribute to positive skewness in model output
uncertainties
– Uncertainties in inputs are often positively skewed (physical,
non-negative quantities)
– The mean value of a probabilistic estimate is often “worse”
(lower performance, higher cost) than the “best guess”
deterministic estimate, and the probability of “worse” outcomes
is typically greater than 50 percent.
– A system approach is needed to account for interactions among
process areas
– Deterministic analysis leads to apparent “cost growth” and
“performance shortfall” because it does not account for
simultaneous interactions among positively skewed inputs
– Uncertainty analysis requires more thought pertaining to
developing input assumptions, but provides more insight into
potential sources of “cost growth” and “performance shortfall”
Technology Assessment:
Findings (2)
• A decision model provides a framework for evaluating
judgments regarding the outcomes of additional research
and prioritizing additional research
– Able to quantify the probability of “pay-offs” as well as downside
risks
– Able to compare competing options under uncertainty and
identify robust choices
– Trade-offs when comparing technologies can be evaluated
probabilistically (e.g., efficiency, emissions, and cost)
• It is possible to combine approaches for quantifying
uncertainty in one assessment, consistent with objectives
Technology Assessment:
Findings (3)
• Thinking about uncertainties leads to better understanding of
what matters most in the assessment
– Often, only a relatively small number of inputs contribute
substantially to uncertainty in a model output
– Reducing uncertainty in only a few key inputs can
substantially reduce downside risk and increase the payoffs of new technology
– Conversely, for those inputs to which the output is not
sensitive, it is not critical to devote resources to
refinement
• When basing inputs on expert judgments, only those
disagreements that really matter to the decision need
become the focus of further discussion and evaluation
• Bottom Line: Probabilistic analysis helps improve decisions
and avoid unpleasant “surprises”.
Emission Factors and Inventories
• Significance to Regulatory Processes:
– Assessment of capability of technology to reduce/prevent
emissions
– Evaluation of regulatory alternatives
– Regulatory Impact Analysis – including benefit/cost analysis
– Component of air quality management at various temporal and
spatial scales
– Component of human and ecological exposure and risk
assessment
• Modeling Aspects
– Some emission factors are estimated using models
– Emission Inventories are linear models
– Specialized models for some emission factors and inventories
(e.g., Mobile6, NONROAD, MOVES)
Motivations for Probabilistic Emission
Factors and Inventories
• How good are the estimates?
• What are the key sources of uncertainty in the estimates that
should be targeted for improvement?
• Likelihood of meeting an emissions budget?
• Which emission sources are the most significant?
• What is the inter-unit variability in emissions?
• What is the uncertainty in mean emissions for a group/fleet of
sources?
• What are the implications of uncertainty in emissions for air quality
management, risk management of human exposures?
• Consideration of geographic extent and averaging time
• Estimation for future scenarios versus retrospective estimates of
past emissions or assessment of current emissions
Motivations for Probabilistic Analysis
• “That a perfect assessment of uncertainty cannot
be done, however, should not stop researchers
from estimating the uncertainties that can be
addressed quantitatively” (p. 150, NRC, 2000)
• “EPA, along with other agencies and industries,
should undertake the necessary measures to
conduct quantitative uncertainty analyses of the
mobile-source emissions models in the modeling
toolkit.” (p. 166, NRC, 2000)
Current Practice for Qualifying Uncertainty
in Emission Factors and Inventories
• Qualitative ratings for emission factors (AP-42)
• Data Attribute Rating System (DARS) (not
really used in practice)
• Both methods are qualitative
• No quantitative interpretation
• Some sources of uncertainty (i.e. nonrepresentativeness) difficult to quantify
• Qualitative methods can complement
quantitative methods
Statistical Methodological Approach
• Compilation and evaluation of database
• Visualization of data by developing empirical
cumulative distribution functions
• Fitting, evaluation, and selection of alternative
probability distribution models
• Characterization of uncertainty in the distributions
for variability (e.g., uncertainty in the mean)
• Propagation of uncertainty in activity and
emissions factors to estimate uncertainty in total
emissions
• Calculation of importance of uncertainty
Summary of Approaches to Emission
Factor and Inventory Uncertainty
• Probabilistic Methods
– Empirical, Parametric
– Mixture distributions
– Censored distributions (non-detects)
– Vector autoregressive time series (intra- and inter-unit correlation)
– Bootstrap simulation
– Expert Judgment
– Monte Carlo simulation
– Sensitivity analysis
• Software tools:
– AUVEE – Analysis of Uncertainty and Variability in Emissions
Estimation
– AuvTool – standalone software
Summary of Probabilistic Emissions Case
Studies at NCSU
• Case Studies (examples):
– Point sources
» Power Plants
» Natural gas-fired engines (e.g., compressor stations)
– Mobile sources
» On-Road Highway Vehicles
» Non-Road Vehicles (e.g., Lawn & Garden, Construction, Farm, &
Industrial)
– Area sources
»
»
»
»
»
»
Consumer/Commercial Product Use
Natural Gas-Fueled Internal Combustion Engines
Gasoline Terminal Loading Loss
Cutback Asphalt Paving
Architectural Coatings
Wood Furniture Coatings
• Pollutants
– NOx
– VOC
– Urban air toxics (e.g., Houston case study)
Example Results: Lawn & Garden Equipment
Engine
Type
Pollutant
# of
Data
Fitted
Distribution
Mean
(g/hp-hr)
Relative
Uncertainty
NOx
16
Lognormal
0.81
-46% to +65%
THC
16
Lognormal
222
-32% to +38%
NOx
27
Lognormal
2.05
-25% to +38%
THC
22
Lognormal
21.5
-38% to +45%
2-Stroke
4-Stroke
Based on Frey and Bammi (2002)
Probabilistic CO Emission Factors for On-Road Light
Duty Gasoline Vehicles (Mobile5)
Driving
Cycle
Speed
(mph)
Standard Temperature and Reid Vapor
Adjusted for Temperature and Reid
Pressure
Vapor Pressure
Mean
System- Random Error Mean System- Random
Error
atic Error ( - ) %
(+)%
atic Error ( - ) %
(+)%
LSP1
2.45
36.9
23.6
-59
62
148
75
-90
282
LSP2
3.64
38.9
0.92
-70
66
154
-5
-91
279
LSP3
4.02
50.9
-16.4
-62
65
222
-88
-91
227
NYCC
7.1
33.1
-11.8
-25
23
130
-57
-88
256
SCC12
FTP
BAG2
SCC36
12.1
14.3
-2.89
-28
27
56.8
-16
-89
229
16.1
8.78
-0.71
-10
10
34.7
-6
-87
224
35.9
6.65
-0.33
-20
23
26.3
-3
-88
236
HFET
48.4
4.16
0.21
-21
22
16.5
-1
-88
246
HSP1
50.9
4.50
-38
43
18.2
-88
262
HSP2
57.6
0.26
-42
43
1.03
-88
236
HSP3
64.3
0.27
-57
57
1.04
-90
272
Based on Frey and Zheng (2002)
MOVES
• Conceptual Basis for MOVES, the successor to
Mobile6 and NONROAD
(www.epa.gov/otaq/ngm.htm)
– “Shootout”
– NCSU report on modal/binning approach
• NCSU recommended approaches for quantification of
inter-vehicle variability and fleet average uncertainty
in modal emission rates and estimates of emissions
for driving cycles (details in our report to EPA)
• EPA requested further assessment of an approximate
analytical procedure for propagating error (report by
Frey to EPA)
• At last report, EPA was considering Monte Carlo
simulation
Probabilistic AP-42 Emission Factors for Natural Gasfueled Engines (July 2000 Version)
Engine and
Load Range
AP-42
Emission
Factora
No. of
Data /
Engines
Fitted
Distribution.b
Mean of
Bootstrap
sample meansa
Relative 95% CI
of meanc (%)
Uncontrolled NOx
2SLB, 90-105%
3.17
34 / 11
W
3.05
-24 to +24
2SLB, <90%
1.94
24 / 11
W
2.18
-41 to +46
4SLB, 90-105%
4.08
12 / 4
G
4.06
-39 to +49
4SLB, <90%
0.847
13 / 5
W
1.81
-80 to +180
Uncontrolled TOC
2SLB, all load
1.64
57 / 14
W
1.45
-16 to +18
4SLB, all load
1.47
37 / 4
G
1.12
-45 to +57
aUnits
are lb/106 BTU.
bMLE is used for 2SLB engine, MoMM is used for 4SLB engine,
W=Weibull distribution, G=Gamma distribution.
cCalculated based upon bootstrap simulation results.
Based on Frey and Li (2003) (submitted)
Summary of Probabilistic Emission
Inventories for Selected Air Toxics
City
Houston
Jacksonville
Pollutant
95% C.I. (%, %)
Benzene
(-46, 108)
Formaldehyde
(-35, 67)
Chromium
(-20, 34)
Arsenic
(-69, 203)
1,3-butadiene
(-46, 108)
Mercury
(-25, 30)
Arsenic
(-83, 243)
Benzene
(-56, 146)
Formaldehyde
(-42, 89)
Lead
(-54, 175)
Key Sources of Uncertainty
City
Houston
Jacksonville
Pollutant
Key Sources of Uncertainty (number of dominant
sources)
Benzene
Gasoline onroad mobile sources (1)
Formaldehyde
Onroad mobile sources; Nonroad mobile sources (2)
Chemical manufacturing-fuel fired heaters (1)
Chromium
External utility coal combustion boilers;
Hard chromium electroplating (2)
Arsenic
External coal combustion utility boilers (1)
1, 3-butadiene
Onroad mobile sources (1)
Mercury
External coal combustion boilers (1)
Arsenic
External coal combustion boilers (1)
Benzene
Onroad mobile source (1)
Formaldehyde
Onroad mobile source; Aircraft (2)
Lead
External coal combustion boilers; Fuel oil external
combustion; Waste oil external combustion (3)
Cumulative Probability
Example of Benzene Emission Factor Category 3b:
Nonwinter Storage Losses at a Bulk Terminal : Empirical
Distribution
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
Benzene Emission Factor
(ton/yr/tank)
1
Example of Benzene Emission Factor
Category 3b: Fitted Lognormal Distribution
Cumulative Probability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
Benzene Emission Factor
(ton/yr/tank)
1
Example of Benzene Emission Factor
Category 3b: Confidence Interval in the CDF
Cumulative Probability
1.0
0.8
0.6
Dat a Set
Fit t ed Dist ribut ion
Confidence90Int
erval
percent
50 percent
90 percent
95 percent
0.4
0.2
0.0
-3
10
-2
10
-1
10
Benzene Emission Fact or
(t on/yr/t ank)
0
10
Example of Benzene Emission Factor Category 3b:
Uncertainty in the Mean
Cumulative Probability
1
0.8
0.06
mean = 0.6
0.6
0.4
95% Probability
Range (0.016, 0.18)
0.2
0
0
0.05
0.1
0.15
Benzene Emission Factor
(ton/yr/tank)
Uncertainty in mean -73% to +200%
0.2
Using AuvTool to Fit a Distribution for Variability
Using AuvTool for Bootstrap Simulation
Using AuvTool to Quantify Uncertainty in the Mean
Uncertainty in Total Emission Inventory:
AUVEE Prototype Software
Summary of
Probabilistic Emission Inventory
Identification of Key Sources of Uncertainty
in an Inventory
Detection Limits and Air Toxic Emission
Factor Data
• Many air toxic emission factor data contain
one or more measurements below a “detection
limit”
• Detection limits can be unique to each
measurement because of differences in
sample volumes and analytical chemistry
methods among sites or contractors
• A database can contain some non-detected
data with detection limits larger than detected
values measured at other sites
Methodology: Conventional Methods for
Censored Data
• Conventional approaches to estimate the mean:
-Remove non detected values (biased)
-Replace values below DL with zero (underestimate)
-Replace values below DL with DL/2 (biased)
-Replace values below DL with DL (overestimate)
• Cause biased estimates of the mean
• Does not provide adequate insights regarding
– Population distribution
– Unbiased statistics
– Uncertainty in statistics
Methodology: Quantification of the
Inter-Unit Variability in Censored Data
• Maximum Likelihood Estimation (MLE) is used
to fit parametric distributions to censored data
• MLE is asymptotically unbiased
• Fitted distribution is the best estimate of
variability
• Can estimate mean and other statistics from
the fitted distribution
• Can quantify uncertainty caused by random
sampling error using Bootstrap simulation
Results: Fitted Lognormal Distribution, No Censoring
1.00
Cumulative Probability
0.80
Data Set
Fitted Distribution
Confidence Interval
0.60
0.40
50 percent
90 percent
95 percent
0.20
0.00
0
1
2
3
Value of Random Variable
4
5
6
Results: Fitted Lognormal Distribution, 30% Censoring
1.00
Cumulative Probability
0.80
Data Set
Fitted Distribution
Confidence Interval
Data Set
50 percent
Fitted 90
Distribution
percent
Confidence Interval
95 percent
0.60
0.40
50 percent
90 percent
95 percent
0.20
Detection Limt
0.00
0
1
2
3
Value of Random Variable
4
5
6
Results: Fitted Lognormal Distribution, 60% Censoring
Cumulative Probability
1.00
0.80
Data Set
Fitted Distribution
Confidence Interval
0.60
50 percent
90 percent
95 percent
0.40
0.20
Detection Limt
0.00
0
1
2
3
4
Value of Random Variable
5
6
Example Case Study: Formaldehyde Emission
Factor from External Coal Combustion
• 14 data points including 5 censored values
• Each censored data point has a different
detection limit
• Some detected data values are less than some
detection limits
• There is uncertainty regarding the empirical
cumulative probability of such detected data
values
Results of Example Case Study: Empirical
Cumulative Probability
Cumulative Probability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
1
10
Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
100
Results of Example Case Study: Lognormal
Distribution Representing Inter-Unit Variability
Cumulative Probability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
1
10
Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
100
Results of Example Case: Uncertainty in
Inter-Unit Variability
Cumulative Probability
1.0
0.8
Data Set
Fit ted Distribution
Confidence Interval
0.6
0.4
50 percent
90 percent
95 percent
Detection Limit
Possible Plotting Position
0.2
0.0
-3
10
-2
10
-1
10
0
10
1
10
Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
2
10
Results of Example Case: Uncertainty in the Mean
(Basis to Develop Probabilistic Emission Inventory)
Cumulative Probability
1
0.8
Mean
=1.84
0.6
95 Percent Probability
Range: (0.42, 5.67)
0.4
0.2
0
0
4
8
12
Mean of Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
Uncertainty in mean -77% to +208%
16
Mixtures of Distributions
Cumulative Probability
1.0
Data Set
Fit ted Mixture Lognormal
Distribution
0.8
0.6
Confidence Interval
50 percent
90 percent
95 percent
0.4
0.2
0.0
100
300
500
700
900
NOx Emission Factor (gram/GJ fuel input)
Percent of data in 50% CI: 92%
Percent of data in 95% CI: 100%
1100
Case Study
• Charlotte modeling domain
• 32 units from 9 different coal-fired
power plants
• 1995 and 1998 data used
• Propagation of uncertainty
investigated using July 12 – July 16
1995 meteorological data
• Data available for emission and
activity factors
• Vector autoregressive time-series
modeling of emissions from each
unit
Emissions (t/hr) .
Time Series and Uncertainty
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
Real.
Obs.
12
36
60
Hour
84
108
132
Different uncertainty ranges for different hours of day
Emission Factors and Inventories:
Findings (1)
• Visualization of data used to develop an inventory is highly informative to choices
of empirical or parametric distribution models for quantification of variability or
uncertainty
• A key difficulty in developing probabilistic emission factors inventories is to find the
original data used by EPA and others.
– When data are found, they are typically poorly documented.
– The time required to assemble databases when original data could not be found
was substantial
• Test methods used for some emission sources are not representative of real world
operation, implying the need for real world data and/or expert judgment when
estimating uncertainty
• Uncertainty in measurement methods is not adequately reported. There is a need
for more systematic reporting of the precision and accuracy of measurement/test
methods
• Emissions databases should not be arbitrarily fragmented into too many
subcategories. Conversely, subcategories should be created when there is a
good (empirical) basis for doing so.
Emission Factors and Inventories:
Findings (2)
• Uncertainties in emission factors are typically positively skewed,
unless the uncertainties are relatively small (e.g., less than about
plus or minus 30 percent)
• Uncertainty estimates might be sensitive to the choice of
parametric distribution models if there is variation in the goodnessof-fit among the alternatives. However, in such cases, there is
typically a preferred best fit. When several alternative models
provide equivalent fits, results are not sensitive to the choice of the
model
• The quantifiable portion of uncertainty attributable to random
sampling error can be large and should be accounted for when
using emission factors and inventories
• Variability in emissions among units could be a basis for assessing
the potential of emissions trading programs
Emission Factors and Inventories:
Findings (3)
• Intra-unit dependence in hourly emissions is significant for some
sources (e.g., power plants), including hourly and daily lag effects
• Inter-unit dependence in emissions is important for some sources,
such as power plants
• Range of variability and uncertainty is typically much greater as the
averaging time decreases
• Even for sources with continuous emissions monitoring data, there
is uncertainty regarding predictions of future emissions that can be
informed by analysis of historical data
• Prototype software demonstrates the feasibility of increasing the
convenience of performing probabilistic analysis
• Uncertainties in total inventories are often attributable to just a few
key emission sources
Mobile5 and Mobile6
Findings
• Range of variability and uncertainty in correction factors (e.g.,
temperature) dominate and are large
• Uncertainties in average emissions are large in some cases (e.g., 80 to +220 percent): normality assumptions are not valid
• Asymmetry in uncertainties associated with non-negative
quantities and large inter-unit variability
• Sensitivity analysis was used to identify key sources of uncertainty
and recommend future data collection priorities in order to reduce
uncertainty
• There is a proliferation of driving cycles. Some of them are
redundant and, therefore, unnecessary
• When comparing model predictions with validation data, or when
comparing models with each other, their prediction uncertainty
ranges should be considered
• It is difficult to do an uncertainty analysis for a model such as
Mobile5 or Mobile6 after the fact, but would be much easier if
integrated into the data management and modeling approach.
Air Quality Modeling
• Widely used for:
–Assessment of regulatory options
–Identification and evaluation of control
strategies (which pollutants, how much control,
where to control?)
–Emissions permit applications
–Identification of contributing factors to local,
urban, and regional air quality problems
–Human exposure assessment
PROBABILISTIC MODELING
Input
Uncertainties
Output
Uncertainties
Emissions
Peak Ozone
Chemistry
Variable-Grid
Urban Airshed Model
(UAM-V)
Local Ozone
Meteorology
Local NOx
Initial & Boundary
Conditions
Local VOC
Case Study of Hanna et al. (2001)
•
•
•
•
OTAG Modeling Domain (eastern U.S.)
UAM-V model with Carbon Bond-IV mechanism
Uncertainty in peak ozone concentrations
Assessment of effects of 50% reductions in each of
NOx and VOC emissions
• Quantification of uncertainty in 128 inputs
– Literature review for chemical kinetic rate constants
– Expert elicitation for emissions, meteorological, and
initial and boundary condition inputs
• Monte Carlo simulation with n=100
• Correlations used to identify key sources of
uncertainty
Key Findings of Case Study of Hanna et al.
(2001)
•
•
•
•
•
•
•
It was feasible to perform Monte Carlo simulation on UAM-V for the OTAG domain and a
seven day episode
Simulation results include base case uncertainty estimates for ozone concentrations and
estimates of differences in ozone concentrations because of emissions reduction strategies
There was less uncertainty in estimates of differences in concentration than in absolute
estimates of total concentration
Reductions in NOx emissions led to higher estimated reductions in O3 than did reductions in
VOC emissions. This is consistent with the expectation that most of the domain is NO xlimited.
Key uncertainties include NOx photolysis rate, several meteorological variables, and biogenic
VOC emissions
Compared to Hanna et al. (1998), there was more disaggregation of uncertainty estimates for
emission sources, and this may tend to weaken the sensitivity to any one source. It is
possible, however, that model outputs would be more sensitive to an aggregated collection of
emissions sources.
There is a need for improved methods of uncertainty estimation, particularly for the chemical
mechanism and the meteorological fields, and for better accounting of correlations and
dependencies (e.g., temperature dependence of biogenic emissions).
Case Study of Abdel-Aziz and Frey (2004)
• Focus was on evaluating implications of uncertainties
in hourly coal-fired power plant NOx emissions with
respect to ozone for the Charlotte, NC domain
• Key questions:
– (1) what is the uncertainty in ozone predictions solely
attributable to uncertainty in coal-fired utility NOx
emissions?;
– (2) can uncertainties in maximum ozone levels be
attributed to specific power plant units?;
– (3) how likely is it that National Ambient Air Quality
Standards (NAAQS) will be exceeded?; and
– (4) how important is it to account for inter-unit
correlation in emission uncertainties
Probability of Exceeding NAAQS:
Comparison of 1-hour and 8-hour Standards
Location of Power Plant Impact
Correlation Coeff = 0.87
Ozone Conc. (ppm)
0.135
0.13
0.125
0.12
0.115
0
20
40
60
Emissions (t/4 hrs)
80
100
Analysis of
Correlation in
Emissions
versus Ozone
Levels in a
Specific Grid
Cell Can
Detect
Influence of a
Specific Plant
Key Findings from Abdel-Aziz and Frey
(2004) study
• The uncertainty in maximum 1-hour ozone predictions is potentially
large enough to create ambiguity regarding compliance with the
NAAQS for any given emissions management strategy.
• Control strategies can be developed to achieve attainment with an
acceptable degree of confidence, such as 90 or 95 percent.
• There was a substantial difference in results when comparing
“independent” versus “dependent” units – thus, it can be important to a
decision to account for dependencies between units
• Probabilistic air quality modeling results provide insight regarding
where to site monitoring stations and regarding the number of stations
needed
• Under the old 1-hour standard, uncertainties in the maximum domainwide ozone levels could be traced to an individual power plant,
thereby implying that control strategies must include that plant.
• Under the new 8-hour standard, uncertainties in maximum ozone
levels are attributable to many plants, implying the need for a more
widespread control strategy
Risk Assessment Modeling
• Risk assessment is growing in importance as a
basis for regulatory decision making
–E.g., Phase 2 of MACT standards
–Urban air toxics
–Food safety and international trade
–(etc.)
Human Exposure and Risk Analysis
• Over the last 10-15 years, there has been
growing acceptance and incorporation of
probabilistic approaches to dealing with interindividual variability and uncertainty
• EPA has issued various guidance
• International guidance: e.g., FAO/WHO
Example of Probabilistic Techniques in an
Exposure and Risk Model: SHEDS
Listeria Monocytogenes Model
Hazard Identification
Hazard Characterization
•Listeriosis
•Listeria monocytogen
•Food Categories:Seafood,
Produce, Dairy, Meats,
Combination foods
•Human Susceptibility
•Virulence
•Food Composition
•Dose-Response Model with
Adjustment
Exposure Assessment
•Food Consumption
•Food Contamination
•Post-Retail Growth of
Listeria
Population
•Perinatal
•Elderly
•Intermediate
Risk Characterization
•Estimated likely
hood of adverse
health effects
E. Coli O:157 in Ground Beef Risk
Assessment Model
Scope
Hazard Identification
Exposure Assessment
Slaughter
Production
Preparation
Production Slaughter Preparation
On-Farm
Dehiding
Grinding
Transport Evisceration
Growth
Marketing
Splitting
Cooking
of Live
Chilling Consumption
Animals Fabrication
Dose-response
Dose-Response
Assessment
Morbidity
Mortality
Findings Based Upon Risk Assessment
• Risk assessment applications often differ from others because of
distinction between inter-individual variability and uncertainty
• A two-dimensional probabilistic simulation framework is used
• Expert judgment is inherent in the process of fitting distributions to
data for variability and is often used to estimate uncertainty in the
parameters of such distributions
• Sensitivity analysis is critical to interpretation of risk assessment
results:
– Assists in risk management decision-making
– Prioritize future work to improve the assessment
• There was a lack of practical guidance regarding sensitivity
analysis of risk models that has been addressed by recent work
General Recommendations (1)
• Uncertainty and sensitivity analysis should be used to answer key
decision maker and stakeholder questions, e.g.,:
– prioritize scarce resources toward additional research or data
collection
– make choices among alternatives in the face of uncertainty,
– evaluate trends over time, etc.
• Where relevant to decision making, uncertainty and sensitivity
analysis should be included as functional requirements from the
beginning and incorporated into model and input data
development
• There should be minimum reporting requirements for uncertainty in
data (e.g., summary statistics such as mean, standard deviation,
sample size)
• Federal agencies should continue to improve documentation and
accessibility of models and data for public peer review
General Recommendations (2)
• Foster greater acceptance of appropriate methods for including,
documenting, and reviewing expert judgment in regulatorymotivated modeling and analysis
• There is a need for flexibility since there are many possible
approaches to analysis of uncertainty and sensitivity. Specific
choices should be appropriate to assessment objectives, which are
typically context-specific
• Human resources for modeling, including uncertainty and
sensitivity analysis, should be appropriately committed.
– Adequate time and budget to do the job right the first time
(could save time and money in the long run)
– Adequate training and peer review
– Promote workshops and other training opportunities, and
periodic refinement of authoritative compilations of techniques
and recommended practice
General Recommendations (3)
• Software tools substantially facilitate both uncertainty and
sensitivity analysis (e.g., Crystal Ball) but in some ways also limit
what is done in practice – there is a long-term need for software
tools appropriate to specific types of applications
• Some areas need more research – e.g., best techniques for
communication, real-world information needs for decision makers
• The relevance of analyses to decision making needs to be
emphasized and considered by analysts
• Decision makers need or should have access to information on
why/how they should use probabilistic results
• A multi-disciplinary compilation of relevant case studies and
insights from them is a useful way to help convince others of the
value of doing uncertainty and sensitivity analysis
• Uncertainty and sensitivity analysis should be an open and
transparent process that can be subject to scrutiny and peer
review