Transcript Probabililistic and Sesitivity Analysis of Risk Assessment Models

Methods and Applications of Probabilistic and
Sensitivity Analysis for Models Used in Decision
Processes
H. Christopher Frey, Ph.D.
Professor
Department of Civil, Construction, and Environmental Engineering
North Carolina State University
Raleigh, NC 27695
Currently on sabbatical as Exposure Modeling Advisor to the
National Exposure Research Laboratory, U.S. Environmental
Protection Agency, Research Triangle Park, NC 27701
Prepared for:
Annual Meeting
Society for Risk Analysis – Europe
Ljubljana, Slovenia
September 12, 2006
Outline
• Why are probabilistic and sensitivity
analysis needed?
• Overview of methods for probabilistic
analysis
– Model inputs
» Empirical data
» Expert judgment
– Model uncertainty
– Scenario uncertainty
• Overview of methods for sensitivity analysis
• Recommendations
Variability and Uncertainty
• Variability: refers to the certainty that
–different members of a population will have
different values (inter-individual variability)
–values will vary over time for a given member
of the population (intra-individual variability)
• Uncertainty: refers to lack of knowledge
regarding
–True value of a fixed but unknown quantity
–True population distribution for variability
• Both depend on averaging time
Variability
• Sources of Variability
–Stochasticity
–Periodicity, seasonality
–Mixtures of subpopulations
–Variation that could be explained with better
models
–Variation that could be reduced through control
measures
Uncertainty
• Sources of Uncertainty:
– Random sampling error for a random sample of data
– Measurement errors
» Systematic error (bias, lack of accuracy)
» Random error (imprecision)
– Non-representativeness
» Not a random sample, leading to bias in mean (e.g., only
measured loads not typical of daily operations)
» Direct monitoring versus infrequent sampling versus
estimation, averaging time
» Omissions
– Surrogate data (analogies with similar sources)
– Lack of relevant data
– Problem and scenario specification
– Modeling
Sensitivity Analysis
• A study of how the variation in the outputs of a model can
be attributed to, qualitatively or quantitatively, different
sources of variation in model inputs.
• Sensitivity analysis provides a tool to identify the inputs of
greatest importance by:
» Quantifying the impact of changes in input values
on model output
» Evaluating how variation in output values can be
apportioned among model inputs
» Identifying inputs contributing to best/worst
outcomes of interest
Why are probabilistic and sensitivity
analysis needed?
• Strategies for answering this question:
–what happens when we ignore variability,
uncertainty and sensitivity?
–what do decision makers want to know that
motivates doing variability, uncertainty and
sensitivity analysis?
–what constitutes best scientific practice?
• Decision makers may not care about all three,
but might find at least one to be convincing
(and useful)
When is Probabilistic
Analysis Needed or Useful?
• Consequences of poor or biased estimates are unacceptably
high
• A (usually conservative) screening level analysis indicates a
potential concern, but carries a level of uncertainty
• Determining the value of collecting additional information
• Uncertainty stems from multiple sources
• Significant equity issues are associated with variability
• Ranking or prioritizing significance of multiple pathways,
pollutants, sites, etc.
• Need for identifying possible risk management strategies
• Cost of remediation or intervention is high
• Scientific credibility is important
• Obligation to indicate what is known and how well it is known
When is a Probabilistic Approach Not
Needed?
• When a (usually conservative) screening level analysis
indicates a negligible problem
• When the cost of intervention is smaller than the cost of
analysis
• When safety is an urgent and/or obvious issue
• When there is little variability or uncertainty
Myths: Barriers to Use of Methods
• Myth: it takes more resources to do probabilistic analysis, we have
deadlines, we don’t know what to do with it, let’s just go with what we
have…
• Hypothesis 1: poorly informed decisions based upon misleading
deterministic/point estimates can be very costly, leading to a longer
term and larger resource allocation to correct mistakes that could have
been avoided or to find better solutions
• Hypothesis 2: Probabilistic analysis helps to determine when a
robust decision can be made versus when more information is needed
first
• Hypothesis 3: Variability, uncertainty and sensitivity analysis help
identify risk management priorities, identify key weaknesses and focus
limited resources to help improve estimates
• Hypothesis 4: Doing probabilistic analysis actually reduces overall
resource requirements, especially if it is integrated into the process of
model development and applications
Role of Modeling in Decision-Making
• Modeling should provide insight
• Modeling should help inform a decision
• Modeling should be in response to clearly
defined objectives that are relevant to a
decision.
Questions that Decision-Makers and
Stakeholders Typically Ask
• How well do we know these numbers?
– What is the precision of the estimates?
– Is there a systematic error (bias) in the estimates?
– Are the estimates based upon measurements,
modeling, or expert judgment?
• How significant are differences between two
alternatives?
• How significant are apparent trends over time?
• How effective are proposed control or management
strategies?
• What is the key source of uncertainty in these numbers?
• How can uncertainty be reduced?
Application of Uncertainty to Decision
Making
• Risk preference
–Risk averse
–Risk neutral
–Risk seeking
• Utility theory
• Benefits of quantifying uncertainty: Expected
Value of Including Uncertainty
• Benefits of reducing uncertainty: Expected
Value of Perfect Information (and others)
Overview of “State of the Science”
• Statistical Methods Based Upon
Empirical Data
• Statistical Methods Based Upon
Judgment
• Other Quantitative Methods
• Scenario Uncertainty
• Model Uncertainty
• Sensitivity Analysis
• Communication
Statistical Methods
Based Upon Empirical Data
• Frequentist, classical
• Statistical inference from sample data
–Parametric approaches
» Parameter estimation
» Goodness-of-fit
–Nonparametric approaches
–Mixture distributions
–Censored data
–Dependencies, correlations, deconvolution
Statistical Methods
Based Upon Empirical Data
• Variability and Uncertainty
– Sampling distributions for parameters
– Analytical solutions
– Bootstrap simulation
Cumulative Probability
Example of Benzene Emission Factor Category 3b: Nonwinter
Storage Losses at a Bulk Terminal : Empirical Distribution
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
Benzene Emission Factor
(ton/yr/tank)
1
Example of Benzene Emission Factor
Category 3b: Fitted Lognormal Distribution
Cumulative Probability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
Benzene Emission Factor
(ton/yr/tank)
1
Example of Benzene Emission Factor
Category 3b: Confidence Interval in the CDF
1.0
Cumulative Probability
0.8
0.6
Data Set
Fitted Distribution
Confidence 90
Interval
percent
50 percent
90 percent
95 percent
0.4
0.2
0.0
-3
10
-2
-1
10
10
Benzene Emission Factor
(ton/yr/tank)
0
10
Example of Benzene Emission Factor Category 3b:
Uncertainty in the Mean
Cumulative Probability
1
0.8
0.06
mean = 0.6
0.6
0.4
95% Probability
Range (0.016, 0.18)
0.2
0
0
0.05
0.1
0.15
Benzene Emission Factor
(ton/yr/tank)
Uncertainty in mean -73% to +200%
0.2
Results: Fitted Lognormal Distribution, No Censoring
1.00
Cumulative Probability
0.80
Data Set
Fitted Distribution
Confidence Interval
0.60
0.40
50 percent
90 percent
95 percent
0.20
0.00
0
1
2
3
Value of Random Variable
4
5
6
Results: Fitted Lognormal Distribution, 30% Censoring
1.00
Cumulative Probability
0.80
Data Set
Fitted Distribution
Confidence Interval
Data Set
50 percent
Fitted 90
Distribution
percent
Confidence Interval
95 percent
0.60
0.40
50 percent
90 percent
95 percent
0.20
Detection Limt
0.00
0
1
2
3
Value of Random Variable
4
5
6
Results: Fitted Lognormal Distribution, 60% Censoring
Cumulative Probability
1.00
0.80
Data Set
Fitted Distribution
Confidence Interval
0.60
0.40
50 percent
90 percent
95 percent
0.20
Detection Limt
0.00
0
1
2
3
4
Value of Random Variable
5
6
Results of Example Case Study: Empirical
Cumulative Probability
Cumulative Probability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
1
10
Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
100
Results of Example Case Study: Lognormal
Distribution Representing Inter-Unit Variability
Cumulative Probability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
0.1
1
10
Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
100
Results of Example Case: Uncertainty in
Inter-Unit Variability
Cumulative Probability
1.0
0.8
Data Set
Fitted Distribution
Confidence Interval
0.6
0.4
50 percent
90 percent
95 percent
Detection Limit
Possible Plotting Position
0.2
0.0
-3
10
-2
10
-1
10
0
10
1
10
Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
2
10
Results of Example Case: Uncertainty in the Mean
(Basis to Develop Probabilistic Emission Inventory)
Cumulative Probability
1
0.8
Mean
=1.84
0.6
95 Percent Probability
Range: (0.42, 5.67)
0.4
0.2
0
0
4
8
12
Mean of Formaldehyde Emission Factor
(0.0001 lb pollutants/ton coal combusted)
Uncertainty in mean -77% to +208%
16
Mixtures of Distributions
Cumulative Probability
1.0
Data Set
Fitted Mixture Lognormal
Distribution
0.8
0.6
Confidence Interval
50 percent
90 percent
95 percent
0.4
0.2
0.0
100
300
500
700
900
NOx Emission Factor (gram/GJ fuel input)
Percent of data in 50% CI: 92%
Percent of data in 95% CI: 100%
1100
Emissions (t/hr) .
Time Series and Uncertainty
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
Real.
Obs.
12
36
60
Hour
84
108
132
Different uncertainty ranges for different hours of day
Propagating Variability and Uncertainty
– Analytical techniques
» Exact solutions (limited applicability)
» Approximate solutions
– Numerical methods
» Monte Carlo
» Latin Hypercube Sampling
» Other sampling methods (e.g., Hammersley,
Importance, stochastic response surface method,
Fourier Amplitude Sensitivity Test, Sobol’s method,
Quasi-Monte Carlo methods, etc.)
Monte Carlo Simulation
• Probabilistic approaches are widely used
• Monte Carlo (and similar types of) simulation
are widely used.
• Why?
–Extremely flexible
» Inputs
» Models
–Relatively straightforward to conceptualize
Conceptual Diagram of Probabilistic
Modeling
Input
Uncertainties
Engineering Performance
and Cost Model of a
New Process Technology
Exhaust Gas
Blowdown
Raw
water
Output
Uncertainties
Boiler Feedwater
Boiler
Feedwater
Treatment
HRSG
& Steam
Cycle
Return Water
Steam
Turbine
Steam
Performance
Gasifier Steam
Performance
Inputs
Shift &
Regeneration
Steam
Coal
Coal
Handling
Gasification, Cyclone
CoalParticulate &
Ash Removal,
Fines Recycle
Raw
Syngas
Gas Turbine Exhaust
Cyclone
Hot Gas
Desulfurization
Gas
Turbine
Clean
Syngas
Emissions
Ash
Gasifier Air
Ash
Fines
Fines
Sulfuric
Acid
Plant
Cost Inputs
Tailgas
Sulfuric Acid
Air
Air
Electricity
Cost
Comparison of Probabilistic and PointEstimate Results for an IGCC System
Cumulative Probability
1
0.8
0.6
All Uncertainties
Point Estimate
0.4
Key Uncertainties
0.2
0
1600
1700
1800
1900
2000
Cumulative Probability
Total Capital Requirement (1998 $/kW)
1
0.8
0.6
All Uncertainties
0.4
Point Estimate
0.2
Key Uncertainties
0
47
48
49
50
51
52
53
54
55
56
Levelized Cost of Electricity, (1998 mills/kWh)
57
58
Input
Uncertainties
Output
Uncertainties
Emissions
Peak Ozone
Chemistry
Variable-Grid
Urban Airshed Model
(UAM-V)
Local Ozone
Meteorology
Local NOx
Initial & Boundary
Conditions
Local VOC
Probability of Exceeding NAAQS:
Comparison of 1-hour and 8-hour Standards
Tiered Approach to Analysis
• Purpose of Analyses (examples)
– Screening to prioritize resources
– Regulatory decision-making
– Research planning
• Types of Analyses
– Screening level point-estimates
– Sensitivity Analysis
– One-Dimensional Probabilistic Analysis
– Two-Dimensional Probabilistic Analysis
– Non-probabilistic approaches
Methods
Based Upon Expert Judgment
• Expert Elicitation
– Heuristics and Biases
» Availability
» Anchoring and Adjustment
» Representativeness
» Others (e.g., Motivational, Expert, etc.)
– Elicitation Protocols
» Motivating the expert
» Structuring
» Conditioning
» Encoding
» Verification
– Documentation
– Individuals and Groups
– When Experts Diasagree
An Example of Elicitation Protocols:
Stanford/SRI Protocol
Motivating
(Establish Rapport)
Structuring
(Identify Variables)
Conditioning
(Get Expert to Think About Evidence)
Encoding
(Quantify Judgment About Uncertainty)
Verify
(Test the Judgment)
Key Ongoing Challenges
• Expert Judgment vs. Data
– Perception that judgment is more biased than analysis
of available data
– Unless data are exactly representative, they too could
be biased
– Statistical methods are “objective” in that the results
can be reproduced by others, but this does not
guarantee absence of bias
– A key area for moving forward is to agree on conditions
under which expert judgment is an acceptable basis for
subjective probability distributions, even for rulemaking
situations
Appropriate Use of Expert Judgment in
Regulatory Decision Making
• There are examples…e.g.,
– analysis of health effects for EPA standards
– Uncertainty in benefit/cost analysis (EPA, OMB)
– Probabilistic risk analysis of nuclear facilities
• Key components of credible use of expert judgment:
– Follow a clear and appropriate protocol for selecting experts and
for elicitation
– For the conditioning step, consider obtaining input via workshop,
but for encoding, work individually with experts – preferably at their
location
– Document (explain) the basis for each judgment
– Compare judgments: identify key similarities and differences
– Evaluate the implications of apparent differences with respect to
decision objectives – do not “combine” judgments without first
doing this
– Where possible, allow for iteration
Other Quantitative Methods
• Interval Methods
–Simple intervals
–Probability bounds
–Produce “optimally” narrow bounds – cannot be
any narrower and still enclose all possible
outcomes, including dependencies among
inputs
–Bounds can be very wide in comparison to
confidence intervals
Other Quantitative Methods
• Fuzzy methods
– Representation of vagueness, rather than uncertainty
– Approximate/semi-quantitative
– Has been applied in many fields
• Meta-analysis
– Quantitatively combine, synthesize, and summarize data and
results from different sources
– Requires assessment of homogeneity among studies prior to
combining
– Produces data with larger sample sizes than the constituent inputs
– Can be applied to summary data
– If raw data are available, other methods may be preferred
Scenario Uncertainty
• A need for formal methods
• Creativity, brainstorming, imagination
• Key dimensions (e.g., human exposure assessment)
– Pollutants
– Transport pathways
– Exposure routes
– Susceptible populations
– Averaging time
– Geographic extent
– Time Periods
– Activity Patterns
• Which dimensions/combinations matter, which ones don’t?
• Uncertainty associated with mis-specification of a scenario –
systematic error
• Scenario definition should be considered when developing and
applying models
Model Uncertainty
• Model Boundaries (related to
scenario)
• Simplifications
–Aggregation
–Exclusion
•
•
•
•
•
Resolution
Structure
Calibration
Validation, Partial validation
Extrapolation
Model Uncertainty
• Methods for Dealing with Model Uncertainty
–Compare alternative models, but do not
combine
–Weight predictions of alternative models (e.g.,
probability trees)
–Meta-models that degenerate into alternative
models (e.g., Y = a(|x-t|)n , where n determines
linear/nonlinear and t determines threshold or
not)
Probability Density
Probability Density
Weighting vs. Averaging
Each Model has Equal Weight
Model B
Model A
Average of
Both Models
Output of Interest
Neither Model
Supports This
Range of Outcomes
Output of Interest
Sensitivity Analysis
• Objectives of Sensitivity Analysis (examples):
– Help identify key sources of variability (to aid management
strategy)
» Critical control points?
» Critical limits?
– Help identify key sources of uncertainty (to prioritize additional
data collection to reduce uncertainty)
– What causes worst/best outcomes?
– Evaluate model behavior to assist verification/validation
– To assist in process of model development
• Local vs. Global Sensitivity Analysis
• Model Dependent vs. Model Independent Sensitivity Analysis
• Applicability of methods often depends upon characteristics of a
model (e.g., nonlinear, thresholds, categorical inputs, etc.)
Sensitivity Analysis Methods (Examples)
•
•
•
•
•
•
•
•
•
•
•
•
Nominal Range Sensitivity Analysis
Differential Sensitivity Analysis
Conditional Analysis
Correlation coefficients (sample, rank)
Linear regression (sample, rank, variety of basis functions
possible)
Other regression methods
Analysis of Variance (ANOVA)
Categorical and Regression Trees (CART) (a.k.a. Hierarchical
Tree-Based Regression)
Sobol’s method
Fourier Amplitude Sensitivity Test (FAST)
Mutual Information Index
Scatter Plots
Schematic Diagram of the Simplified Stochastic Human Exposure
and Dose Simulation (SHEDS)-Pesticides Model
Example input Assumptions for the Simplified
SHEDS-Pesticides Model: Inhalation Pathway
Selected Sensitivity Analysis Methods
• Approximately a dozen methods were identified and
reviewed, including their advantages and
disadvantages
• The review provided a decision framework for
selecting sensitivity analysis methods based on the
objectives of the analysis and the characteristics of
the model
• A set of seven sensitivity analysis methods was finally
selected for detailed and quantitative evaluation
• Methods include: Pearson and Spearman correlation,
sample and rank regression, ANOVA, FAST, and
Sobol’s method
Three Temporal Scenarios for Exposure
Individual a
Individual b
Individual c
Individual d
(a) Scenario I: Daily Exposure
Total Exposure
( m g/kg)
100
50
0
0
5
10
15
20
25
30
15
Time (day)
20
25
30
15
Time (day)
20
25
30
Time (day)
(b) Scenario II: Incremental Change in Daily Exposure
Total Exposure
( m g/kg)
60
30
0
-30
-60
0
5
10
(c) Scenario III: Cumulative Exposure
Total Exposure
( m g/kg)
800
600
400
200
0
0
5
10
Temporal Variation of Sensitivity Indices
Based on Sobol’s Method
Main Effect
Total Effect
(b) Scenario II: Incremental Change in Daily Exposure
1
1
Sensitivity Index
Sensitivity Index
(a) Scenario I: Daily Exposure
0.8
0.6
0.4
0.2
0.8
0.6
0.4
0.2
0
0
5
10
15
Time (day)
20
25
30
Average main and total effects and
corresponding uncertainty ranges
based on 500 bootstrap simulations
for fraction of chemicals available for
transfer (FTR) as a daily input for three
temporal scenarios.
0
5
10
15
Time (day)
20
25
30
25
30
(c) Scenario III: Cumulative Exposure
1
Sensitivity Index
0
0.8
0.6
0.4
0.2
0
0
5
10
15
Time (day)
20
Temporal Variation of Sensitivity Indices
Based on Sobol’s Method
(a) Scenario I: Daily Exposure
Total Effect
(b) Scenario II: Incremental Change in Daily Exposure
1
1
Sensitivity Index
Sensitivity Index
Main Effect
0.8
0.6
0.4
0.2
0
0
5
10
15
20
25
0.8
0.6
0.4
0.2
0
30
0
5
10
Time (day)
20
25
30
20
25
30
(c) Scenario III: Cumulative Exposure
1
Sensitivity Index
Average main and total effects and
corresponding uncertainty ranges
based on 500 bootstrap simulations for
residue decay rate (DR) as a monthly
input for three temporal scenarios.
15
Time (day)
0.8
0.6
0.4
0.2
0
0
5
10
15
Time (day)
Contribution of Inputs to the Output Variance
(Scenario I) Based on Sobol’s Method
AM
BW
DR
FTR
WB
= Mass of applied pesticide
= Body washing removal efficiency
= Fraction of pesticides that dissipates daily
= Fraction of pesticide available for transfer from surface to body or hands
= Body weight
Guidance on Sensitivity Analysis
Guidance for Practitioners, with a focus on food
safety process risk models (Frey et al., 2004):
• When to perform sensitivity analysis
• Information needed depending upon objectives
• Preparation of existing or new models
• Defining the case study/scenarios
• Selection of sensitivity analysis methods
• Procedures for application of methods
• Presentation and interpretion of results
Summary of Evaluation Results for
Selected Sensitivity Analysis Methods
Example of Guidance on Selection of
Sensitivity Analysis Methods
Source: Frey et al., 2004, www.ce.ncsu.edu/risk/
Example of Guidance on Selection of
Sensitivity Analysis Methods
Communication
• Case Studies (scenarios)
• Graphical Methods
–Influence Diagrams
–Decision Tree
–Others
• Summary statistics/data
• Evaluation of effectiveness of methods for
communication (e.g., Bloom et al., 1993;
Ibrekk and Morgan, 1987)
General Recommendations (1)
• Probabilistic and sensitivity analysis should be used to address
key issues, e.g.,:
– prioritize scarce resources toward additional research or data
collection
– make choices among alternatives in the face of uncertainty,
– evaluate trends over time,
– Identify key controllable sources of variability, etc.
• When needed, such analyses should be included as functional
requirements from the beginning
• There should be minimum reporting requirements for uncertainty in
data (e.g., summary statistics such as mean, standard deviation,
sample size)
• Government agencies should continue to improve documentation
and accessibility of models and data for public peer review
General Recommendations (2)
• Foster greater acceptance of appropriate methods for including,
documenting, and reviewing expert judgment in regulatory-motivated
modeling and analysis
• There is a need for flexibility since there are many possible
approaches to analysis of variability, uncertainty and sensitivity.
• Human resources for modeling, including probabilistic and sensitivity
analysis, should be appropriately committed.
– Adequate time and budget to do the job right the first time
(could save time and money in the long run)
– Adequate training and peer review
– Promote workshops and other training opportunities, and
periodic refinement of authoritative compilations of techniques
and recommended practice
General Recommendations (3)
• Software tools substantially facilitate analyses (e.g., Crystal Ball)
but also limit what is done in practice – there is a long-term need
for software tools appropriate to specific types of applications
• Some areas need more research – e.g., best techniques for
communication, real-world information needs for decision makers
• The relevance of analyses to decision making needs to be
emphasized and considered by analysts
• Decision makers need or should have access to information on
why/how they should use probabilistic results
• A multi-disciplinary compilation of relevant case studies and
insights from them is a useful way to help convince others of the
value of doing probabilistic and sensitivity analysis
• Probabilistic and sensitivity analysis should be an open and
transparent process that can be subject to scrutiny and peer
review
Acknowledgments
• The material reported here has been
supported in part by:
–National Science Foundation
–U.S. Environmental Protection Agency
–U.S. Department of Agriculture
–U.S. Department of Energy