Transcript Methods for Developing Input Distributions for

Incorporating Risk and Uncertainty into the
Assessment of Impacts of Global Climate Change on
Transportation Systems
H. Christopher Frey, Ph.D.
Professor
Department of Civil, Construction, and Environmental Engineering
North Carolina State University
Raleigh, NC 27695
Prepared for:
2nd Workshop on Impacts of Global Climate Change
On Hydraulics and Hydrology and Transportation
Center for Transportation and the Environment
Washington, DC
March 29, 2006
Outline
• Risk and Uncertainty
• Overview of impacts of climate
change on transportation systems
• Risk assessment methodologies
• Uncertainty analysis methodologies
• Qualitative assessments
• Recommendations
2
Definitions
• Risk: Probability and severity of an adverse
outcome
• Uncertainty: Lack of knowledge regarding the
true value of a quantity
3
POSSIBLE IMPACTS OF GLOBAL CLIMATE
CHANGE ON TRANSPORTATION SYSTEMS
• All modes:
–highway, rail, air, shipping, pipeline, pedestrian
–Passenger and freight
• Possible climate impacts (natural processes)
–Sea-level rise
–Increased frequency and severity of storms
–Higher average temperatures (location-specific)
4
Implications of Possible Climate Change
(Effects Processes)
• Loss of coastal land area
• Damage to infrastructure via storms (e.g.,
winds, flooding)
• Damage to infrastructure because of
temperature extremes (e.g., rail kinks,
pavement damage)
• Impede operations and safety
• Design, construction, operation, maintenance,
repair, decommissioning
5
METHODOLOGICAL FRAMEWORKS FOR
DEALING WITH RISK
•
•
•
•
•
Vulnerability or hazard assessment
Exposure assessment
Effects processes
Quantification of risk
Risk management
6
Vulnerability Assessment
• Physical, social, political, economic, cultural, and psychological
harms to which individuals and modern societies are susceptible
(Slovic, 2002).
• Identify valuable targets at risk
• Conceptualize various ways in which they are vulnerable to such
an attack by defining various scenarios.
• Clearly state the scale and the scope of the analysis (e.g., the
world, a country, or specific region) considering that the risk
assessment process will become easier as the scope narrows
down.
• Does not include assessment of the likelihood of such an event.
• For example, coastal cities are vulnerable to the effects of sea
level rise.
7
Paradigm for Human Health Risk
Assessment (NRC, 1983)
Research
Risk Assessment
Laboratory and
Field Work
Hazard
Identification
Extrapolation
Methods
Dose-Reponse
Assessment
Field
Measurements,
Modeling
Exposure
Assessment
Regulatory
Options
Risk
Characterization
Evaluations of
Options
Decisions and
Actions
8
An Alternative View of Human Health Risk
Assessment (PCRARM, 1997)
Problem/
Context
Risks
Evaluation
Stakeholder
Collaboration
Options
Actions
Decisions
9
Example of A General Risk Assessment
Framework (Morgan)
Natural Environment
Natural
Processes
Exposure of objects
and processes in
natural and human
environment to the
possibility of change
Exposure
Processes
Human
Activities
Effects on objects
and processes in
the natural and
human
environment
Effects
Processes
Human
Perceptions
of exposures
and of effects
Human
Perception
Processes
Human Environment
Costs and
Benefits
Human
Evaluation
Processes
10
Risk Analysis and Risk Management
• Analysis should be free of policy-motivated
assumptions
• Yet, analysis should include scenarios relevant
to decision-making
• Some argue for analysts and decision makers
to be kept apart to avoid biases in the analysis
• Others argue that they must interact in order to
define the assessment objective
• A practical, useful analysis needs to balance
both concerns
11
Realities of Decision-Making
• Decision-making regarding response to the impacts of
climate change will involve:
– multiple parties;
– a local context;
– considerations beyond just the science and technology
(such as equity, justice, culture, and others); and
– implications for potentially large transfers of resources
among different societal stakeholders.
• Such decision-making may not produce an “optimal”
outcome when viewed from a particular (e.g.,
national, analytical) perspective.
Based on Morgan (2003)
12
METHODOLOGICAL FRAMEWORKS FOR
DEALING WITH UNCERTAINTY
•
•
•
•
Role of uncertainty in decision making
Scenarios
Models
Model inputs
–Empirically-based
–Expert judgment-based
• Model outputs
• Other quantitative approaches
• Qualitative approaches
13
Uncertainty and Decision Making
• How well do we know these numbers?
– What is the precision of the estimates?
– Is there a systematic error (bias) in the estimates?
– Are the estimates based upon measurements,
modeling, or expert judgment?
• How significant are differences between two
alternatives?
• How significant are apparent trends over time?
• How effective are proposed control or management
strategies?
• What is the key source of uncertainty in these numbers?
14
• How can uncertainty be reduced?
Implications of Uncertainty in Decision
Making
• Risk preference
–Risk averse
–Risk neutral
–Risk seeking
• Utility theory
• Benefits of quantifying uncertainty: Expected
Value of Including Uncertainty
• Benefits of reducing uncertainty: Expected
Value of Perfect Information
15
Framing the Problem: Objectives and
Scenarios
• Need a well-formulated study objective that is
relevant to decision making
• A scenario is a set of structural assumptions about
the situation to be analyzed:
– spatial and temporal dimensions
– specific hazards, exposures, and adverse outcomes
• Typical errors: description, aggregation, expert
judgment, incompleteness
• Failure to properly specify scenario(s) leads to bias in
the analysis, even if all other elements are perfect.
16
Model Uncertainty
• A model is a hypothesis regarding how a system
works.
• Ideally, the model should be tested by comparing its
predictions with observations from the real world
system, under specified conditions.
• Difficult for unique or future events.
• In practice, validation is often incomplete.
• Extrapolation.
• Other factors: simplifications, aggregation, exclusion,
structure, resolution, model boundaries, boundary
conditions, and calibration.
17
System Response
Examples of Alternative Models
State
Change?
Sublinear
Linear
Superlinear
Threshold
Explanatory Variable
18
Model Uncertainty – Climate Change
Impacts
• Enumeration of a set of plausible or possible
alternative models,
• Comparisons of their predictions or
development of a weighting scheme to
combine the predictions of multiple models into
one estimate
• It seems inappropriate to increase the
complexity of the analysis in situations where
less is known (Casman et al., 1999)
19
Model Uncertainty
Model 1
w1
w2
Model 2
w3
Model 3
Weighted Combination
Of Model Outputs
20
The Role of Models When Structural
Uncertainties are Large
• Assessment of climate change impacts involves many
component models
• Some are better than others, and they “degrade” at
different rates as one goes farther into the future.
• For problem areas in which there is little relevant
data, theory, or experience, a simpler “order-ofmagnitude” model may be adequate.
• For problem areas in which little is known, very simple
bounding analyses may be all that can be justified.
• For poorly supported models, it is no longer possible
to search for optimal decision strategies. Instead,
one can attempt to find feasible or robust strategies
21
Quantification of Uncertainty in Inputs and
Outputs of Models
Input Uncertainties
Output
Uncertainty
Model
22
Statistical Methods
Based Upon Empirical Data
• Frequentist, classical
• Statistical inference from sample data
–Parametric approaches
» Parameter estimation
» Goodness-of-fit
–Nonparametric approaches
–Mixture distributions
–Censored data
–Dependencies, correlations, deconvolution
–Time series, autocorrelation
23
Statistical Methods Based on Empirical
Data
• Need a random, representative sample
• Not always available when predicting events
into the future
24
Cumulative Probability
Example of an Empirical Data Set
Regarding Variability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
Empirical Quantity
0.1
1
Benzene Emission Factor
(ton/yr/tank)
25
Fitted Lognormal Distribution
Cumulative Probability
1
0.8
0.6
0.4
0.2
0
0.001
0.01
Empirical Quantity
0.1
1
Benzene Emission Factor
(ton/yr/tank)
26
Bootstrap Simulation to Quantify
Uncertainty
Cumulative Probability
1.0
0.8
0.6
Dat a Set
Fit t ed Dist ribut ion
Confidence90Int
erval
percent
50 percent
90 percent
95 percent
0.4
0.2
0.0
-3
10
-2
-1
10
10
0
10
Empirical
Quantity
Benzene Emission Fact or
(t on/yr/t ank)
27
Results of Bootstrap Simulation:
Uncertainty in the Mean
Cumulative Probability
1
0.8
mean =0.06
0.6
0.6
0.4
95% Probability
Range (0.016, 0.18)
0.2
0
0
0.05
0.1
0.15
0.2
BenzeneQuantity
Emission Factor
Empirical
(ton/yr/tank)
Uncertainty in mean -73% to +200%
28
Estimating Uncertainties Based on Expert
Judgment
• Probability can be used to quantify the state of knowledge (or
ignorance) regarding a quantity.
• Bayesian methods for statistical inference are based upon sample
information (e.g., empirical data, when available) and a prior
distribution.
• A prior distribution is a quantitative statement of the degree of
belief a person has that a particular outcome will occur.
• Methods for eliciting subjective probability distributions are
intended to produce estimates that accurately reflect the true state
of knowledge and that are free of significant cognitive and
motivational biases
• Useful when random, representative data, or models, are not
available, but when there is some “epistemic status” upon which to
base a judgment
29
Heuristics and Possible Biases in Expert
Judgment
• Heuristics and Biases
– Availability
– Anchoring and Adjustment
– Representativeness
– Others (e.g., Motivational, Expert, etc.)
• Consider motivational bias when choosing experts
• Deal with cognitive heuristics via an appropriate
elicitation protocol
30
An Example of an Elicitation Protocol:
Stanford/SRI Protocol
Motivating
(Es tablish Rapport)
Structuring
(Identify Variables )
Conditioning
(Get Expert to Think About Evidence)
Encoding
(Quantify Judgment About Uncertainty)
Verify
(Tes t the Judgment)
31
Frequently Asked Questions Regarding
Expert Elicitation
• How to choose the experts
• How many experts are needed
• Whether to perform elicitation individually or with
groups of experts
• Elicitation of correlated uncertainties
• What to do if experts disagree
• Whether and how to combine judgments from
multiple experts
• What resources are needed for expert elicitation
32
Propagating Uncertainties Through Models
• Analytical solutions – exact but of limited
applicability
• Approximate solutions – more broadly
applicable but increase in complexity or error
as model and inputs become more complex
(e.g., Taylor series expansion)
• Numerical methods – flexible and popular
(e.g., Monte Carlo simulation)
33
Monte Carlo Simulation and Similar
Methods
f(x)
1
Probability
Density
Cumulative
Probability, u
PROBABILITY
DENSITY FUNCTION
F(x)==Pr(x≤X)
P(xŠX)
F(x)
CUMULATIVE
DISTRIBUTION
FUNCTION
0
Value of Random Variable, x
LATIN HYPERCUBE SAMPLING
• Divide u into N equal intervals
• Select median of each interval
• Calculate F-1 (u) for each interval
• Rank each sample based on U(0,1)
(or restricted pairing technique)
-1
F (u)
Value of Random
Variable, x
MONTE CARLO SIMULATION
• Generate a random number u~U(0,1)
• Calculate F-1 (u) for each value of u
Value of Random Variable, x
0
INVERSE CUMULATIVE
DISTRIBUTION
FUNCTION
Cumulative Probability, u
1
34
Sensitivity Analysis: Which Model Inputs
Contribute Most to Uncertainty in Output?
• Linearized sensitivity
coefficients
• Statistical methods:
∂z
= sy,b
∂y
b
z
– Correlation
– Regression
– Advanced methods
∂z
= sx,b
∂x
b
∂z
= sy,a
∂y
a
∂z
= sx,a
∂x
a
y
(xb,yb)
(xa,ya)
Interactions,
24%
x
F TR , 25%
DR , 1%
BW, 8%
Main Effect
of Others,
30%
WB, 6%
AM, 6%
Example from
Sobol’s Method
35
Other Quantitative Methods
• Interval Methods: Provide bounds, but not
very informative
• Fuzzy Sets: represents vagueness, rather
than uncertainty
36
Qualitative Methods
• Principles of Rationality
• Lines of Reasoning
• Weight of Evidence
37
Principles of Rationality
• Conceptual clarity: well-defined terminology
• Logical consistency: inferences should follow
from assumptions and data
• Ontological realism: free of scientific error
• Epistemological reflection: evidential support
• Methodological rigor: use of proven
techniques
• Practicality
• Valuational selection: focus on what matters
the most
38
Lines of Reasoning
• Direct empirical evidence
• Semi-empirical evidence (surrogate data)
• Empirical correlations (relationships between
known processes and the unknown process of
interest)
• Theory-based inference – causal mechanisms
• Existential insight – expert judgment
39
Judgment of Epistemic Status
• The result of an analysis of epistemic status is
a judgment regarding the quality of each
premise or alternative – e.g.,
–no basis for using a premise in decisionmaking.
–partial or high confidence basis for using a
particular premise as the basis for decision
making.
40
Weight of Evidence
• Legal context - whether the proof for one
premise is greater than for another.
• Often used when a categorical judgment is
needed.
• However,
–tends to be less formal than the analysis of
epistemic status,
–less transparent than properly documented
analyses of epistemic status
41
Qualitative Statements Regarding
Uncertainty
• Qualitative approaches for describing uncertainty are
best with fundamental problems of ambiguity.
• The same words mean:
– different things to different people,
– different things to the same person in different contexts
• Based on Wallsten et al., 1986:
– “Probable” was associated with quantitative
probabilities of approximately 0.5 to 1.0
– “Possible” was associated with probabilities of
approximately 0.0 to 1.0.
• Qualitative schemes for dealing with uncertainty are
typically not useful
42
CONCLUSIONS - 1
• There is growing recognition that climate
change has the potential to impact
transportation systems.
• The available literature on the impacts of
climate change on transportation systems
appears to be a vulnerability assessment,
rather than a risk analysis.
43
CONCLUSIONS - 2
• The commitment of large resources should be based on,
as thoroughly as necessary or possible, a well-founded
analysis.
• There are many alternative forms of analysis that differ in
their “epistemic status,” depending on what type of
information is available.
• Thus, the key question is what kind of analysis is
appropriate here?
• It may be possible to seek feasible, and perhaps robust
(but not optimal) solutions for dealing with climate change
impacts.
• Actual decisions will be based on a complex deliberative
process, to which analysis is only one input
44
CONCLUSIONS - 3
• There is substantial uncertainty attributable to
the structure of scenarios and models.
• Given the lack of directly relevant empirical
data for making assessments of future
impacts, there is a strong need for the use of
judgments regarding uncertainty elicited from
experts
45
RECOMMENDATIONS
• Vulnerability assessment is only a first step.
• Modeling tools should be used to identify feasible and robust
solutions
• Assessment should be done iteratively over time.
• Expert judgment should be included as a basis for quantifying the
likelihood and severity of various outcomes, as well as
uncertainties.
• Uncertainties should be quantified to the extent possible.
• Sensitivity and uncertainty analysis should be used together to
identify key knowledge gaps that could be prioritized for addition
data collection or research in order to improve confidence in
estimates.
• In order to focus policy debate and inform decision making, these
analyses are highly recommended, despite their limitations
46
ACKNOWLEDGMENTS
• Hyung-Wook Choi, of the Department of Civil,
Construction, and Environmental Engineering
at NC State, provided assistance with the
literature review.
• This work was supported by the Center for
Transportation and the Environment.
However, the author is solely responsible for
the content of this material.
47