Transcript Assessing Uncertainty

Assessing
Uncertainty
HRA – INCOSE
November 8 & 9 2007
Futron Corporation • 2021 Cunningham Drive, Suite 303 • Hampton, Virginia 23666
Phone 757-262-2074 • Fax 757-262-2173 • www.futron.com
Results You Can Trust
Overview
•
•
•
•
•
•
•
•
•
Interactive Exercise
Definitions
Why Assess Uncertainty
Interactive Exercise
Types of Uncertainty
Sources of Uncertainty
Assessing (Bounding) Uncertainty
Interactive Exercise
Summary
Results You Can Trust
• 2
Interactive Exercise
Results You Can Trust
• 3
Definitions: Uncertainty
• Uncertainty: may range from a
falling short of certainty to an
almost complete lack of
conviction or knowledge
especially about an outcome or
result. Lack of clear definition
Results You Can Trust
• 4
Definitions: Risk
• Risk: possibility (chance) of loss or injury (Likelihood, Consequence.
Ordered Pair )
• Probability: (1) : the ratio of the number of outcomes in an exhaustive set of
equally likely outcomes that produce a given event to the total number of
possible outcomes (2) : the chance that a given event will occur (3) A
numerical measure of the likelihood of an event relative to a set of alternative
events.
Generic Risk Matrix
5
• Likelihood: probability
4
Likelihood
3
2
1
1
2
3
4
5
Consequence
Results You Can Trust
• 5
Definitions: Confidence Interval
• Confidence Interval: a group of continuous or discrete adjacent values that is
used to estimate a statistical parameter (as a mean or variance) and that tends
to include the true value of the parameter a predetermined proportion of the
time if the process of finding the group of values is repeated a number of
times
• Used to quantify/communicate amount of certainty
• 90% confidence interval (limit/bounds)
mean
Results You Can Trust
• 6
Why Assess Uncertainty?
• We manage risks to ensure we meet our objectives
• We have tasks, projects, and programs to accomplish




They all have objectives
We all make decisions and commit resources to meet these objectives
Decision are choices between alternatives and involve uncertainty
This uncertainty leads to risk
• Managing uncertainty is a part of our job




Involves understanding our decisions
Managing the uncertainty in those decisions
Identifying the risks to meeting our objectives that come from uncertainty
Make better decisions / select better mitigation strategies
Results You Can Trust
• 7
Interactive Exercise
Results You Can Trust
• 8
Types of Uncertainty: Aleatory
• Aleatory





Randomness or variability that remains when the system is well known
Inherent Variability
You can calculate the odds
Example: Drawing a queen of diamonds from a standard deck of 52
playing cards on the first draw.
Not reducible
Results You Can Trust
• 9
Types of Uncertainty: Epistemic
• Epistemic





Stems from limitations of fundamental
knowledge about a phenomenon of interest.
Model Uncertainty
Can be revealed by divergences in the opinions
of experts who may disagree.
Example: Drawing a queen of diamonds from a
deck of playing cards on the first draw when you
have no idea how many cards are in the deck
This type of uncertainty is reducible:
• Improvements of current knowledge
• Improvements of corresponding models
• Collection of additional data (test results)
Results You Can Trust
• 10
Sources of Uncertainty
• Insufficient data




Measurement errors
State not directly observed (inferred)
Imperfect understanding of processes
Not enough data to be statistically significant
• Natural variability


Varies spatially (with space)
Varies temporally (with time)
• Model imperfections
•
Underestimating uncertainty and the illusion of control - We tend to underestimate future
uncertainty because we tend to believe we have more control over events than we really
do. We believe we have control to minimize potential problems in our decisions.
Results You Can Trust
• 11
Bounding Uncertainty
Uniform(-0.17000, 0.20000)
Six levels of “sophistication”
• Simply establish the existence of uncertainty
(risk)
• Identifying a “worst-case” scenario (unbiased
bounds – one tailed or two tailed)
• Computing the effects of the quasi-worst case
scenario or plausible upper bound
• Computing a maximum-likelihood estimate
of the failure probability of losses
• Modeling the probability distribution of
potential outcomes (risk curve)
• Modeling explicitly the uncertainties about
the risk, separating epistemic from aleatory
3
X <= -0.15150
5.0%
X <= 0.18150
95.0%
2.5
2
1.5
1
0.5
0
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.2
0.25
Triang(-.17, 0, .2)
X <= -0.11392
5.0%
6
X <= 0.13917
95.0%
5
4
3
2
1
0
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
Normal(0, .113) Trunc(-.4,+inf) Shift=+.015
X <= -0.17066
5.0%
4
X <= 0.20088
95.0%
3.5
3
2.5
2
1.5
1
0.5
0
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Results You Can Trust
• 12
Interactive Exercise
Results You Can Trust
• 13
Summary
•
•
•
•
It is not easy
We think we know more than we really do
Understand, assess and communicate your own uncertainty
Understand how others assess and communicate their
uncertainty
• Focus Mitigation Strategies
Results You Can Trust
• 14
Bibliography
•
•
•
•
•
•
Ang, Alfred H-S. and Tang, Wilson H. “Probability Concepts in Engineering
Planning and Design, Volume I – Basic Principles”
Ang, Alfred H-S. and Tang, Wilson H. “Probability Concepts in Engineering
Planning and Design, Volume II – Decision, Risk and Reliability.” John Wiley,
New York 1984
Roberds, William J. (1990). “Methods for Developing Defensible Subjective
Probability Assessments,” Transportation Research Record No. 1288, 183-190.
CE311S Elementary Probability and Statistics for Civil Engineers class notes,
University of Texas – Austin
Futron Technical University (FTU) Fundamentals of Risk Management course
notes
Decision Analysis Training for ESMD Risk Management Officers, by Dr. Douglas
Stanley, Georgia Institute of Technology, National Institute of Aerospace
• The Epistimic Uncertainty Project, Sandia National Laboratories
http://www.sandia.gov/epistemic/
• Wikipedia (interactive exercises) http://en.wikipedia.org/wiki/Main_Page
• Encyclopedia Britannica Online http://www.britannica.com/
Results You Can Trust
• 15